stosim

sim

Assisting code to support other code and prepare simulations

commands

This module provides all commands you can use in StoSim: run, resume, status, kill, snapshot, run_more, make_plots, run_ttests, list_data

stosim.sim.commands.kill(simfolder)

Kill simulation Warning: On pbs, kills all your jobs (ignores –project)!

stosim.sim.commands.list_data(simfolder)

List the number of runs that have been made per configuration.

Parameters:simfolder (string) – relative path to simfolder Returns:True if successful, False otherwise

stosim.sim.commands.make_plots(simfolder, plot_nrs=[])

generate plots as specified in the simulation conf

Parameters:

• simfolder (string) – relative path to simfolder
• plot_nrs (list) – a list with plot indices. If empty, plot all

stosim.sim.commands.prepare_folders_and_jobs(simfolder, limit_to={}, more=False)

ensure that data and job directories exist, create jobs. limit_to can contain parameter settings we want to limit ourselves to (this is in case we add more data)

Parameters:

• simfolder (string) – relative path to simfolder
• limit_to (dict) – key-value pairs that narrow down the dataset, when empty (default) all possible configs are run
• more (boolean) – when True, new data will simply be added

stosim.sim.commands.resume(simfolder)

(Re)start dispatching jobs

Parameters:simfolder (string) – relative path to simfolder Returns:True if successful, False otherwise

stosim.sim.commands.run(simfolder)

The main function to start running simulations

Parameters:simfolder (string) – relative path to simfolder Returns:True if successful, False otherwise

stosim.sim.commands.run_more(simfolder)

let the user make more runs on current config, in addition to the given data

Parameters:simfolder (string) – relative path to simfolder Returns:True if successful, False otherwise

stosim.sim.commands.run_ttests(simfolder)

Make statistical t tests

Parameters:simfolder (string) – relative path to simfolder

stosim.sim.commands.snapshot(simfolder, identifier=None)

Make a snapshot of the current state, using rsync with an example from http://schlutech.com/2011/11/rsync-full-incremental-differential-snapshots/

Parameters:

• simfolder (string) – relative path to simfolder
• identifier (string) – custom identifier for this snapshot

Returns:

True if successful, False otherwise

stosim.sim.commands.status(simfolder)

Check status of simulation. If on PBS scheduling, show status of nodes. Then, ask the fjd-dispatcher about status of jobs.

Parameters:simfolder (string) – relative path to simfolder Returns:True if successful, False otherwise

job_creator

This module sets up the stage - with it, you can make a configuration file for each mix of parameter settings (for each job) that needs to be run.

stosim.sim.job_creator.create_jobs(main_conf, simfolder, limit_to={}, more=False)

Writes a conf file for each run that the parameters in conf suggest.

Parameters:

• conf (ConfigParser) – main configuration
• simfolder (string) – relative path to simfolder
• limit_to (dict) – dict of configuration settings we should limit to (default is an empty dict)
• more (boolean) – True if runs should be appended to existing data (default is False)

utils

stosim.sim.utils.check_conf(simfolder)

check if stosim.conf contains all necessary sections and options :param string simfolder: relative path to simfolder

stosim.sim.utils.check_for_older_data(simfolder, more=False)

check if old data is lying around, ask if it can go

Parameters:more (boolean) – when True, new data will simply be added to existing data

stosim.sim.utils.decode_search_from_confstr(s, sim='')

Make dict out of configuration string that describes a search for a sub-dataset (for plots, ttests)

Parameters:s (string) – a string of comma-separated key-value pairs Returns:dictionary made from the string

stosim.sim.utils.ensure_name(simfolder)

make sure we have the actual name of the folder and not just ‘.’

Parameters:simfolder (string) – relative path to simfolder Returns:the full name (without the path to it)

stosim.sim.utils.get_combined_conf(simfolder)

Make a configuration with all used params and their values - this means we gather all param values from eventual subconfs, next to stosim.conf

Parameters:simfolder (string) – relative path to simfolder Returns:ConfigParser combined conf

stosim.sim.utils.get_delimiter(conf)

get the delimiter used in the log files between values

Parameters:conf (ConfigParser) – the main config file Returns:the user-defined delimiter or the assumed default one (,)

stosim.sim.utils.get_interval(simfolder)

get the interval for FJD workers and dispatchers to check the queue, in seconds (fractions of seconds are allowed).

Parameters:simfolder (string) – relative path to simfolder Returns:float interval

stosim.sim.utils.get_jobtime(simfolder)

get the expected maximal jobtime for PBS jobs, the format is HH:MM:SS

Parameters:simfolder (string) – relative path to simfolder Returns:string jobtime

stosim.sim.utils.get_main_conf(simfolder)

Return ConfigParser object read from main conf, with all relevant subsimulation configs set

Parameters:simfolder (string) – relative path to simfolder Returns:ConfigParser object

stosim.sim.utils.get_numcores(simfolder)

get the number of cores on PBS nodes

Parameters:simfolder (string) – relative path to simfolder Returns:int number of cores

stosim.sim.utils.get_relevant_confs(simfolder)

Returns:ConfigParser objects for all config files being used Parameters:simfolder (string) – relative path to simfolder

stosim.sim.utils.get_scheduler(simfolder)

get the scheduler (fjd or pbs)

Parameters:simfolder (string) – relative path to simfolder Returns:string scheduler

stosim.sim.utils.get_simulation_name(simfolder, conf_filename, fallback=None)

The user can give a pretty name to the simulation under [meta], this function returns it.

Parameters:

• conf_filename (string) – name of the config file for the simulation
• fallback (string) – return this if the user didn’t specify any

Returns:

string pretty name

stosim.sim.utils.get_subsimulation_names(conf)

get sub-simulation config names.

Parameters:conf (ConfigParser) – main configuration Returns:A list with names. If no subsimulations are configured, the list will have an empty string as only element

stosim.sim.utils.make_simdir_name(simfolder)

Make the name for a simulation dir from the simulation name and the name of conf files [This and make_screen_name need overhaul and a common approach when a session management is implemented]

Parameters:simfolder (string) – relative path to simfolder

stosim.sim.utils.read_args()

read in cmd line arguments for StoSim, print usage if something is unexpected :returns: arg object returned by argparse

stosim.sim.utils.runs_in_folder(simfolder, fname)

Returns:number of runs that have been made in this data folder Parameters:simfolder (string) – relative path to simfolder

analysis

Code used to analyse results: Plotting and T-testing. The general workflow for an analysis: We collect values from the whole data set, be it whole files which are averaged or just selected values from files. The one result file will be plotted or is input to a T-Test.

harvester

This module helps to collect certain files and values from the whole data set, based on search criteria. Main functions to use are collect_files and collect_values.

stosim.analysis.harvester.collect_files(searches, filepath, target_dir)

For each search, collect all .log-files from the data directories that matches the criteria. Create a subdir for each search in target_dir where the .log-files go, numbered from 1 to n.

Parameters:

• searches (dict) – a dict, where the keys are names of searches and values are lists of key-value tuples which dirnames should match
• filepath (string) – path to data directories
• target_dir (string) – directory where the subdirs should go

Returns:

a list with searches that didn’t match any folders

stosim.analysis.harvester.collect_values(filepath, delim, outfile_name, cols=[], selector='all')

Collect specific x/y values from a bunch of .log files (to be found via the filepath) and write them into a new file.

TODO: how to pass custom selectors?

Parameters:

• filepath (string) – path to the log files
• delim (string) – delimiter in the data files
• outfile_name (string) – path and name of file to write into
• cols (list) – columns to select, at least one, we consider the first as x, the second as y
• selector – method to specify how to choose lines from the data files one of [‘all’, ‘last’, ‘max_x’, ‘max_y’, ‘min_x’, ‘min_y’]

stosim.analysis.harvester.extreme(filep, cols, delim, sel=<built-in function max>, by=0)

Helper for selectors. Gets lines with maximal or minimal value,

looking for those values in a column of choice.

Parameters:

• filep (file) – pointer to data file
• cols (list) – list of column indices from which to collect
• delim (string) – delimiter in the data files
• sel (function) – function to selct value from a list, max or min
• by (int) – the column of choice (0 for x, 1 for y), default 0

Returns:

a string with the values from the line with minimal y-value

stosim.analysis.harvester.matches(string, search)

Returns:

True if string contains all key/value pairs in search s, False otherwise

Parameters:

• string (string) – string to search in
• search (dict) – key-values in this dict are the search

stosim.analysis.harvester.select_all(filep, cols, delim)

select column values of all lines

Parameters:

• filep (file) – pointer to data file
• cols (list) – list of column indices from which to collect
• delim (string) – delimiter in the data files

Returns:

string with lines

stosim.analysis.harvester.select_last(filep, cols, delim)

Returns:

a string with the last values from this file and columns

Parameters:

• filep (file) – pointer to data file
• cols (list) – list of column indices from which to collect
• delim (string) – delimiter in the data files

Returns:

string with line

stosim.analysis.harvester.select_max_x(filep, cols, delim)

Returns:

a string with the values from the line with maximal x-value

Parameters:

• filep (file) – pointer to data file
• cols (list) – list of column indices from which to collect
• delim (string) – delimiter in the data files

stosim.analysis.harvester.select_max_y(filep, cols, delim)

Returns:

a string with the values from the line with maximal y-value

Parameters:

• filep (file) – pointer to data file
• cols (list) – list of column indices from which to collect
• delim (string) – delimiter in the data files

stosim.analysis.harvester.select_min_x(filep, cols, delim)

Returns:

a string with the values from the line with minimal x-value

Parameters:

• filep (file) – pointer to data file
• cols (list) – list of column indices from which to collect
• delim (string) – delimiter in the data files

stosim.analysis.harvester.select_min_y(filep, cols, delim)

Returns:

a string with the values from the line with minimal y-value

Parameters:

• filep (file) – pointer to data file
• cols (list) – list of column indices from which to collect
• delim (string) – delimiter in the data files

compressor

Compresses data (e.g. averaging)

stosim.analysis.compressor.avg_stats(xCol, yCol, numFiles, filePrefix='', fileSuffix='', filePath='.', delim=', ', outName=None)

This function can take several data files and transfer them into a file that is formatted ready to be plotted by gnuplot (each line is “<x> <mean of y vals> <std of y vals>”). Data files should be named using an index starting by 1 and all have the same prefix and/or suffix arround that index.

In gnuplot, you could then say “plot outName smooth unique with yerrorlines”

Parameters:

• xCol (int) – x column
• yCol (int) – y column
• numFiles (int) – the number of files to average over
• filePrefix (string) – prefix in filenames
• fileSuffix (string) – suffix in filenames
• filePath (string) – path to files
• delim (string) – delimiter used between columns
• outName (string) – Name of the result file, defaults to <filePrefix><yCol><fileSuffix>.out

plotter

stosim.analysis.plotter.plot(filepath='', delim=', ', outfile_name='', name='My simulation', xcol=1, x_label='iteration', y_label='value', x_range=None, y_range='[0:10]', use_y_errorbars=False, errorbar_every=1, infobox_pos='top left', use_colors=True, use_tex=False, line_width=6, font_size=22, custom_script='', plots=[])

Make plots for specific portions of data in one figure.

For each graph, this function …

• selects folders with data files depending on parameter values you provide
• collects all log files contained in them in a temporary folder
• averages over the contents or just selects values from them
• makes a (gnu)plot out of that, with yerrorbars if you want

The output is one PDF file. Creating PDF for this yields far nicer linetypes. Also, this makes different linetypes (e.g. dashes/dots) possible, bcs papers are often printed b/w.

In addition to gnuplot, you need epstopdf installed.

Parameters:

• filepath (string) – path to data
• delim (string) – delimiter used between columns
• outfile_name (string) – name of the PDF file you want to make
• name (string) – Title of the simulation
• xcol (int) – column >= 1
• x_label (string) – label on x axis
• y_label (string) – label on y axis
• x_range (string) – the range of values for x axis, in the form of “[a:b]”, defaults to None, meaning that gnuplot should take to the actual value range of x values
• y_range (string) – the range of values for y axis, defaults to ‘[0:10]’
• use_y_errorbars (boolean) – True if errorbars should be shown, default is False
• errorbar_every (int) – show an errorbar every x steps, defaults to 1
• infobox_pos (string) – where the infobox should go, e.g. ‘bottom left’, defaults to ‘top left’ when given empty
• use_colors (boolean) – whether to use colors or be b+w, defaults True
• use_tex (boolean) – whether to use enhanced mode, which interpretes tex-like encoding (e.g. subscript, math, greek symbols), defaults to False
• line_width (int) – line width in pixels, defaults to 6
• font_size (int) – font size (a number), defaults to 22
• plots (list) – list of plot descriptions, defaults to empty list

tester

stosim.analysis.tester.ttest(simfolder, c, i, delim)

Run a T-test with Gnu R

Parameters:

• simfolder (string) – path to simulation folder
• c (ConfigParser) – the config file where this test is described
• i (int) – number of test in that config file