Using stochastic features¶
Summary¶
In this example, we revisit the example from Using subsimulations, only now we look at all the features StoSim offers for stochastic simulations. We look at error-bars in line plots, fixing a seed for each run for repeatability, performing T-Tests and adding more runs to the database (as well as checking what is in the database).
Error-bars¶
First, let’s add error-bars to the line graphs, so we can see how much variance is in the data. There is two configuration settings that are of interest for this:
[plot-settings]
use-y-errorbars: 1
First, we say that we indeed want to see error-bars and then we say how far apart, horizontally, they should be plotted. Here is one plot from the second subsimulation, to show you what it looks like:
Errorbar plot from sub-simulation example
We see that there is not much variance anywhere, except maybe when all agents are learners.
Note
This feature is restricted to line plots. It doesn’t make a lot of sense in a scatter plot.
Seeds¶
You can provide a seed for every run of your simulation, for repeatability of your randomised results. In this example, we do five runs, so we provide a distinct seed for every one. Here are the first five:
[seeds]
1: 12345678
2: 23456789
3: 34567891
4: 45678912
5: 56789123
The seed is written into the job configuration which is passed to your executable. Have a look in the executable for this simulation (near the bottom) to see how it uses the gets the seed and seeds the randomizer with it.
Note
The length of a seed is up to you. Your program needs to handle it. For instance, it might matter if you store it in an integer or a long variable.
T-tests¶
A T-Test is configured much like a Figure, only it is simpler. In the first sub-simulation, we saw that payoff was bigger in the cooperative scenario. Let’s put that to a T-Test:
[ttest1]
name: payoff_coop_vs_noncoop
set1: _name:coop, _col:3, sim:1, mean_coop:0.8
Here is the output of the Gnu R T-Test, which confirms our hypothesis that the difference is significant:
nic@fidel:/media/data/projects/stosim/trunk/examples/stochastic$ stosim --ttests
********************************************************************************
[StoSim] Running T-tests ...
********************************************************************************
Test payoff_coop_vs_noncoop:
> coop <- read.table('coop.dat')
> noncoop <- read.table('noncoop.dat')
> t.test(coop,noncoop)
Welch Two Sample t-test
data: coop and noncoop
t = 61.9744, df = 3477.865, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.6861417 0.7309741
sample estimates:
mean of x mean of y
2.406372 1.697814
Adding more runs¶
Sometimes you might want to add a couple of runs, to add statistical
weight to the results. That is quite easy in StoSim. The following session
shows how to use the --list and --more commands to see how
many runs you have and add more. Here, I had only made runs for simulation 2
(stosim --simulations=sim2) and I add another 5 runs (note that
I now have 10 runs for some of the configurations, so I’d better also
have 10 seeds):
nic@fidel:/media/data/projects/stosim/trunk/examples/stochastic$ stosim --list
[StoSim] The configurations and number of runs made so far:
sim1
No runs found for simulation sim1
sim2
--------------------------------------------------------------------------------------------------------
| mean_coop | n | epochs | pd_t | ratio_learning | pd_p | pd_s | pd_r | | runs |
--------------------------------------------------------------------------------------------------------
| 0.2 | 100| 200 | 5 | 0.25 | 1 | 1 | 5 | | 5 |
--------------------------------------------------------------------------------------------------------
| 0.8 | 100| 200 | 5 | 0.25 | 1 | 1 | 5 | | 5 |
--------------------------------------------------------------------------------------------------------
| 0.8 | 100| 200 | 5 | 0.75 | 1 | 1 | 5 | | 5 |
--------------------------------------------------------------------------------------------------------
| 0.2 | 100| 200 | 5 | 0.75 | 1 | 1 | 5 | | 5 |
--------------------------------------------------------------------------------------------------------
| 0.2 | 100| 200 | 5 | 1 | 1 | 1 | 5 | | 5 |
--------------------------------------------------------------------------------------------------------
| 0.8 | 100| 200 | 5 | 1 | 1 | 1 | 5 | | 5 |
--------------------------------------------------------------------------------------------------------
nic@fidel:/media/data/projects/stosim/trunk/examples/stochastic$ stosim --more
[StoSim] Let's make 5 more runs! Please tell me on which configurations.
Enter any parameter values you want to narrow down to, nothing otherwise.
ratio_learning ? (out of [0.25,0.75,1])
1
mean_coop ? (out of [0.2,0.8])
No restriction chosen.
You selected: {'ratio_learning': ['1']}. Do this? [Y|n]
(Remember that configuration and code should still be the same!)
********************************************************************************
Running simulation The stochastic-features example
********************************************************************************
********************************************************************************
[StoSim] Running jobs on cpu 1 of server fidel
[StoSim] Processing 1/4
(section sim1_mean_coop0.2_n100_epochs200_pd_t5_ratio_learning1_pd_p1_pd_s0_pd_r3)
. . . . .
[StoSim] Processing 2/4
(section sim2_mean_coop0.2_n100_epochs200_pd_t5_ratio_learning1_pd_p1_pd_s1_pd_r5)
. . . . .
[StoSim] Processing 3/4
(section sim1_mean_coop0.8_n100_epochs200_pd_t5_ratio_learning1_pd_p1_pd_s0_pd_r3)
. . . . .
[StoSim] Processing 4/4
(section sim2_mean_coop0.8_n100_epochs200_pd_t5_ratio_learning1_pd_p1_pd_s1_pd_r5)
. . . . .
********************************************************************************
nic@fidel:/media/data/projects/stosim/trunk/examples/stochastic$ stosim --list
[StoSim] The configurations and number of runs made so far:
sim1
--------------------------------------------------------------------------------------------------------
| mean_coop | n | epochs | pd_t | ratio_learning | pd_p | pd_s | pd_r | | runs |
--------------------------------------------------------------------------------------------------------
| 0.2 | 100| 200 | 5 | 1 | 1 | 0 | 3 | | 5 |
--------------------------------------------------------------------------------------------------------
| 0.8 | 100| 200 | 5 | 1 | 1 | 0 | 3 | | 5 |
--------------------------------------------------------------------------------------------------------
sim2
--------------------------------------------------------------------------------------------------------
| mean_coop | n | epochs | pd_t | ratio_learning | pd_p | pd_s | pd_r | | runs |
--------------------------------------------------------------------------------------------------------
| 0.2 | 100| 200 | 5 | 0.25 | 1 | 1 | 5 | | 5 |
--------------------------------------------------------------------------------------------------------
| 0.8 | 100| 200 | 5 | 0.25 | 1 | 1 | 5 | | 5 |
--------------------------------------------------------------------------------------------------------
| 0.8 | 100| 200 | 5 | 0.75 | 1 | 1 | 5 | | 5 |
--------------------------------------------------------------------------------------------------------
| 0.2 | 100| 200 | 5 | 0.75 | 1 | 1 | 5 | | 5 |
--------------------------------------------------------------------------------------------------------
| 0.2 | 100| 200 | 5 | 1 | 1 | 1 | 5 | | 10 |
--------------------------------------------------------------------------------------------------------
| 0.8 | 100| 200 | 5 | 1 | 1 | 1 | 5 | | 10 |
--------------------------------------------------------------------------------------------------------