Controlling Randomness¶
Many experiments rely on some form of randomness. Controlling this randomness is key to ensure reproducibility of the results. This typically happens by manually seeding the Pseudo Random Number Generator (PRNG). Sacred can help you manage this error-prone procedure.
Automatic Seed¶
Sacred auto-generates a seed for each run as part of the configuration (You might have noticed it, when printing the configuration of an experiment). This seed has a different value everytime the experiment is run and is stored as part part of the configuration. You can easily set it by:
>>./experiment.py with seed=123
This root-seed is the central place to control randomness, because internally all other seeds and PRNGs depend on it in a deterministic way.
Global Seeds¶
Upon starting the experiment, sacred automatically sets the global seed of
random
and (if installed) numpy.random
(which is with v1.19 mark as legacy),
tensorflow.set_random_seed
, pytorch.manual_seed
to the auto-generated
root-seed of the experiment. This means that even if you don’t take any further
steps, at least the randomness stemming from those two libraries is properly seeded.
If you rely on any other library that you want to seed globally you should do
so manually first thing inside your main function. For this you can either take
the argument seed
(the root-seed), or _seed
(a seed generated for this
call of the main function). In this case it doesn’t really matter.
Special Arguments¶
To generate random numbers that are controlled by the root-seed Sacred provides
two special arguments: _rnd
and _seed
.
You can just accept them as a parameters in any captured function:
@ex.capture
def do_random_stuff(_rnd, _seed):
print(_seed)
print(_rnd.randint(1, 100))
_seed
is an integer that is different every time the function is called.
Likewise _rnd
is a PRNG that you can directly use to generate random numbers.
Note
If numpy
is installed _rnd
will either be a numpy.random.Generator object or a numpy.random.RandomState object. Default behavior is dependen on the numpy version, i. e. with version v1.19 numpy.random.RandomState is marked as legacy. To use the legacy numpy random API regardless of the numpy version set NUMPY_RANDOM_LEGACY_API to True.
Otherwise it will be random.Random object.
All _seed
and _rnd
instances depend deterministically on the root-seed
so they can be controlled centrally.
Resilience to Change¶
The way Sacred generates these seeds and PRNGs actually offers some amount of
resilience to changes in your experiment or your program flow. So suppose for
example you have an experiment that has two methods that use randomness:
A
and B
. You want to run and compare two variants of that experiment:
- Only call
B
.- First call
A
and thenB
.
If you use just a single global PRNG that would mean that for a fixed seed the
call to B
gives different results for the two variants, because the call to
A
changed the state of the global PRNG.
Sacred generates these seeds and PRNGS in a hierarchical way. That makes the
calls to A
and B
independent from one another. So B
would give the
same results in both cases.