Many experiments rely on some form of randomness. Controlling this randomness is key to ensure reproducibility of the results. This typically happens by manually seeding the Pseudo Random Number Generator (PRNG). Sacred can help you manage this error-prone procedure.
Sacred auto-generates a seed for each run as part of the configuration (You might have noticed it, when printing the configuration of an experiment). This seed has a different value everytime the experiment is run and is stored as part part of the configuration. You can easily set it by:
>>./experiment.py with seed=123
This root-seed is the central place to control randomness, because internally all other seeds and PRNGs depend on it in a deterministic way.
Upon starting the experiment, sacred automatically sets the global seed of
random and (if installed)
numpy.random (which is with v1.19 mark as legacy),
pytorch.manual_seed to the auto-generated
root-seed of the experiment. This means that even if you don’t take any further
steps, at least the randomness stemming from those two libraries is properly seeded.
If you rely on any other library that you want to seed globally you should do
so manually first thing inside your main function. For this you can either take
seed (the root-seed), or
_seed (a seed generated for this
call of the main function). In this case it doesn’t really matter.
To generate random numbers that are controlled by the root-seed Sacred provides
two special arguments:
You can just accept them as a parameters in any captured function:
@ex.capture def do_random_stuff(_rnd, _seed): print(_seed) print(_rnd.randint(1, 100))
_seed is an integer that is different every time the function is called.
_rnd is a PRNG that you can directly use to generate random numbers.
numpy is installed
_rnd will either be a numpy.random.Generator object or a numpy.random.RandomState object. Default behavior is dependen on the numpy version, i. e. with version v1.19 numpy.random.RandomState is marked as legacy. To use the legacy numpy random API regardless of the numpy version set NUMPY_RANDOM_LEGACY_API to True.
Otherwise it will be random.Random object.
_rnd instances depend deterministically on the root-seed
so they can be controlled centrally.
Resilience to Change¶
The way Sacred generates these seeds and PRNGs actually offers some amount of
resilience to changes in your experiment or your program flow. So suppose for
example you have an experiment that has two methods that use randomness:
B. You want to run and compare two variants of that experiment:
- Only call
- First call
If you use just a single global PRNG that would mean that for a fixed seed the
B gives different results for the two variants, because the call to
A changed the state of the global PRNG.
Sacred generates these seeds and PRNGS in a hierarchical way. That makes the
B independent from one another. So
B would give the
same results in both cases.