Experiment Overview¶

Experiment is the central class of the Sacred framework. This section provides an overview of what it does and how to use it.

Create an Experiment¶

To create an Experiment just instantiate it and add main method:

from sacred import Experiment
ex = Experiment()

@ex.main
def my_main():
    pass

The function decorated with @ex.main is the main function of the experiment. It is executed if you run the experiment and it is also used to determine the source-file of the experiment.

Instead of @ex.main it is recommended to use @ex.automain. This will automatically run the experiment if you execute the file. It is equivalent to the following:

from sacred import Experiment
ex = Experiment()

@ex.main
def my_main():
    pass

if __name__ == '__main__':
    ex.run_commandline()

Note

For this to work the automain function needs to be at the end of the file. Otherwise everything below it is not defined yet when the experiment is run.

Run the Experiment¶

The easiest way to run your experiment is to just use the command-line. This requires that you used automain (or an equivalent). You can then just execute the experiments python file and use the powerful Command-Line Interface.

You can also run your experiment directly from python. This is especially useful if you want to run it multiple times with different configurations. So lets say your experiment is in a file called my_experiment.py. Then you can import it from there and run it like this:

from my_experiment import ex

r = ex.run()

Warning

By default, Sacred experiments will fail if run in an interactive environment like a REPL or a Jupyter Notebook. This is an intended security measure since in these environments reproducibility cannot be ensured. If needed, this safeguard can be deactivated by passing interactive=True to the experiment like this:

ex = Experiment('jupyter_ex', interactive=True)

The run function accepts config_updates to specify how the configuration should be changed for this run. It should be a (possibly nested) dictionary containing all the values that you wish to update. For more information see Configuration:

from my_experiment import ex

r = ex.run(config_updates={'foo': 23})

Note

Under the hood a Run object is created every time you run an Experiment (this is also the object that ex.run() returns). It holds some information about that run (e.g. final configuration and later the result) and is responsible for emitting all the events for the Observing an Experiment.

While the experiment is running you can access it by accepting the special _run argument in any of your Captured Functions. That is also used for Saving Custom Information.

Configuration¶

There are multiple ways of adding configuration to your experiment. The easiest way is through Config Scopes:

@ex.config
def my_config():
    foo = 42
    bar = 'baz'

The local variables from that function are collected and form the configuration of your experiment. You have full access to the power of python when defining the configuration that way. The parameters can even depend on each other.

Note

Only variables that are JSON serializable (i.e. a numbers, strings, lists, tuples, dictionaries) become part of the configuration. Other variables are ignored.

If you think that is too much magic going on, you can always use a plain dictionary to add configuration or, if you prefer, you can also directly load configuration entries from a file.

And of course you can combine all of them and even have several of each kind. They will be executed in the order that you added them, and possibly overwrite each others values.

Capture Functions¶

To use a configuration value all you have to do is capture a function and accept the configuration value as a parameter. Whenever you now call that function Sacred will try to fill in missing parameters from the configuration. To see how that works we need to capture some function:

from sacred import Experiment
ex = Experiment('my_experiment')

@ex.config
def my_config():
    foo = 42
    bar = 'baz'

@ex.capture
def some_function(a, foo, bar=10):
    print(a, foo, bar)

@ex.main
def my_main():
    some_function(1, 2, 3)     #  1  2   3
    some_function(1)           #  1  42  'baz'
    some_function(1, bar=12)   #  1  42  12
    some_function()            #  TypeError: missing value for 'a'

More on this in the Captured Functions Section.

Note

Configuration values are preferred over default values. So in the example above, bar=10 is never used because there is a value of bar = 'baz' in the configuration.

Observe an Experiment¶

Experiments in Sacred collect lots of information about their runs like:

time it was started and time it stopped

the used configuration

the result or any errors that occurred

basic information about the machine it runs on

packages the experiment depends on and their versions

all imported local source-files

files opened with ex.open_resource

files added with ex.add_artifact

To access this information you can use the observer interface. First you need to add an observer like this:

from sacred.observers import MongoObserver

ex.observers.append(MongoObserver())

MongoObserver is one of the default observers shipped with Sacred. It connects to a MongoDB and puts all these information into a document in a collection called experiments. You can also add this observer from the Command-Line Interface like this:

>> python my_experiment.py -m my_database

For more information see Observing an Experiment

Capturing stdout / stderr¶

Sacred tries to capture all outputs and transmits that information to the observers. This behaviour is configurable and can happen in three different modes: no, sys and fd. This mode can be set from the commandline or in the Settings.

In the no mode none of the outputs are captured. This is the default behaviour if no observers are added to the experiment.

If the capture mode is set to sys then sacred captures all outputs written to sys.stdout and sys.stderr such as print statements, stacktraces and logging. In this mode outputs by system-calls, C-extensions or subprocesses are likely not captured. This behaviour is default for Windows.

Finally, the fd mode captures outputs on the file descriptor level, and should include all outputs made by the program or any child-processes. This is the default behaviour for Linux and OSX.

The captured output contains all printed characters and behaves like a file and not like a terminal. Sometimes this is unwanted, for example when the output contains lots of live-updates like progressbars. To prevent the captured out from retaining each and every update that is written to the console one can add a captured out filter to the experiment like this:

from sacred.utils import apply_backspaces_and_linefeeds

ex.captured_out_filter = apply_backspaces_and_linefeeds

Here apply_backspaces_and_linefeeds is a simple function that interprets all backspace and linefeed characters like in a terminal and returns the modified text. Any function that takes a string as input and outputs a (modified) string can be used as a captured_out_filter. For a simple example see examples/captured_out_filter.py.

Interrupted and Failed Experiments¶

If a run is interrupted (e.g. Ctrl+C) or if an exception occurs, Sacred will gather the stacktrace and the fail time and report them to the observers. The resulting entries will have their status set to INTERRUPTED or to FAILED. This allows to quickly see the reason for a non-successful run, and enables later investigation of the errors.

Detecting Hard Failures¶

Sometimes an experiment can fail without an exception being thrown (e.g. power loss, kernel panic, …). In that case the failure cannot be logged to the database and their status will still be RUNNING. Runs that fail in that way are most easily detected by investigating their heartbeat time: each running experiment reports to its observers in regular intervals (default every 10 sec) and updates the heartbeat time along with the captured stdout and the info dict (see Saving Custom Information). So if the heartbeat time lies much further back in time than that interval, the run can be considered dead.

Debugging¶

If an Exception occurs, sacred by default filters the stacktrace by removing all sacred-internal calls. The stacktrace is of course also saved in the database (if appropriate observer is added). This helps to quickly spot errors in your own code. However, if you want to use a debugger, stacktrace filtering needs to be disabled, because it doesn’t play well with debuggers like pdb.

If you want to use a debugger with your experiment, you have two options:

Disable Stacktrace Filtering¶

Stacktrace filtering can be deactivated via the -d flag. Sacred then does not interfere with the exception and it can be properly handled by any debugger.

Post-Mortem Debugging¶

For convenience Sacred also supports directly attaching a post-mortem pdb debugger via the -D flag. If this option is set and an exception occurs, sacred will automatically start pdb debugger to investigate the error, and interact with the stack.

Custom Interrupts¶

Sometimes it can be useful to have custom reasons for interrupting an experiment. One example is if there is a limited time budget for an experiment. If the experiment is stopped because of exceeding that limit, that should be reflected in the database entries.

For these cases, Sacred offers a special base exception sacred.utils.SacredInterrupt that can be used to provide a custom status code. If an exception derived from this one is raised, then the status of the interrupted run will be set to that code.

For the aforementioned timeout usecase there is the sacred.utils.TimeoutInterrupt exception with the status code TIMEOUT. But any status code can be used by simply creating a custom exception that inherits from sacred.utils.SacredInterrupt and defines a STATUS member like this:

from sacred.utils import SacredInterrupt

class CustomInterrupt(SacredInterrupt)
    STATUS = 'MY_CUSTOM_STATUS'

When this exception is raised during any run, its status is set to MY_CUSTOM_STATUS.

Queuing a Run¶

Sacred also supports queuing runs by passing the Queue flag (-q/--queue). This will not run the experiment, but instead only create a database entry that holds all information needed to start the run. This feature could be useful for having a distributed pool of workers that get configurations from the database and run them. As of yet, however, there is no further support for this workflow.