Command-Line Interface

Sacred provides a powerful command-line interface for every experiment out of box. All you have to do to use it is to either have a method decorated with @ex.automain or to put this block at the end of your file:

if __name__ == '__main__':
    ex.run_commandline()

Configuration Updates

You can easily change any configuration entry using the powerful with argument on the command-line. Just put with config=update after your experiment call like this:

>>> ./example.py with 'a=10'

Or even multiple values just separated by a space:

>>> ./example.py with 'a=2.3' 'b="FooBar"' 'c=True'

Note

The single quotes (') around each statement are to make sure the bash does not interfere. In simple cases you can omit them:

>>> ./example.py with a=-1 b=2.0 c=True

But be careful especially with strings, because the outermost quotes get removed by bash. So for example all of the following values will be int:

>>> ./example.py with a=1 b="2" c='3'

You can use the standard python literal syntax to set numbers, bools, lists, dicts, strings and combinations thereof:

>>> ./example.py with 'my_list=[1, 2, 3]'
>>> ./example.py with 'nested_list=[["a", "b"], [2, 3], False]'
>>> ./example.py with 'my_dict={"a":1, "b":[-.2, "two"]}'
>>> ./example.py with 'alpha=-.3e-7'
>>> ./example.py with 'mask=0b111000'
>>> ./example.py with 'message="Hello Bob!"'

Note

Note however, that changing individual elements of a list is not supported now.

Dotted Notation

If you want to set individual entries of a dictionary you can use the dotted notation to do so. So if this is the ConfigScope of our experiment:

@ex.config
def cfg():
    d = {
        "foo": 1,
        "bar": 2,
    }

Then we could just change the "foo" entry of our dictionary to 100 like this:

>>> ./example.py with 'd.foo=100'

Named Updates

If there are any Named Configurations set up for an experiment, then you can apply them using the with argument. So for this experiment:

ex = Experiment('named_configs_demo')

@ex.config
def cfg():
    a = 10
    b = 3 * a
    c = "foo"

@ex.named_config
def variant1():
    a = 100
    c = "bar"

The named configuration variant1 can be applied like this:

>>> ./named_configs_demo.py with variant1

Multiple Named Updates

You can have multiple named configurations, and you can use as many of them as you like for any given run. But notice that the order in which you include them matters: The ones you put first will be evaluated first and the values they set might be overwritten by further named configurations.

Combination With Regular Updates

If you combine named updates with regular updates, and the latter have precedence. Sacred will first set an fix all regular updates and then run through all named updates in order, while keeping the regular updates fixed. The resulting configuration is then kept fixed and sacred runs through all normal configurations.

The following will set a=23 first and then execute variant1 treating a as fixed:

>>> ./named_configs_demo.py with variant1 'a=23'

So this configuration becomes {'a':23, 'b':69, 'c':"bar"}.

Config Files As Named Updates

Config files can be used as named updates, by just passing their name to the with argument. So assuming there is a variant2.json this works:

>>> ./named_configs_demo.py with variant2.json

Supported formats are the same as with Config Files.

If there should ever be a name-collision between a named config and a config file the latter takes precedence.

Commands

Apart from running the main function (the default command), the command-line interface also supports other (built-in or custom) commands. The name of the command has to be first on the commandline:

>>> ./my_demo.py COMMAND_NAME with seed=123

If the COMMAND_NAME is ommitted it defaults to the main function, but the name of that function can also explicitly used as the name of the command. So for this experiment

@ex.automain
def my_main():
    return 42

the following two lines are equivalent:

>>> ./my_demo.py with seed=123
>>> ./my_demo.py my_main with seed=123

Save Configuration

Use the save_config command for saving the current/updated configuration into a file:

./03_hello_config_scope.py save_config with recipient=Bob

This will store a file called config.json with the following content:

{
  "message": "Hello Bob!",
  "recipient": "Bob",
  "seed": 151625947
}

The filename can be configured by setting config_filename like this:

./03_hello_config_scope.py save_config with recipient=Bob config_filename=mine.yaml

The format for exporting the config is inferred from the filename and can be any format supported for config files.

Custom Commands

If you just run an experiment file it will execute the default command, that is the method you decorated with @ex.main or @ex.automain. But you can also add other commands to the experiment by using @ex.command:

from sacred import Experiment

ex = Experiment('custom_command')

@ex.command
def scream():
    """
    scream, and shout, and let it all out ...
    """
    print('AAAaaaaaaaahhhhhh...')

# ...

This command can then be run like this:

>> ./custom_command.py scream
INFO - custom_command - Running command 'scream'
INFO - custom_command - Started
AAAaaaaaaaahhhhhh...
INFO - custom_command - Completed after 0:00:00

It will also show up in the usage message and you can get the signature and the docstring by passing it to help:

>> ./custom_command.py help scream

scream()
    scream, and shout, and let it all out ...

Commands are of course also captured functions, so you can take arguments that will get filled in from the config, and you can use with config=update to change parameters from the command-line:

@ex.command
def greet(name):
    """
    Print a simple greet message.
    """
    print('Hello %s!' % name)

And call it like this:

>> ./custom_command.py greet with 'name="Bob"'
INFO - custom_command - Running command 'scream'
INFO - custom_command - Started
Hello Bob!
INFO - custom_command - Completed after 0:00:00

Like other Captured Functions, commands also accept the prefix keyword-argument.

Many commands like print_config are helper functions, and should not trigger observers. This can be accomplished by passing unobserved=True to the decorator:

@ex.command(unobserved=True)
def helper(name):
    print('Running this command will not result in a DB entry!')

Flags

Help

-h print usage
--help

This prints a help/usage message for your experiment. It is equivalent to typing just help.

Comment

-c COMMENT add a comment to this run
--comment COMMENT

The COMMENT can be any text and will be stored with the run.

Logging Level

-l LEVEL control the logging level
--loglevel=LEVEL

With this flag you can adjust the logging level.

Level Numeric value
CRITICAL 50
ERROR 40
WARNING 30
INFO 20
DEBUG 10
NOTSET 0

See Adjusting Log-Levels for more details.

MongoDB Observer

-m DB add a MongoDB observer
--mongo_db=DB

This flag can be used to add a MongoDB observer to your experiment. DB must be of the form [host:port:]db_name[.collection][!priority].

See Mongo Observer for more details.

FileStorage Observer

-F BASEDIR add a file storage observer
--file_storage=BASEDIR

This flag can be used to add a file-storage observer to your experiment. BASEDIR specifies the directory the observer will use to store its files.

See File Storage Observer for more details.

TinyDB Observer

-t BASEDIR add a TinyDB observer
--tiny_db=BASEDIR

This flag can be used to add a TinyDB observer to your experiment. BASEDIR specifies the directory the observer will use to store its files.

See TinyDB Observer for more details.

Note

For this flag to work you need to have the tinydb, tinydb-serialization, and hashfs packages installed.

SQL Observer

-s DB_URL add a SQL observer
--sql=DB_URL

This flag can be used to add a SQL observer to your experiment. DB_URL must be parseable by the sqlalchemy package, which is typically means being of the form dialect://username:password@host:port/database (see their documentation for more detail).

Note

For this flag to work you need to have the sqlalchemy package installed.

See Mongo Observer for more details.

Debug Mode

-d don’t filter the stacktrace
--debug

This flag deactivates the stacktrace filtering. You should usually not need this. It is mainly used for debugging experiments using a debugger (see Debugging).

PDB Debugging

-D Enter post-mortem debugging with pdb on failure.
--pdb

If this flag is set and an exception occurs, sacred automatically starts a pdb post-mortem debugger to investigate the error and interact with the stack (see Debugging).

Beat Interval

-b BEAT_INTERVAL set the interval between heartbeat events
--beat_interval=BEAT_INTERVAL

A running experiment regularly fires a Heartbeat event to synchronize the info dict (see Saving Custom Information). This flag can be used to change the interval from 10 sec (default) to BEAT_INTERVAL sec.

Unobserved

-u Ignore all observers for this run.
--unobserved

If this flag is set, sacred will remove all observers from the current run and also silence the warning for having no observers. This is useful for some quick tests or debugging runs.

Queue

-q Only queue this run, do not start it.
--queue

Instead of running the experiment, this will only create an entry in the database (or where the observers put it) with the status QUEUED. This entry will contain all the information about the experiment and the configuration. But the experiment will not be run. This can be useful to have some distributed workers fetch and start the queued up runs.

Priority

-p Only queue this run, do not start it.
--queue

Enforce Clean

-e Fail if any version control repository is dirty.
--enforce_clean

This flag can be used to enforce that experiments are only being run on a clean repository, i.e. with no uncommitted changes.

Note

For this flag to work you need to have the GitPython package installed.

Print Config

-P PRIORITY Always print the config first.
--priority=PRIORITY

If this flag is set, sacred will always print the current configuration including modifications (like the Print Config command) before running the main method.

Name

-n NAME Set the name for this run.
--name=NAME

This option changes the name of the experiment before starting the run.

Capture Mode

-C CAPTURE_MODE Control the way stdout and stderr are captured.
--capture=CAPTURE_MODE

This option controls how sacred captures outputs to stdout and stderr. Possible values for CAPTURE_MODE are no, sys (default under Windows), or fd (default for Linux/OSX). For more information see here.

Custom Flags

It is possible to add custom flags to an experiment by inheriting from sacred.commandline_option.CommandLineOption like this:

from sacred.commandline_option import CommandLineOption

class OwnFlag(CommandLineOption):
""" This is my personal flag """

    @classmethod
    def apply(cls, args, run):
        # useless feature: add some string to the info dict
        run.info['some'] = 'prepopulation of the info dict'

The name of the flag is taken from the class name and here would be -o/-own_flag. The short flag can be customized by setting a short_flag class variable. The documentation for the flag is taken from the docstring. The apply method of that class is called after the Run object has been created, but before it has been started.

In this case the args parameter will be always be True. But it is also possible to add a flag which takes an argument, by specifying the arg and arg_description class variables:

from sacred.commandline_option import CommandLineOption

class ImprovedFlag(CommandLineOption):
""" This is my even better personal flag """

    short_flag = 'q'
    arg = 'MESSAGE'
    arg_description = 'The cool message that gets saved to info'

    @classmethod
    def apply(cls, args, run):
        run.info['some'] = args

Here the flag would be -q MESSAGE / -improved_flag=MESSAGE and the args parameter of the apply function would contain the MESSAGE as a string.