API Documentation

This is a construction site…

Experiment

Note

Experiment inherits from Ingredient, so all methods from there also available in the Experiment.

class sacred.Experiment(name: Optional[str] = None, ingredients: Sequence[sacred.ingredient.Ingredient] = (), interactive: bool = False, base_dir: Union[str, bytes, pathlib.Path, None] = None, additional_host_info: Optional[List[sacred.host_info.HostInfoGetter]] = None, additional_cli_options: Optional[Sequence[sacred.commandline_options.CLIOption]] = None, save_git_info: bool = True)

The central class for each experiment in Sacred.

It manages the configuration, the main function, captured methods, observers, commands, and further ingredients.

An Experiment instance should be created as one of the first things in any experiment-file.

__init__(name: Optional[str] = None, ingredients: Sequence[sacred.ingredient.Ingredient] = (), interactive: bool = False, base_dir: Union[str, bytes, pathlib.Path, None] = None, additional_host_info: Optional[List[sacred.host_info.HostInfoGetter]] = None, additional_cli_options: Optional[Sequence[sacred.commandline_options.CLIOption]] = None, save_git_info: bool = True)

Create a new experiment with the given name and optional ingredients.

Parameters:
  • name – Optional name of this experiment, defaults to the filename. (Required in interactive mode)
  • ingredients (list[sacred.Ingredient], optional) – A list of ingredients to be used with this experiment.
  • interactive – If set to True will allow the experiment to be run in interactive mode (e.g. IPython or Jupyter notebooks). However, this mode is discouraged since it won’t allow storing the source-code or reliable reproduction of the runs.
  • base_dir – Optional full path to the base directory of this experiment. This will set the scope for automatic source file discovery.
  • additional_host_info – Optional dictionary containing as keys the names of the pieces of host info you want to collect, and as values the functions collecting those pieces of information.
  • save_git_info – Optionally save the git commit hash and the git state (clean or dirty) for all source files. This requires the GitPython package.
add_artifact(filename: Union[str, bytes, pathlib.Path], name: Optional[str] = None, metadata: Optional[dict] = None, content_type: Optional[str] = None) → None

Add a file as an artifact.

In Sacred terminology an artifact is a file produced by the experiment run. In case of a MongoObserver that means storing the file in the database.

This function can only be called during a run, and just calls the sacred.run.Run.add_artifact() method.

Parameters:
  • filename – name of the file to be stored as artifact
  • name – optionally set the name of the artifact. Defaults to the relative file-path.
  • metadata – optionally attach metadata to the artifact. This only has an effect when using the MongoObserver.
  • content_type – optionally attach a content-type to the artifact. This only has an effect when using the MongoObserver.
add_config(cfg_or_file=None, **kw_conf)

Add a configuration entry to this ingredient/experiment.

Can be called with a filename, a dictionary xor with keyword arguments. Supported formats for the config-file so far are: json, pickle and yaml.

The resulting dictionary will be converted into a
ConfigDict.
Parameters:
  • cfg_or_file (dict or str) – Configuration dictionary of filename of config file to add to this ingredient/experiment.
  • kw_conf – Configuration entries to be added to this ingredient/experiment.
add_named_config(name, cfg_or_file=None, **kw_conf)

Add a named configuration entry to this ingredient/experiment.

Can be called with a filename, a dictionary xor with keyword arguments. Supported formats for the config-file so far are: json, pickle and yaml.

The resulting dictionary will be converted into a
ConfigDict.

See Named Configurations

Parameters:
  • name (str) – name of the configuration
  • cfg_or_file (dict or str) – Configuration dictionary of filename of config file to add to this ingredient/experiment.
  • kw_conf – Configuration entries to be added to this ingredient/experiment.
add_package_dependency(package_name, version)

Add a package to the list of dependencies.

Parameters:
  • package_name (str) – The name of the package dependency
  • version (str) – The (minimum) version of the package
add_resource(filename: Union[str, bytes, pathlib.Path]) → None

Add a file as a resource.

In Sacred terminology a resource is a file that the experiment needed to access during a run. In case of a MongoObserver that means making sure the file is stored in the database (but avoiding duplicates) along its path and md5 sum.

This function can only be called during a run, and just calls the sacred.run.Run.add_resource() method.

Parameters:filename – name of the file to be stored as a resource
add_source_file(filename)

Add a file as source dependency to this experiment/ingredient.

Parameters:filename (str) – filename of the source to be added as dependency
automain(function)

Decorator that defines and runs the main function of the experiment.

The decorated function is marked as the default command for this experiment, and the command-line interface is automatically run when the file is executed.

The method decorated by this should be last in the file because is equivalent to:

Example

@ex.main
def my_main():
    pass

if __name__ == '__main__':
    ex.run_commandline()
capture(function=None, prefix=None)

Decorator to turn a function into a captured function.

The missing arguments of captured functions are automatically filled from the configuration if possible. See Captured Functions for more information.

If a prefix is specified, the search for suitable entries is performed in the corresponding subtree of the configuration.

captured_out_filter = None

Filter function to be applied to captured output of a run

command(function=None, prefix=None, unobserved=False)

Decorator to define a new command for this Ingredient or Experiment.

The name of the command will be the name of the function. It can be called from the command-line or by using the run_command function.

Commands are automatically also captured functions.

The command can be given a prefix, to restrict its configuration space to a subtree. (see capture for more information)

A command can be made unobserved (i.e. ignoring all observers) by passing the unobserved=True keyword argument.

config(function)

Decorator to add a function to the configuration of the Experiment.

The decorated function is turned into a ConfigScope and added to the Ingredient/Experiment.

When the experiment is run, this function will also be executed and all json-serializable local variables inside it will end up as entries in the configuration of the experiment.

config_hook(func)

Decorator to add a config hook to this ingredient.

Config hooks need to be a function that takes 3 parameters and returns a dictionary: (config, command_name, logger) –> dict

Config hooks are run after the configuration of this Ingredient, but before any further ingredient-configurations are run. The dictionary returned by a config hook is used to update the config updates. Note that they are not restricted to the local namespace of the ingredient.

gather_commands()

Collect all commands from this ingredient and its sub-ingredients.

Yields:
  • cmd_name (str) – The full (dotted) name of the command.
  • cmd (function) – The corresponding captured function.
gather_named_configs() → Generator[Tuple[str, Union[sacred.config.config_scope.ConfigScope, sacred.config.config_dict.ConfigDict, str]], None, None]

Collect all named configs from this ingredient and its sub-ingredients.

Yields:
  • config_name – The full (dotted) name of the named config.
  • config – The corresponding named config.
get_default_options() → dict

Get a dictionary of default options as used with run.

Returns:
  • A dictionary containing option keys of the form ‘–beat_interval’.
  • Their values are boolean if the option is a flag, otherwise None or
  • its default value.
get_experiment_info()

Get a dictionary with information about this experiment.

Contains:
  • name: the name
  • sources: a list of sources (filename, md5)
  • dependencies: a list of package dependencies (name, version)
Returns:experiment information
Return type:dict
get_usage(program_name=None)

Get the commandline usage string for this experiment.

info

Access the info-dict for storing custom information.

Only works during a run and is essentially a shortcut to:

Example

@ex.capture
def my_captured_function(_run):
    # [...]
    _run.info   # == ex.info
log_scalar(name: str, value: float, step: Optional[int] = None) → None

Add a new measurement.

The measurement will be processed by the MongoDB* observer during a heartbeat event. Other observers are not yet supported.

Parameters:
  • name – The name of the metric, e.g. training.loss
  • value – The measured value
  • step – The step number (integer), e.g. the iteration number If not specified, an internal counter for each metric is used, incremented by one.
main(function)

Decorator to define the main function of the experiment.

The main function of an experiment is the default command that is being run when no command is specified, or when calling the run() method.

Usually it is more convenient to use automain instead.

named_config(func)

Decorator to turn a function into a named configuration.

See Named Configurations.

open_resource(filename: Union[str, bytes, pathlib.Path], mode: str = 'r')

Open a file and also save it as a resource.

Opens a file, reports it to the observers as a resource, and returns the opened file.

In Sacred terminology a resource is a file that the experiment needed to access during a run. In case of a MongoObserver that means making sure the file is stored in the database (but avoiding duplicates) along its path and md5 sum.

This function can only be called during a run, and just calls the sacred.run.Run.open_resource() method.

Parameters:
  • filename – name of the file that should be opened
  • mode – mode that file will be open
Returns:

The opened file-object.

option_hook(function)

Decorator for adding an option hook function.

An option hook is a function that is called right before a run is created. It receives (and potentially modifies) the options dictionary. That is, the dictionary of commandline options used for this run.

Notes

The decorated function MUST have an argument called options.

The options also contain 'COMMAND' and 'UPDATE' entries, but changing them has no effect. Only modification on flags (entries starting with '--') are considered.

post_process_name(name, ingredient)

Can be overridden to change the command name.

post_run_hook(func, prefix=None)

Decorator to add a post-run hook to this ingredient.

Post-run hooks are captured functions that are run, just after the main function is executed.

pre_run_hook(func, prefix=None)

Decorator to add a pre-run hook to this ingredient.

Pre-run hooks are captured functions that are run, just before the main function is executed.

run(command_name: Optional[str] = None, config_updates: Optional[dict] = None, named_configs: Sequence[str] = (), info: Optional[dict] = None, meta_info: Optional[dict] = None, options: Optional[dict] = None) → sacred.run.Run

Run the main function of the experiment or a given command.

Parameters:
  • command_name – Name of the command to be run. Defaults to main function.
  • config_updates – Changes to the configuration as a nested dictionary
  • named_configs – list of names of named_configs to use
  • info – Additional information for this run.
  • meta_info – Additional meta information for this run.
  • options – Dictionary of options to use
Returns:

The Run object corresponding to the finished run.

run_commandline(argv=None) → Optional[sacred.run.Run]

Run the command-line interface of this experiment.

If argv is omitted it defaults to sys.argv.

Parameters:argv – Command-line as string or list of strings like sys.argv.
Returns:The Run object corresponding to the finished run.
traverse_ingredients()

Recursively traverse this ingredient and its sub-ingredients.

Yields:
  • ingredient (sacred.Ingredient) – The ingredient as traversed in preorder.
  • depth (int) – The depth of the ingredient starting from 0.
Raises:

CircularDependencyError: – If a circular structure among ingredients was detected.

Ingredient

class sacred.Ingredient(path: Union[str, bytes, pathlib.Path], ingredients: Sequence[Ingredient] = (), interactive: bool = False, _caller_globals: Optional[dict] = None, base_dir: Union[str, bytes, pathlib.Path, None] = None, save_git_info: bool = True)

Ingredients are reusable parts of experiments.

Each Ingredient can have its own configuration (visible as an entry in the parents configuration), named configurations, captured functions and commands.

Ingredients can themselves use ingredients.

__init__(path: Union[str, bytes, pathlib.Path], ingredients: Sequence[Ingredient] = (), interactive: bool = False, _caller_globals: Optional[dict] = None, base_dir: Union[str, bytes, pathlib.Path, None] = None, save_git_info: bool = True)

Initialize self. See help(type(self)) for accurate signature.

add_config(cfg_or_file=None, **kw_conf)

Add a configuration entry to this ingredient/experiment.

Can be called with a filename, a dictionary xor with keyword arguments. Supported formats for the config-file so far are: json, pickle and yaml.

The resulting dictionary will be converted into a
ConfigDict.
Parameters:
  • cfg_or_file (dict or str) – Configuration dictionary of filename of config file to add to this ingredient/experiment.
  • kw_conf – Configuration entries to be added to this ingredient/experiment.
add_named_config(name, cfg_or_file=None, **kw_conf)

Add a named configuration entry to this ingredient/experiment.

Can be called with a filename, a dictionary xor with keyword arguments. Supported formats for the config-file so far are: json, pickle and yaml.

The resulting dictionary will be converted into a
ConfigDict.

See Named Configurations

Parameters:
  • name (str) – name of the configuration
  • cfg_or_file (dict or str) – Configuration dictionary of filename of config file to add to this ingredient/experiment.
  • kw_conf – Configuration entries to be added to this ingredient/experiment.
add_package_dependency(package_name, version)

Add a package to the list of dependencies.

Parameters:
  • package_name (str) – The name of the package dependency
  • version (str) – The (minimum) version of the package
add_source_file(filename)

Add a file as source dependency to this experiment/ingredient.

Parameters:filename (str) – filename of the source to be added as dependency
capture(function=None, prefix=None)

Decorator to turn a function into a captured function.

The missing arguments of captured functions are automatically filled from the configuration if possible. See Captured Functions for more information.

If a prefix is specified, the search for suitable entries is performed in the corresponding subtree of the configuration.

command(function=None, prefix=None, unobserved=False)

Decorator to define a new command for this Ingredient or Experiment.

The name of the command will be the name of the function. It can be called from the command-line or by using the run_command function.

Commands are automatically also captured functions.

The command can be given a prefix, to restrict its configuration space to a subtree. (see capture for more information)

A command can be made unobserved (i.e. ignoring all observers) by passing the unobserved=True keyword argument.

config(function)

Decorator to add a function to the configuration of the Experiment.

The decorated function is turned into a ConfigScope and added to the Ingredient/Experiment.

When the experiment is run, this function will also be executed and all json-serializable local variables inside it will end up as entries in the configuration of the experiment.

config_hook(func)

Decorator to add a config hook to this ingredient.

Config hooks need to be a function that takes 3 parameters and returns a dictionary: (config, command_name, logger) –> dict

Config hooks are run after the configuration of this Ingredient, but before any further ingredient-configurations are run. The dictionary returned by a config hook is used to update the config updates. Note that they are not restricted to the local namespace of the ingredient.

gather_commands()

Collect all commands from this ingredient and its sub-ingredients.

Yields:
  • cmd_name (str) – The full (dotted) name of the command.
  • cmd (function) – The corresponding captured function.
gather_named_configs() → Generator[Tuple[str, Union[sacred.config.config_scope.ConfigScope, sacred.config.config_dict.ConfigDict, str]], None, None]

Collect all named configs from this ingredient and its sub-ingredients.

Yields:
  • config_name – The full (dotted) name of the named config.
  • config – The corresponding named config.
get_experiment_info()

Get a dictionary with information about this experiment.

Contains:
  • name: the name
  • sources: a list of sources (filename, md5)
  • dependencies: a list of package dependencies (name, version)
Returns:experiment information
Return type:dict
named_config(func)

Decorator to turn a function into a named configuration.

See Named Configurations.

post_process_name(name, ingredient)

Can be overridden to change the command name.

post_run_hook(func, prefix=None)

Decorator to add a post-run hook to this ingredient.

Post-run hooks are captured functions that are run, just after the main function is executed.

pre_run_hook(func, prefix=None)

Decorator to add a pre-run hook to this ingredient.

Pre-run hooks are captured functions that are run, just before the main function is executed.

traverse_ingredients()

Recursively traverse this ingredient and its sub-ingredients.

Yields:
  • ingredient (sacred.Ingredient) – The ingredient as traversed in preorder.
  • depth (int) – The depth of the ingredient starting from 0.
Raises:

CircularDependencyError: – If a circular structure among ingredients was detected.

The Run Object

The Run object can be accessed from python after the run is finished: run = ex.run() or during a run using the _run special value in a captured function.

class sacred.run.Run(config, config_modifications, main_function, observers, root_logger, run_logger, experiment_info, host_info, pre_run_hooks, post_run_hooks, captured_out_filter=None)

Represent and manage a single run of an experiment.

__call__(*args)

Start this run.

Parameters:*args – parameters passed to the main function
Returns:the return value of the main function
add_artifact(filename, name=None, metadata=None, content_type=None)

Add a file as an artifact.

In Sacred terminology an artifact is a file produced by the experiment run. In case of a MongoObserver that means storing the file in the database.

See also sacred.Experiment.add_artifact().

Parameters:
  • filename (str) – name of the file to be stored as artifact
  • name (str, optional) – optionally set the name of the artifact. Defaults to the filename.
  • metadata (dict) – optionally attach metadata to the artifact. This only has an effect when using the MongoObserver.
  • content_type (str, optional) – optionally attach a content-type to the artifact. This only has an effect when using the MongoObserver.
add_resource(filename)

Add a file as a resource.

In Sacred terminology a resource is a file that the experiment needed to access during a run. In case of a MongoObserver that means making sure the file is stored in the database (but avoiding duplicates) along its path and md5 sum.

See also sacred.Experiment.add_resource().

Parameters:filename (str) – name of the file to be stored as a resource
beat_interval = None

The time between two heartbeat events measured in seconds

capture_mode = None

Determines the way the stdout/stderr are captured

captured_out = None

Captured stdout and stderr

captured_out_filter = None

Filter function to be applied to captured output

config = None

The final configuration used for this run

config_modifications = None

A ConfigSummary object with information about config changes

debug = None

Determines whether this run is executed in debug mode

experiment_info = None

A dictionary with information about the experiment

fail_trace = None

A stacktrace, in case the run failed

force = None

Disable warnings about suspicious changes

host_info = None

A dictionary with information about the host

info = None

Custom info dict that will be sent to the observers

log_scalar(metric_name, value, step=None)

Add a new measurement.

The measurement will be processed by the MongoDB observer during a heartbeat event. Other observers are not yet supported.

Parameters:
  • metric_name – The name of the metric, e.g. training.loss
  • value – The measured value
  • step – The step number (integer), e.g. the iteration number If not specified, an internal counter for each metric is used, incremented by one.
main_function = None

The main function that is executed with this run

meta_info = None

A custom comment for this run

observers = None

A list of all observers that observe this run

open_resource(filename, mode='r')

Open a file and also save it as a resource.

Opens a file, reports it to the observers as a resource, and returns the opened file.

In Sacred terminology a resource is a file that the experiment needed to access during a run. In case of a MongoObserver that means making sure the file is stored in the database (but avoiding duplicates) along its path and md5 sum.

See also sacred.Experiment.open_resource().

Parameters:
  • filename (str) – name of the file that should be opened
  • mode (str) – mode that file will be open
Returns:

file – the opened file-object

pdb = None

If true the pdb debugger is automatically started after a failure

post_run_hooks = None

List of post-run hooks (captured functions called after this run)

pre_run_hooks = None

List of pre-run hooks (captured functions called before this run)

queue_only = None

If true then this run will only fire the queued_event and quit

result = None

The return value of the main function

root_logger = None

The root logger that was used to create all the others

run_logger = None

The logger that is used for this run

start_time = None

The datetime when this run was started

status = None

The current status of the run, from QUEUED to COMPLETED

stop_time = None

The datetime when this run stopped

unobserved = None

Indicates whether this run should be unobserved

warn_if_unobserved()

ConfigScope

class sacred.config.config_scope.ConfigScope(func)

ConfigDict

class sacred.config.config_dict.ConfigDict(d)

Observers

class sacred.observers.RunObserver

Defines the interface for all run observers.

artifact_event(name, filename, metadata=None, content_type=None)
completed_event(stop_time, result)
failed_event(fail_time, fail_trace)
heartbeat_event(info, captured_out, beat_time, result)
interrupted_event(interrupt_time, status)
join()
log_metrics(metrics_by_name, info)
priority = 0
queued_event(ex_info, command, host_info, queue_time, config, meta_info, _id)
resource_event(filename)
started_event(ex_info, command, host_info, start_time, config, meta_info, _id)
class sacred.observers.MongoObserver(url: Optional[str] = None, db_name: str = 'sacred', collection: str = 'runs', collection_prefix: str = '', overwrite: Union[int, str, None] = None, priority: int = 30, client: Optional[pymongo.MongoClient] = None, failure_dir: Union[str, bytes, pathlib.Path, None] = None, **kwargs)
COLLECTION_NAME_BLACKLIST = {'_properties', 'fs.chunks', 'fs.files', 'search_space', 'search_spaces', 'system.indexes'}
VERSION = 'MongoObserver-0.7.0'
artifact_event(name, filename, metadata=None, content_type=None)
completed_event(stop_time, result)
classmethod create(*args, **kwargs)
classmethod create_from(*args, **kwargs)
failed_event(fail_time, fail_trace)
final_save(attempts)
heartbeat_event(info, captured_out, beat_time, result)
initialize(runs_collection, fs, overwrite=None, metrics_collection=None, failure_dir=None, priority=30)
insert()
interrupted_event(interrupt_time, status)
log_metrics(metrics_by_name, info)

Store new measurements to the database.

Take measurements and store them into the metrics collection in the database. Additionally, reference the metrics in the info[“metrics”] dictionary.

queued_event(ex_info, command, host_info, queue_time, config, meta_info, _id)
resource_event(filename)
save()
save_sources(ex_info)
started_event(ex_info, command, host_info, start_time, config, meta_info, _id)

Host Info

Helps to collect information about the host of an experiment.

sacred.host_info.get_host_info(additional_host_info: List[sacred.host_info.HostInfoGetter] = None)

Collect some information about the machine this experiment runs on.

Returns:dict – A dictionary with information about the CPU, the OS and the Python version of this machine.
sacred.host_info.host_info_getter(func, name=None)

The decorated function is added to the process of collecting the host_info.

This just adds the decorated function to the global sacred.host_info.host_info_gatherers dictionary. The functions from that dictionary are used when collecting the host info using get_host_info().

Parameters:
  • func (callable) – A function that can be called without arguments and returns some json-serializable information.
  • name (str, optional) – The name of the corresponding entry in host_info. Defaults to the name of the function.
Returns:

The function itself.

Custom Exceptions

class sacred.utils.SacredInterrupt

Base-Class for all custom interrupts.

For more information see Custom Interrupts.

class sacred.utils.TimeoutInterrupt

Signal that the experiment timed out.

This exception can be used in client code to indicate that the run exceeded its time limit and has been interrupted because of that. The status of the interrupted run will then be set to TIMEOUT.

For more information see Custom Interrupts.