API Documentation¶
This is a construction site…
Experiment¶
Note
Experiment inherits from Ingredient, so all methods from there also available in the Experiment.
-
class
sacred.
Experiment
(name: Optional[str] = None, ingredients: Sequence[sacred.ingredient.Ingredient] = (), interactive: bool = False, base_dir: Union[str, bytes, pathlib.Path, None] = None, additional_host_info: Optional[List[sacred.host_info.HostInfoGetter]] = None, additional_cli_options: Optional[Sequence[sacred.commandline_options.CLIOption]] = None, save_git_info: bool = True)¶ The central class for each experiment in Sacred.
It manages the configuration, the main function, captured methods, observers, commands, and further ingredients.
An Experiment instance should be created as one of the first things in any experiment-file.
-
__init__
(name: Optional[str] = None, ingredients: Sequence[sacred.ingredient.Ingredient] = (), interactive: bool = False, base_dir: Union[str, bytes, pathlib.Path, None] = None, additional_host_info: Optional[List[sacred.host_info.HostInfoGetter]] = None, additional_cli_options: Optional[Sequence[sacred.commandline_options.CLIOption]] = None, save_git_info: bool = True)¶ Create a new experiment with the given name and optional ingredients.
Parameters: - name – Optional name of this experiment, defaults to the filename. (Required in interactive mode)
- ingredients (list[sacred.Ingredient], optional) – A list of ingredients to be used with this experiment.
- interactive – If set to True will allow the experiment to be run in interactive mode (e.g. IPython or Jupyter notebooks). However, this mode is discouraged since it won’t allow storing the source-code or reliable reproduction of the runs.
- base_dir – Optional full path to the base directory of this experiment. This will set the scope for automatic source file discovery.
- additional_host_info – Optional dictionary containing as keys the names of the pieces of host info you want to collect, and as values the functions collecting those pieces of information.
- save_git_info – Optionally save the git commit hash and the git state (clean or dirty) for all source files. This requires the GitPython package.
-
add_artifact
(filename: Union[str, bytes, pathlib.Path], name: Optional[str] = None, metadata: Optional[dict] = None, content_type: Optional[str] = None) → None¶ Add a file as an artifact.
In Sacred terminology an artifact is a file produced by the experiment run. In case of a MongoObserver that means storing the file in the database.
This function can only be called during a run, and just calls the
sacred.run.Run.add_artifact()
method.Parameters: - filename – name of the file to be stored as artifact
- name – optionally set the name of the artifact. Defaults to the relative file-path.
- metadata – optionally attach metadata to the artifact. This only has an effect when using the MongoObserver.
- content_type – optionally attach a content-type to the artifact. This only has an effect when using the MongoObserver.
-
add_config
(cfg_or_file=None, **kw_conf)¶ Add a configuration entry to this ingredient/experiment.
Can be called with a filename, a dictionary xor with keyword arguments. Supported formats for the config-file so far are:
json
,pickle
andyaml
.- The resulting dictionary will be converted into a
ConfigDict
.
Parameters: - cfg_or_file (dict or str) – Configuration dictionary of filename of config file to add to this ingredient/experiment.
- kw_conf – Configuration entries to be added to this ingredient/experiment.
-
add_named_config
(name, cfg_or_file=None, **kw_conf)¶ Add a named configuration entry to this ingredient/experiment.
Can be called with a filename, a dictionary xor with keyword arguments. Supported formats for the config-file so far are:
json
,pickle
andyaml
.- The resulting dictionary will be converted into a
ConfigDict
.
Parameters: - name (str) – name of the configuration
- cfg_or_file (dict or str) – Configuration dictionary of filename of config file to add to this ingredient/experiment.
- kw_conf – Configuration entries to be added to this ingredient/experiment.
-
add_package_dependency
(package_name, version)¶ Add a package to the list of dependencies.
Parameters: - package_name (str) – The name of the package dependency
- version (str) – The (minimum) version of the package
-
add_resource
(filename: Union[str, bytes, pathlib.Path]) → None¶ Add a file as a resource.
In Sacred terminology a resource is a file that the experiment needed to access during a run. In case of a MongoObserver that means making sure the file is stored in the database (but avoiding duplicates) along its path and md5 sum.
This function can only be called during a run, and just calls the
sacred.run.Run.add_resource()
method.Parameters: filename – name of the file to be stored as a resource
-
add_source_file
(filename)¶ Add a file as source dependency to this experiment/ingredient.
Parameters: filename (str) – filename of the source to be added as dependency
-
automain
(function)¶ Decorator that defines and runs the main function of the experiment.
The decorated function is marked as the default command for this experiment, and the command-line interface is automatically run when the file is executed.
The method decorated by this should be last in the file because is equivalent to:
Example
@ex.main def my_main(): pass if __name__ == '__main__': ex.run_commandline()
-
capture
(function=None, prefix=None)¶ Decorator to turn a function into a captured function.
The missing arguments of captured functions are automatically filled from the configuration if possible. See Captured Functions for more information.
If a
prefix
is specified, the search for suitable entries is performed in the corresponding subtree of the configuration.
-
captured_out_filter
= None¶ Filter function to be applied to captured output of a run
-
command
(function=None, prefix=None, unobserved=False)¶ Decorator to define a new command for this Ingredient or Experiment.
The name of the command will be the name of the function. It can be called from the command-line or by using the run_command function.
Commands are automatically also captured functions.
The command can be given a prefix, to restrict its configuration space to a subtree. (see
capture
for more information)A command can be made unobserved (i.e. ignoring all observers) by passing the unobserved=True keyword argument.
-
config
(function)¶ Decorator to add a function to the configuration of the Experiment.
The decorated function is turned into a
ConfigScope
and added to the Ingredient/Experiment.When the experiment is run, this function will also be executed and all json-serializable local variables inside it will end up as entries in the configuration of the experiment.
-
config_hook
(func)¶ Decorator to add a config hook to this ingredient.
Config hooks need to be a function that takes 3 parameters and returns a dictionary: (config, command_name, logger) –> dict
Config hooks are run after the configuration of this Ingredient, but before any further ingredient-configurations are run. The dictionary returned by a config hook is used to update the config updates. Note that they are not restricted to the local namespace of the ingredient.
-
gather_commands
()¶ Collect all commands from this ingredient and its sub-ingredients.
Yields: - cmd_name (str) – The full (dotted) name of the command.
- cmd (function) – The corresponding captured function.
-
gather_named_configs
() → Generator[Tuple[str, Union[sacred.config.config_scope.ConfigScope, sacred.config.config_dict.ConfigDict, str]], None, None]¶ Collect all named configs from this ingredient and its sub-ingredients.
Yields: - config_name – The full (dotted) name of the named config.
- config – The corresponding named config.
-
get_default_options
() → dict¶ Get a dictionary of default options as used with run.
Returns: - A dictionary containing option keys of the form ‘–beat_interval’.
- Their values are boolean if the option is a flag, otherwise None or
- its default value.
-
get_experiment_info
()¶ Get a dictionary with information about this experiment.
- Contains:
- name: the name
- sources: a list of sources (filename, md5)
- dependencies: a list of package dependencies (name, version)
Returns: experiment information Return type: dict
-
get_usage
(program_name=None)¶ Get the commandline usage string for this experiment.
-
info
¶ Access the info-dict for storing custom information.
Only works during a run and is essentially a shortcut to:
Example
@ex.capture def my_captured_function(_run): # [...] _run.info # == ex.info
-
log_scalar
(name: str, value: float, step: Optional[int] = None) → None¶ Add a new measurement.
The measurement will be processed by the MongoDB* observer during a heartbeat event. Other observers are not yet supported.
Parameters: - name – The name of the metric, e.g. training.loss
- value – The measured value
- step – The step number (integer), e.g. the iteration number If not specified, an internal counter for each metric is used, incremented by one.
-
main
(function)¶ Decorator to define the main function of the experiment.
The main function of an experiment is the default command that is being run when no command is specified, or when calling the run() method.
Usually it is more convenient to use
automain
instead.
-
named_config
(func)¶ Decorator to turn a function into a named configuration.
See Named Configurations.
-
open_resource
(filename: Union[str, bytes, pathlib.Path], mode: str = 'r')¶ Open a file and also save it as a resource.
Opens a file, reports it to the observers as a resource, and returns the opened file.
In Sacred terminology a resource is a file that the experiment needed to access during a run. In case of a MongoObserver that means making sure the file is stored in the database (but avoiding duplicates) along its path and md5 sum.
This function can only be called during a run, and just calls the
sacred.run.Run.open_resource()
method.Parameters: - filename – name of the file that should be opened
- mode – mode that file will be open
Returns: The opened file-object.
-
option_hook
(function)¶ Decorator for adding an option hook function.
An option hook is a function that is called right before a run is created. It receives (and potentially modifies) the options dictionary. That is, the dictionary of commandline options used for this run.
Notes
The decorated function MUST have an argument called options.
The options also contain
'COMMAND'
and'UPDATE'
entries, but changing them has no effect. Only modification on flags (entries starting with'--'
) are considered.
-
post_process_name
(name, ingredient)¶ Can be overridden to change the command name.
-
post_run_hook
(func, prefix=None)¶ Decorator to add a post-run hook to this ingredient.
Post-run hooks are captured functions that are run, just after the main function is executed.
-
pre_run_hook
(func, prefix=None)¶ Decorator to add a pre-run hook to this ingredient.
Pre-run hooks are captured functions that are run, just before the main function is executed.
-
run
(command_name: Optional[str] = None, config_updates: Optional[dict] = None, named_configs: Sequence[str] = (), info: Optional[dict] = None, meta_info: Optional[dict] = None, options: Optional[dict] = None) → sacred.run.Run¶ Run the main function of the experiment or a given command.
Parameters: - command_name – Name of the command to be run. Defaults to main function.
- config_updates – Changes to the configuration as a nested dictionary
- named_configs – list of names of named_configs to use
- info – Additional information for this run.
- meta_info – Additional meta information for this run.
- options – Dictionary of options to use
Returns: The Run object corresponding to the finished run.
-
run_commandline
(argv=None) → Optional[sacred.run.Run]¶ Run the command-line interface of this experiment.
If
argv
is omitted it defaults tosys.argv
.Parameters: argv – Command-line as string or list of strings like sys.argv
.Returns: The Run object corresponding to the finished run.
-
traverse_ingredients
()¶ Recursively traverse this ingredient and its sub-ingredients.
Yields: - ingredient (sacred.Ingredient) – The ingredient as traversed in preorder.
- depth (int) – The depth of the ingredient starting from 0.
Raises: CircularDependencyError: – If a circular structure among ingredients was detected.
-
Ingredient¶
-
class
sacred.
Ingredient
(path: Union[str, bytes, pathlib.Path], ingredients: Sequence[Ingredient] = (), interactive: bool = False, _caller_globals: Optional[dict] = None, base_dir: Union[str, bytes, pathlib.Path, None] = None, save_git_info: bool = True)¶ Ingredients are reusable parts of experiments.
Each Ingredient can have its own configuration (visible as an entry in the parents configuration), named configurations, captured functions and commands.
Ingredients can themselves use ingredients.
-
__init__
(path: Union[str, bytes, pathlib.Path], ingredients: Sequence[Ingredient] = (), interactive: bool = False, _caller_globals: Optional[dict] = None, base_dir: Union[str, bytes, pathlib.Path, None] = None, save_git_info: bool = True)¶ Initialize self. See help(type(self)) for accurate signature.
-
add_config
(cfg_or_file=None, **kw_conf)¶ Add a configuration entry to this ingredient/experiment.
Can be called with a filename, a dictionary xor with keyword arguments. Supported formats for the config-file so far are:
json
,pickle
andyaml
.- The resulting dictionary will be converted into a
ConfigDict
.
Parameters: - cfg_or_file (dict or str) – Configuration dictionary of filename of config file to add to this ingredient/experiment.
- kw_conf – Configuration entries to be added to this ingredient/experiment.
-
add_named_config
(name, cfg_or_file=None, **kw_conf)¶ Add a named configuration entry to this ingredient/experiment.
Can be called with a filename, a dictionary xor with keyword arguments. Supported formats for the config-file so far are:
json
,pickle
andyaml
.- The resulting dictionary will be converted into a
ConfigDict
.
Parameters: - name (str) – name of the configuration
- cfg_or_file (dict or str) – Configuration dictionary of filename of config file to add to this ingredient/experiment.
- kw_conf – Configuration entries to be added to this ingredient/experiment.
-
add_package_dependency
(package_name, version)¶ Add a package to the list of dependencies.
Parameters: - package_name (str) – The name of the package dependency
- version (str) – The (minimum) version of the package
-
add_source_file
(filename)¶ Add a file as source dependency to this experiment/ingredient.
Parameters: filename (str) – filename of the source to be added as dependency
-
capture
(function=None, prefix=None)¶ Decorator to turn a function into a captured function.
The missing arguments of captured functions are automatically filled from the configuration if possible. See Captured Functions for more information.
If a
prefix
is specified, the search for suitable entries is performed in the corresponding subtree of the configuration.
-
command
(function=None, prefix=None, unobserved=False)¶ Decorator to define a new command for this Ingredient or Experiment.
The name of the command will be the name of the function. It can be called from the command-line or by using the run_command function.
Commands are automatically also captured functions.
The command can be given a prefix, to restrict its configuration space to a subtree. (see
capture
for more information)A command can be made unobserved (i.e. ignoring all observers) by passing the unobserved=True keyword argument.
-
config
(function)¶ Decorator to add a function to the configuration of the Experiment.
The decorated function is turned into a
ConfigScope
and added to the Ingredient/Experiment.When the experiment is run, this function will also be executed and all json-serializable local variables inside it will end up as entries in the configuration of the experiment.
-
config_hook
(func)¶ Decorator to add a config hook to this ingredient.
Config hooks need to be a function that takes 3 parameters and returns a dictionary: (config, command_name, logger) –> dict
Config hooks are run after the configuration of this Ingredient, but before any further ingredient-configurations are run. The dictionary returned by a config hook is used to update the config updates. Note that they are not restricted to the local namespace of the ingredient.
-
gather_commands
()¶ Collect all commands from this ingredient and its sub-ingredients.
Yields: - cmd_name (str) – The full (dotted) name of the command.
- cmd (function) – The corresponding captured function.
-
gather_named_configs
() → Generator[Tuple[str, Union[sacred.config.config_scope.ConfigScope, sacred.config.config_dict.ConfigDict, str]], None, None]¶ Collect all named configs from this ingredient and its sub-ingredients.
Yields: - config_name – The full (dotted) name of the named config.
- config – The corresponding named config.
-
get_experiment_info
()¶ Get a dictionary with information about this experiment.
- Contains:
- name: the name
- sources: a list of sources (filename, md5)
- dependencies: a list of package dependencies (name, version)
Returns: experiment information Return type: dict
-
named_config
(func)¶ Decorator to turn a function into a named configuration.
See Named Configurations.
-
post_process_name
(name, ingredient)¶ Can be overridden to change the command name.
-
post_run_hook
(func, prefix=None)¶ Decorator to add a post-run hook to this ingredient.
Post-run hooks are captured functions that are run, just after the main function is executed.
-
pre_run_hook
(func, prefix=None)¶ Decorator to add a pre-run hook to this ingredient.
Pre-run hooks are captured functions that are run, just before the main function is executed.
-
traverse_ingredients
()¶ Recursively traverse this ingredient and its sub-ingredients.
Yields: - ingredient (sacred.Ingredient) – The ingredient as traversed in preorder.
- depth (int) – The depth of the ingredient starting from 0.
Raises: CircularDependencyError: – If a circular structure among ingredients was detected.
-
The Run Object¶
The Run object can be accessed from python after the run is finished:
run = ex.run()
or during a run using the _run
special value in a
captured function.
-
class
sacred.run.
Run
(config, config_modifications, main_function, observers, root_logger, run_logger, experiment_info, host_info, pre_run_hooks, post_run_hooks, captured_out_filter=None)¶ Represent and manage a single run of an experiment.
-
__call__
(*args)¶ Start this run.
Parameters: *args – parameters passed to the main function Returns: the return value of the main function
-
add_artifact
(filename, name=None, metadata=None, content_type=None)¶ Add a file as an artifact.
In Sacred terminology an artifact is a file produced by the experiment run. In case of a MongoObserver that means storing the file in the database.
See also
sacred.Experiment.add_artifact()
.Parameters: - filename (str) – name of the file to be stored as artifact
- name (str, optional) – optionally set the name of the artifact. Defaults to the filename.
- metadata (dict) – optionally attach metadata to the artifact. This only has an effect when using the MongoObserver.
- content_type (str, optional) – optionally attach a content-type to the artifact. This only has an effect when using the MongoObserver.
-
add_resource
(filename)¶ Add a file as a resource.
In Sacred terminology a resource is a file that the experiment needed to access during a run. In case of a MongoObserver that means making sure the file is stored in the database (but avoiding duplicates) along its path and md5 sum.
See also
sacred.Experiment.add_resource()
.Parameters: filename (str) – name of the file to be stored as a resource
-
beat_interval
= None¶ The time between two heartbeat events measured in seconds
-
capture_mode
= None¶ Determines the way the stdout/stderr are captured
-
captured_out
= None¶ Captured stdout and stderr
-
captured_out_filter
= None¶ Filter function to be applied to captured output
-
config
= None¶ The final configuration used for this run
-
config_modifications
= None¶ A ConfigSummary object with information about config changes
-
debug
= None¶ Determines whether this run is executed in debug mode
-
experiment_info
= None¶ A dictionary with information about the experiment
-
fail_trace
= None¶ A stacktrace, in case the run failed
-
force
= None¶ Disable warnings about suspicious changes
-
host_info
= None¶ A dictionary with information about the host
-
info
= None¶ Custom info dict that will be sent to the observers
-
log_scalar
(metric_name, value, step=None)¶ Add a new measurement.
The measurement will be processed by the MongoDB observer during a heartbeat event. Other observers are not yet supported.
Parameters: - metric_name – The name of the metric, e.g. training.loss
- value – The measured value
- step – The step number (integer), e.g. the iteration number If not specified, an internal counter for each metric is used, incremented by one.
-
main_function
= None¶ The main function that is executed with this run
-
meta_info
= None¶ A custom comment for this run
-
observers
= None¶ A list of all observers that observe this run
-
open_resource
(filename, mode='r')¶ Open a file and also save it as a resource.
Opens a file, reports it to the observers as a resource, and returns the opened file.
In Sacred terminology a resource is a file that the experiment needed to access during a run. In case of a MongoObserver that means making sure the file is stored in the database (but avoiding duplicates) along its path and md5 sum.
See also
sacred.Experiment.open_resource()
.Parameters: - filename (str) – name of the file that should be opened
- mode (str) – mode that file will be open
Returns: file – the opened file-object
-
pdb
= None¶ If true the pdb debugger is automatically started after a failure
-
post_run_hooks
= None¶ List of post-run hooks (captured functions called after this run)
-
pre_run_hooks
= None¶ List of pre-run hooks (captured functions called before this run)
-
queue_only
= None¶ If true then this run will only fire the queued_event and quit
-
result
= None¶ The return value of the main function
-
root_logger
= None¶ The root logger that was used to create all the others
-
run_logger
= None¶ The logger that is used for this run
-
start_time
= None¶ The datetime when this run was started
-
status
= None¶ The current status of the run, from QUEUED to COMPLETED
-
stop_time
= None¶ The datetime when this run stopped
-
unobserved
= None¶ Indicates whether this run should be unobserved
-
warn_if_unobserved
()¶
-
Observers¶
-
class
sacred.observers.
RunObserver
¶ Defines the interface for all run observers.
-
artifact_event
(name, filename, metadata=None, content_type=None)¶
-
completed_event
(stop_time, result)¶
-
failed_event
(fail_time, fail_trace)¶
-
heartbeat_event
(info, captured_out, beat_time, result)¶
-
interrupted_event
(interrupt_time, status)¶
-
join
()¶
-
log_metrics
(metrics_by_name, info)¶
-
priority
= 0¶
-
queued_event
(ex_info, command, host_info, queue_time, config, meta_info, _id)¶
-
resource_event
(filename)¶
-
started_event
(ex_info, command, host_info, start_time, config, meta_info, _id)¶
-
-
class
sacred.observers.
MongoObserver
(url: Optional[str] = None, db_name: str = 'sacred', collection: str = 'runs', collection_prefix: str = '', overwrite: Union[int, str, None] = None, priority: int = 30, client: Optional[pymongo.MongoClient] = None, failure_dir: Union[str, bytes, pathlib.Path, None] = None, **kwargs)¶ -
COLLECTION_NAME_BLACKLIST
= {'_properties', 'fs.chunks', 'fs.files', 'search_space', 'search_spaces', 'system.indexes'}¶
-
VERSION
= 'MongoObserver-0.7.0'¶
-
artifact_event
(name, filename, metadata=None, content_type=None)¶
-
completed_event
(stop_time, result)¶
-
classmethod
create
(*args, **kwargs)¶
-
classmethod
create_from
(*args, **kwargs)¶
-
failed_event
(fail_time, fail_trace)¶
-
final_save
(attempts)¶
-
heartbeat_event
(info, captured_out, beat_time, result)¶
-
initialize
(runs_collection, fs, overwrite=None, metrics_collection=None, failure_dir=None, priority=30)¶
-
insert
()¶
-
interrupted_event
(interrupt_time, status)¶
-
log_metrics
(metrics_by_name, info)¶ Store new measurements to the database.
Take measurements and store them into the metrics collection in the database. Additionally, reference the metrics in the info[“metrics”] dictionary.
-
queued_event
(ex_info, command, host_info, queue_time, config, meta_info, _id)¶
-
resource_event
(filename)¶
-
save
()¶
-
save_sources
(ex_info)¶
-
started_event
(ex_info, command, host_info, start_time, config, meta_info, _id)¶
-
Host Info¶
Helps to collect information about the host of an experiment.
-
sacred.host_info.
get_host_info
(additional_host_info: List[sacred.host_info.HostInfoGetter] = None)¶ Collect some information about the machine this experiment runs on.
Returns: dict – A dictionary with information about the CPU, the OS and the Python version of this machine.
-
sacred.host_info.
host_info_getter
(func, name=None)¶ The decorated function is added to the process of collecting the host_info.
This just adds the decorated function to the global
sacred.host_info.host_info_gatherers
dictionary. The functions from that dictionary are used when collecting the host info usingget_host_info()
.Parameters: - func (callable) – A function that can be called without arguments and returns some json-serializable information.
- name (str, optional) – The name of the corresponding entry in host_info. Defaults to the name of the function.
Returns: The function itself.
Custom Exceptions¶
-
class
sacred.utils.
SacredInterrupt
¶ Base-Class for all custom interrupts.
For more information see Custom Interrupts.
-
class
sacred.utils.
TimeoutInterrupt
¶ Signal that the experiment timed out.
This exception can be used in client code to indicate that the run exceeded its time limit and has been interrupted because of that. The status of the interrupted run will then be set to
TIMEOUT
.For more information see Custom Interrupts.