Experiment Builder

Create experiment from user options

Functions which build orion.core.worker.experiment.Experiment objects based on user configuration.

The instantiation of an orion.core.worker.experiment.Experiment is not a trivial process when the user request an experiment with specific options. One can easily create a new experiment with Experiment('some_experiment_name'), but the configuration of a _writable_ experiment is less straightforward. This is because there is many sources of configuration and they have a strict hierarchy. From the more global to the more specific, there is:

  1. Global configuration:

Defined by orion.core.DEF_CONFIG_FILES_PATHS. Can be scattered in user file system, defaults could look like:

  • /some/path/to/.virtualenvs/orion/share/orion.core

  • /etc/xdg/xdg-ubuntu/orion.core

  • /home/${USER}/.config/orion.core

Note that most variables have default value even if user do not defined them in global configuration. These are defined in orion.core.__init__.

  1. Oríon specific environment variables:

    Environment variables which can override global configuration

    • Database specific:

      • ORION_DB_NAME

      • ORION_DB_TYPE

      • ORION_DB_ADDRESS

  2. Experiment configuration inside the database

Configuration of the experiment if present in the database. Making this part of the configuration of the experiment makes it possible for the user to execute an experiment by only specifying partial configuration. The rest of the configuration is fetched from the database.

For example, a user could:

  1. Rerun the same experiment

Only providing the name is sufficient to rebuild the entire configuration of the experiment.

  1. Make a modification to an existing experiment

The user can provide the name of the existing experiment and only provide the changes to apply on it. Here is an minimal example where we fully initialize a first experiment with a config file and then branch from it with minimal information.

# Initialize root experiment
orion hunt --init-only --config previous_exeriment.yaml ./userscript -x~'uniform(0, 10)'
# Branch a new experiment
orion hunt -n previous_experiment ./userscript -x~'uniform(0, 100)'
  1. Configuration file

This configuration file is meant to overwrite the configuration coming from the database. If this configuration file was interpreted as part of the global configuration, a user could only modify an experiment using command line arguments.

  1. Command-line arguments

Those are the arguments provided to orion for any method (hunt, insert, etc). It includes the argument to orion itself as well as the user’s script name and its arguments.

class orion.core.io.experiment_builder.ExperimentBuilder(storage: dict | BaseStorageProtocol | None = None, debug: bool = False)[source]

Utility to make new experiments using the same storage object.

Parameters
storage: dict or BaseStorageProtocol, optional

Storage object or storage configuration.

debug: bool, optional.

If True, force using EphemeralDB for the storage. Default: False

Methods

build(name[, version, branching])

Build an experiment object

consolidate_config(name, version, config)

Merge together given configuration with db configuration matching for experiment (name, version)

create_experiment(name, version, mode, space)

Instantiate the experiment and its attribute objects

fetch_config_from_db(name[, version])

Fetch configuration from database

load(name[, version, mode])

Load experiment from database

build(name: str, version: int | None = None, branching: dict | None = None, **config) Experiment[source]

Build an experiment object

If new, space argument must be provided, else all arguments are fetched from the database based on (name, version). If any argument given does not match the corresponding ones in the database for given (name, version), than the version is incremented and the experiment will be a child of the previous version.

Parameters
name: str

Name of the experiment to build

version: int, optional

Version to select. If None, last version will be selected. If version given is larger than largest version available, the largest version will be selected.

space: dict, optional

Optimization space of the algorithm. Should have the form dict(name='<prior>(args)').

algorithm: str or dict, optional

Algorithm used for optimization.

strategy: str or dict, optional

Deprecated and will be remove in v0.4. It should now be set in algorithm configuration directly if it supports it.

max_trials: int, optional

Maximum number of trials before the experiment is considered done.

max_broken: int, optional

Number of broken trials for the experiment to be considered broken.

branching: dict, optional

Arguments to control the branching.

branch_from: str, optional

Name of the experiment to branch from.

manual_resolution: bool, optional

Starts the prompt to resolve manually the conflicts. Defaults to False.

non_monitored_arguments: list of str, optional

Will ignore these arguments while looking for differences. Defaults to [].

ignore_code_changes: bool, optional

Will ignore code changes while looking for differences. Defaults to False.

algorithm_change: bool, optional

Whether to automatically solve the algorithm conflict (change of algo config). Defaults to True.

orion_version_change: bool, optional

Whether to automatically solve the orion version conflict. Defaults to True.

code_change_type: str, optional

How to resolve code change automatically. Must be one of ‘noeffect’, ‘unsure’ or ‘break’. Defaults to ‘break’.

cli_change_type: str, optional

How to resolve cli change automatically. Must be one of ‘noeffect’, ‘unsure’ or ‘break’. Defaults to ‘break’.

config_change_type: str, optional

How to resolve config change automatically. Must be one of ‘noeffect’, ‘unsure’ or ‘break’. Defaults to ‘break’.

consolidate_config(name: str, version: int | None, config: dict)[source]

Merge together given configuration with db configuration matching for experiment (name, version)

create_experiment(name: str, version: int, mode: Mode, space: Space | dict[str, str], algorithm: str | dict | None = None, algorithms: str | dict | None = None, max_trials: int | None = None, max_broken: int | None = None, working_dir: str | None = None, metadata: dict | None = None, refers: dict | None = None, producer: dict | None = None, knowledge_base: KnowledgeBase | dict | None = None, user: str | None = None, _id: int | str | None = None, **kwargs) Experiment[source]

Instantiate the experiment and its attribute objects

All unspecified arguments will be replaced by system’s defaults (orion.core.config.*).

Parameters
name: str

Name of the experiment.

version: int

Version of the experiment.

mode: str

The access rights of the experiment on the database. ‘r’: read access only ‘w’: can read and write to database ‘x’: can read and write to database, algo is instantiated and can execute optimization

space: dict or Space object

Optimization space of the algorithm. If dict, should have the form dict(name=’<prior>(args)’).

algorithm: str or dict, optional

Algorithm used for optimization.

strategy: str or dict, optional

Parallel strategy to use to parallelize the algorithm.

max_trials: int, optional

Maximum number or trials before the experiment is considered done.

max_broken: int, optional

Number of broken trials for the experiment to be considered broken.

storage: dict, optional

Configuration of the storage backend.

knowledge_base: KnowledgeBase | dict, optional

Knowledge base instance, or configuration of the knowledge base. Will be used to warm-start the HPO algorithm, if possible.

fetch_config_from_db(name: str, version: int | None = None)[source]

Fetch configuration from database

Parameters
name: str

Name of the experiment to fetch

version: int, optional

Version to select. If None, last version will be selected. If version given is larger than largest version available, the largest version will be selected.

load(name: str, version: int | None = None, mode: Mode = 'r')[source]

Load experiment from database

An experiment view provides all reading operations of standard experiment but prevents the modification of the experiment and its trials.

Parameters
name: str

Name of the experiment to build

version: int, optional

Version to select. If None, last version will be selected. If version given is larger than largest version available, the largest version will be selected.

mode: str, optional

The access rights of the experiment on the database. ‘r’: read access only ‘w’: can read and write to database Default is ‘r’

orion.core.io.experiment_builder.build(name: str, version: int | None = None, branching: dict | None = None, storage: BaseStorageProtocol | dict | None = None, **config)[source]

Build an experiment.

See also

orion.core.io.experiment_builder.Experiment.build() for more information

orion.core.io.experiment_builder.build_from_args(cmdargs)[source]

Build an experiment based on commandline arguments.

Options provided in commandline and configuration file (–config) will overwrite system’s default values and configuration from database if experiment already exits. Commandline arguments have precedence over configuration file options.

See also

orion.core.io.experiment_builder.build() for more information on experiment creation.

orion.core.io.experiment_builder.clean_config(name: str, config: dict, branching: dict | None)[source]

Clean configuration from hidden fields (ex: _id) and update branching if necessary

orion.core.io.experiment_builder.get_cmd_config(cmdargs) ExperimentConfig[source]

Fetch configuration defined by commandline and local configuration file.

Arguments of commandline have priority over options in configuration file.

orion.core.io.experiment_builder.get_from_args(cmdargs, mode='r')[source]

Build an experiment view based on commandline arguments

See also

orion.core.io.experiment_builder.load() for more information on creation of read-only experiments.

orion.core.io.experiment_builder.load(name, version=None, mode='r', storage=None)[source]

Load an experiment.

See also

orion.core.io.experiment_builder.Experiment.load() for more information

orion.core.io.experiment_builder.merge_algorithm_config(config: dict, new_config: dict) None[source]

Merge given algorithm configuration with db config

orion.core.io.experiment_builder.merge_producer_config(config: dict, new_config: dict) None[source]

Merge given producer configuration with db config