Experiment Builder¶
orion.core.io.experiment_builder
– Create experiment from user options¶
The instantiation of an Experiment is not a trivial process when the user request an experiment with specific options. One can easily create a new experiment with ExperimentView(‘some_experiment_name’), but the configuration of a _writable_ experiment is less straighforward. This is because there is many sources of configuration and they have a strict hierarchy. From the more global to the more specific, there is:
- Global configuration:
Defined by
orion.core.DEF_CONFIG_FILES_PATHS
. Can be scattered in user file system, defaults could look like:
- /some/path/to/.virtualenvs/orion/share/orion.core
- /etc/xdg/xdg-ubuntu/orion.core
- /home/${USER}/.config/orion.core
Note that some variables have default value even if user do not defined them in global configuration:
- max_trials = orion.core.io.resolve_config.DEF_CMD_MAX_TRIALS
- pool_size = orion.core.io.resolve_config.DEF_CMD_POOL_SIZE
- algorithms = random
- Database specific:
- database.name = ‘orion’
- database.type = ‘MongoDB’
- database.host = ${HOST}
Oríon specific environment variables:
Environment variables which can override global configuration
- Database specific:
- ORION_DB_NAME
- ORION_DB_TYPE
- ORION_DB_ADDRESS
- Database specific:
Experiment configuration inside the database
Configuration of the experiment if present in the database. Making this part of the configuration of the experiment makes it possible for the user to execute an experiment by only specifying partial configuration. The rest of the configuration is fetched from the database.
For example, a user could:
- Rerun the same experiment
Only providing the name is sufficient to rebuild the entire configuration of the experiment.
- Make a modification to an existing experiment
The user can provide the name of the existing experiment and only provide the changes to apply on it. Here is an minimal example where we fully initialize a first experiment with a config file and then branch from it with minimal information.
# Initialize root experiment orion hunt --init-only --config previous_exeriment.yaml ./userscript -x~'uniform(0, 10)' # Branch a new experiment orion hunt -n previous_experiment ./userscript -x~'uniform(0, 100)'
- Configuration file
This configuration file is meant to overwrite the configuration coming from the database. If this configuration file was interpreted as part of the global configuration, a user could only modify an experiment using command line arguments.
- Command-line arguments
Those are the arguments provided toorion
for any method (hunt, insert, etc). It includes the argument toorion
itself as well as the user’s script name and its arguments.
-
orion.core.io.experiment_builder.
build
(name, version=None, branching=None, **config)[source]¶ Build an experiment object
If new, space argument must be provided, else all arguments are fetched from the database based on (name, version). If any argument given does not match the corresponding ones in the database for given (name, version), than the version is incremented and the experiment will be a child of the previous version.
Parameters: - name: str
Name of the experiment to build
- version: int, optional
Version to select. If None, last version will be selected. If version given is larger than largest version available, the largest version will be selected.
- space: dict, optional
Optimization space of the algorithm. Should have the form dict(name=’<prior>(args)’).
- algorithms: str or dict, optional
Algorithm used for optimization.
- strategy: str or dict, optional
Parallel strategy to use to parallelize the algorithm.
- max_trials: int, optional
Maximum number or trials before the experiment is considered done.
- storage: dict, optional
Configuration of the storage backend.
- branching: dict, optional
Arguments to control the branching.
- branch_from: str, optional
Name of the experiment to branch from.
- manual_resolution: bool, optional
Starts the prompt to resolve manually the conflicts. Defaults to False.
- non_monitored_arguments: list of str, optional
Will ignore these arguments while looking for differences. Defaults to [].
- ignore_code_changes: bool, optional
Will ignore code changes while looking for differences. Defaults to False.
- algorithm_change: bool, optional
Whether to automatically solve the algorithm conflict (change of algo config). Defaults to True.
- code_change_type: str, optional
How to resolve code change automatically. Must be one of ‘noeffect’, ‘unsure’ or ‘break’. Defaults to ‘break’.
- cli_change_type: str, optional
How to resolve cli change automatically. Must be one of ‘noeffect’, ‘unsure’ or ‘break’. Defaults to ‘break’.
- config_change_type: str, optional
How to resolve config change automatically. Must be one of ‘noeffect’, ‘unsure’ or ‘break’. Defaults to ‘break’.
-
orion.core.io.experiment_builder.
build_from_args
(cmdargs)[source]¶ Build an experiment based on commandline arguments.
Options provided in commandline and configuration file (–config) will overwrite system’s default values and configuration from database if experiment already exits. Commandline arguments have precedence over configuration file options.
See also
orion.core.io.experiment_builder.build()
for more information on experiment creation.
-
orion.core.io.experiment_builder.
build_view
(name, version=None)[source]¶ Build experiment view
An experiment view provides all reading operations of standard experiment but prevents the modification of the experiment and its trials.
Parameters: - name: str
Name of the experiment to build
- version: int, optional
Version to select. If None, last version will be selected. If version given is larger than largest version available, the largest version will be selected.
-
orion.core.io.experiment_builder.
build_view_from_args
(cmdargs)[source]¶ Build an experiment view based on commandline arguments
See also
orion.core.io.experiment_builder.build_view()
for more information on experiment view creation.
-
orion.core.io.experiment_builder.
create_experiment
(name, version, space, **kwargs)[source]¶ Instantiate the experiment and its attribute objects
All unspecified arguments will be replaced by system’s defaults (orion.core.config.*).
Parameters: - name: str
Name of the experiment.
- version: int
Version of the experiment.
- space: dict or Space object
Optimization space of the algorithm. If dict, should have the form dict(name=’<prior>(args)’).
- algorithms: str or dict, optional
Algorithm used for optimization.
- strategy: str or dict, optional
Parallel strategy to use to parallelize the algorithm.
- max_trials: int, optional
Maximum number or trials before the experiment is considered done.
- storage: dict, optional
Configuration of the storage backend.
-
orion.core.io.experiment_builder.
fetch_config_from_db
(name, version=None)[source]¶ Fetch configuration from database
Parameters: - name: str
Name of the experiment to fetch
- version: int, optional
Version to select. If None, last version will be selected. If version given is larger than largest version available, the largest version will be selected.