Experiment Builder¶
Create experiment from user options¶
Functions which build orion.core.worker.experiment.Experiment
objects based on user
configuration.
The instantiation of an orion.core.worker.experiment.Experiment
is not a trivial process
when the user request an experiment with specific options. One can easily create a new experiment
with Experiment('some_experiment_name')
, but the configuration of a _writable_ experiment is
less straighforward. This is because there is many sources of configuration and they have a strict
hierarchy. From the more global to the more specific, there is:
- Global configuration:
Defined by
orion.core.DEF_CONFIG_FILES_PATHS
. Can be scattered in user file system, defaults could look like:
/some/path/to/.virtualenvs/orion/share/orion.core
/etc/xdg/xdg-ubuntu/orion.core
/home/${USER}/.config/orion.core
Note that most variables have default value even if user do not defined them in global configuration. These are defined in
orion.core.__init__
.
Oríon specific environment variables:
Environment variables which can override global configuration
- Database specific:
ORION_DB_NAME
ORION_DB_TYPE
ORION_DB_ADDRESS
- Database specific:
Experiment configuration inside the database
Configuration of the experiment if present in the database. Making this part of the configuration of the experiment makes it possible for the user to execute an experiment by only specifying partial configuration. The rest of the configuration is fetched from the database.
For example, a user could:
- Rerun the same experiment
Only providing the name is sufficient to rebuild the entire configuration of the experiment.
- Make a modification to an existing experiment
The user can provide the name of the existing experiment and only provide the changes to apply on it. Here is an minimal example where we fully initialize a first experiment with a config file and then branch from it with minimal information.
# Initialize root experiment orion hunt --init-only --config previous_exeriment.yaml ./userscript -x~'uniform(0, 10)' # Branch a new experiment orion hunt -n previous_experiment ./userscript -x~'uniform(0, 100)'
- Configuration file
This configuration file is meant to overwrite the configuration coming from the database. If this configuration file was interpreted as part of the global configuration, a user could only modify an experiment using command line arguments.
- Command-line arguments
Those are the arguments provided toorion
for any method (hunt, insert, etc). It includes the argument toorion
itself as well as the user’s script name and its arguments.
-
orion.core.io.experiment_builder.
build
(name, version=None, branching=None, **config)[source]¶ Build an experiment object
If new,
space
argument must be provided, else all arguments are fetched from the database based on (name, version). If any argument given does not match the corresponding ones in the database for given (name, version), than the version is incremented and the experiment will be a child of the previous version.Parameters: - name: str
Name of the experiment to build
- version: int, optional
Version to select. If None, last version will be selected. If version given is larger than largest version available, the largest version will be selected.
- space: dict, optional
Optimization space of the algorithm. Should have the form
dict(name='<prior>(args)')
.- algorithms: str or dict, optional
Algorithm used for optimization.
- strategy: str or dict, optional
Parallel strategy to use to parallelize the algorithm.
- max_trials: int, optional
Maximum number of trials before the experiment is considered done.
- max_broken: int, optional
Number of broken trials for the experiment to be considered broken.
- storage: dict, optional
Configuration of the storage backend.
- branching: dict, optional
Arguments to control the branching.
- branch_from: str, optional
Name of the experiment to branch from.
- manual_resolution: bool, optional
Starts the prompt to resolve manually the conflicts. Defaults to False.
- non_monitored_arguments: list of str, optional
Will ignore these arguments while looking for differences. Defaults to [].
- ignore_code_changes: bool, optional
Will ignore code changes while looking for differences. Defaults to False.
- algorithm_change: bool, optional
Whether to automatically solve the algorithm conflict (change of algo config). Defaults to True.
- orion_version_change: bool, optional
Whether to automatically solve the orion version conflict. Defaults to True.
- code_change_type: str, optional
How to resolve code change automatically. Must be one of ‘noeffect’, ‘unsure’ or ‘break’. Defaults to ‘break’.
- cli_change_type: str, optional
How to resolve cli change automatically. Must be one of ‘noeffect’, ‘unsure’ or ‘break’. Defaults to ‘break’.
- config_change_type: str, optional
How to resolve config change automatically. Must be one of ‘noeffect’, ‘unsure’ or ‘break’. Defaults to ‘break’.
-
orion.core.io.experiment_builder.
build_from_args
(cmdargs)[source]¶ Build an experiment based on commandline arguments.
Options provided in commandline and configuration file (–config) will overwrite system’s default values and configuration from database if experiment already exits. Commandline arguments have precedence over configuration file options.
See also
orion.core.io.experiment_builder.build()
for more information on experiment creation.
-
orion.core.io.experiment_builder.
build_view
(name, version=None)[source]¶ Load experiment from database
This function is deprecated and will be remove in v0.3.0. Use load() instead.
-
orion.core.io.experiment_builder.
clean_config
(name, config, branching)[source]¶ Clean configuration from hidden fields (ex:
_id
) and update branching if necessary
-
orion.core.io.experiment_builder.
consolidate_config
(name, version, config)[source]¶ Merge together given configuration with db configuration matching for experiment (
name
,version
)
-
orion.core.io.experiment_builder.
create_experiment
(name, version, mode, space, **kwargs)[source]¶ Instantiate the experiment and its attribute objects
All unspecified arguments will be replaced by system’s defaults (orion.core.config.*).
Parameters: - name: str
Name of the experiment.
- version: int
Version of the experiment.
- mode: str
The access rights of the experiment on the database. ‘r’: read access only ‘w’: can read and write to database ‘x’: can read and write to database, algo is instantiated and can execute optimization
- space: dict or Space object
Optimization space of the algorithm. If dict, should have the form dict(name=’<prior>(args)’).
- algorithms: str or dict, optional
Algorithm used for optimization.
- strategy: str or dict, optional
Parallel strategy to use to parallelize the algorithm.
- max_trials: int, optional
Maximum number or trials before the experiment is considered done.
- max_broken: int, optional
Number of broken trials for the experiment to be considered broken.
- storage: dict, optional
Configuration of the storage backend.
-
orion.core.io.experiment_builder.
fetch_config_from_db
(name, version=None)[source]¶ Fetch configuration from database
Parameters: - name: str
Name of the experiment to fetch
- version: int, optional
Version to select. If None, last version will be selected. If version given is larger than largest version available, the largest version will be selected.
-
orion.core.io.experiment_builder.
get_cmd_config
(cmdargs)[source]¶ Fetch configuration defined by commandline and local configuration file.
Arguments of commandline have priority over options in configuration file.
-
orion.core.io.experiment_builder.
get_from_args
(cmdargs, mode='r')[source]¶ Build an experiment view based on commandline arguments
See also
orion.core.io.experiment_builder.load()
for more information on creation of read-only experiments.
-
orion.core.io.experiment_builder.
load
(name, version=None, mode='r')[source]¶ Load experiment from database
An experiment view provides all reading operations of standard experiment but prevents the modification of the experiment and its trials.
Parameters: - name: str
Name of the experiment to build
- version: int, optional
Version to select. If None, last version will be selected. If version given is larger than largest version available, the largest version will be selected.
- mode: str, optional
The access rights of the experiment on the database. ‘r’: read access only ‘w’: can read and write to database Default is ‘r’