Client helper functions

Python API

Provides functions for communicating with orion.core.

orion.client.build_experiment(name, version=None, space=None, algorithms=None, strategy=None, max_trials=None, max_broken=None, storage=None, branching=None, max_idle_time=None, heartbeat=None, working_dir=None, debug=False, executor=None)[source]

Build an experiment to be executable

Building the experiment can result in branching if there are any changes in the environment. This is required to ensure coherence between execution of trials. For an experiment in read/write mode without execution rights, see get_experiment.

There is 2 main scenarios

  1. The experiment is new

name and space arguments are required, otherwise NoConfigurationError will be raised.

All other arguments (algorithms, strategy, max_trials, storage, branching and working_dir) will be replaced by system’s defaults if ommited. The system’s defaults can also be overriden in global configuration file as described for the database in Configuring the database. We do not recommand overriding the algorithm configuration using system’s default, but overriding the storage configuration can be very convenient if the same storage is used for all your experiments.

  1. The experiment exist in the database.

We can break down this scenario in two sub-scenarios for clarity.

2.1) Only experiment name is given.

The configuration will be fetched from database.

2.2) Some other arguments than the name are given.

The configuration will be fetched from database and given arguments will override them. max_trials and max_broken may be overwritten in DB, but any other changes will lead to a branching. Instead of creating the experiment (name, version), it will create a new experiment (name, version+1) which will have the same configuration than (name, version) except for the differing arguments given by user. This new experiment will have access to trials of (name, version), adapted according to the differences between version and version+1. A previous version can be accessed by specifying the version argument.

Causes of experiment branching are:

  • Change of search space

    • New dimension

    • Different prior

    • Missing dimension

  • Change of algorithm

  • Change of strategy (Not implemented yet)

  • Change of code version (Only supported by commandline API for now)

  • Change of orion version

Parameters
name: str

Name of the experiment

version: int, optional

Version of the experiment. Defaults to last existing version for a given name or 1 for new experiment.

space: dict, optional

Optimization space of the algorithm. Should have the form dict(name='<prior>(args)').

algorithms: str or dict, optional

Algorithm used for optimization.

strategy: str or dict, optional

Parallel strategy to use to parallelize the algorithm.

max_trials: int, optional

Maximum number or trials before the experiment is considered done.

max_broken: int, optional

Number of broken trials for the experiment to be considered broken.

storage: dict, optional

Configuration of the storage backend.

working_dir: str, optional

Working directory created for the experiment inside which a unique folder will be created for each trial. Defaults to a temporary directory that is deleted at end of execution.

max_idle_time: int, optional

Maximum time the producer can spend trying to generate a new suggestion. Such timeout are generally caused by slow database, large number of concurrent workers leading to many race conditions or small search spaces with integer/categorical dimensions that may be fully explored. Defaults to orion.core.config.worker.max_idle_time.

heartbeat: int, optional

Frequency (seconds) at which the heartbeat of the trial is updated. If the heartbeat of a reserved trial is larger than twice the configured heartbeat, Oríon will reset the status of the trial to interrupted. This allows restoring lost trials (ex: due to killed worker). Defaults to orion.core.config.worker.max_idle_time.

debug: bool, optional

If using in debug mode, the storage config is overrided with legacy:EphemeralDB. Defaults to False.

branching: dict, optional

Arguments to control the branching.

branch_to: str, optional

Name of the experiment to branch to. The parent experiment will be the one specified by (name, version), and the child will be (branch_to, 1).

branch_from: str, optional

Name of the experiment to branch from. The parent experiment will be the one specified by (branch_from, last version), and the child will be (name, 1).

manual_resolution: bool, optional

Starts the prompt to resolve manually the conflicts. Defaults to False.

algorithm_change: bool, optional

Whether to automatically solve the algorithm conflict (change of algo config). Defaults to True.

orion_version_change: bool, optional

Whether to automatically solve the orion version conflict. Defaults to True.

code_change_type: str, optional

How to resolve code change automatically. Must be one of ‘noeffect’, ‘unsure’ or ‘break’. Defaults to ‘break’.

cli_change_type: str, optional

How to resolve cli change automatically. Must be one of ‘noeffect’, ‘unsure’ or ‘break’. Defaults to ‘break’.

config_change_type: str, optional

How to resolve config change automatically. Must be one of ‘noeffect’, ‘unsure’ or ‘break’. Defaults to ‘break’.

executor: `orion.executor.base.BaseExecutor`, optional

Executor to run the experiment

Raises
orion.core.utils.singleton.SingletonAlreadyInstantiatedError

If the storage is already instantiated and given configuration is different. Storage is a singleton, you may only use one instance per process.

orion.core.utils.exceptions.NoConfigurationError

The experiment is not in database and no space is provided by the user.

orion.core.utils.exceptions.RaceCondition

There was a race condition during branching and new version cannot be infered because of that. Single race conditions are normally handled seemlessly. If this error gets raised, it means that different modifications occured during each race condition resolution. This is likely due to quick code change during experiment creation. Make sure your script is not generating files within your code repository.

orion.core.utils.exceptions.BranchingEvent

The configuration is different than the corresponding one in DB and the branching cannot be solved automatically. This usually happens if the version=x is specified but the experiment (name, x) already has a child (name, x+1). If you really need to branch from version x, give it a new name to branch to with branching={'branch_to': <new_name>}.

NotImplementedError

If the algorithm, storage or strategy specified is not properly installed.

orion.client.create_experiment(name, **config)[source]

Build an experiment to be executable

This function is deprecated and will be removed in v0.3.0. Use build_experiment instead.

orion.client.get_experiment(name, version=None, mode='r', storage=None)[source]

Retrieve an existing experiment as orion.client.experiment.ExperimentClient.

Parameters
name: str

The name of the experiment.

version: int, optional

Version to select. If None, last version will be selected. If version given is larger than largest version available, the largest version will be selected.

mode: str, optional

The access rights of the experiment on the database. ‘r’: read access only ‘w’: can read and write to database Default is ‘r’

storage: dict, optional

Configuration of the storage backend.

Returns
An instance of orion.client.experiment.ExperimentClient representing the experiment.
Raises
orion.core.utils.exceptions.NoConfigurationError

The experiment is not in the database provided by the user.

orion.client.interrupt_trial()[source]

Send interrupt signal to Oríon worker

orion.client.report_bad_trial(objective=10000000000.0, name='objective', data=None)[source]

Report a bad trial with large objective to Oríon.

This is especially useful if some parameter values lead to exceptions such as out of memory. Reporting a large objective from such trials will push algorithms towards valid configurations.

Warning

To be called only once in order to report a final evaluation of a particular trial.

Warning

Oríon is only minimizing. Make sure to report a metric that you seek to minimize.

Note

In case that user’s script is not running in a orion’s context, this function will act as a Python print function.

Parameters
objective: float

Objective the return to Oríon for the current trial. The default objective is 1e10. This may not be valid for some metrics and this value should be overrided accordingly. In the case of error rates for instance, the value should be 1.0.

name: str, optional

Name of the objective. Default is ‘objective’.

data: list of dict, optional

A list of dictionary representing the results in the form dict(name=result_name, type=’statistic’, value=0). The types supported are ‘contraint’, ‘gradient’ and ‘statistic’.

orion.client.report_objective(objective, name='objective')[source]

Report only the objective at the end of execution

To send more data (statistic, constraint, gradient), use report_results.

Warning

To be called only once in order to report a final evaluation of a particular trial.

Warning

Oríon is only minimizing. Make sure to report a metric that you seek to minimize.

Note

In case that user’s script is not running in a orion’s context, this function will act as a Python print function.

Parameters
objective: float

Objective the return to Oríon for the current trial.

name: str, optional

Name of the objective. Default is ‘objective’.

orion.client.report_results(data)[source]

Facilitate the reporting of results for a user’s script acting as a black-box computation.

Warning

To be called only once in order to report a final evaluation of a particular trial.

Warning

Oríon is only minimizing. Make sure to report a metric that you seek to minimize.

Note

In case that user’s script is not running in a orion’s context, this function will act as a Python print function.

Parameters
data: list of dict

A list of dictionary representing the results in the form dict(name=result_name, type=’statistic’, value=0). The types supported are ‘objective’, ‘contraint’, ‘gradient’ and ‘statistic’. The list should contain at least one ‘objective’, which is the metric the algorithm will be minimizing.

orion.client.workon(function, space, name='loop', algorithms=None, max_trials=None, max_broken=None)[source]

Optimize a function over a given search space

This will create a new experiment with an in-memory storage and optimize the given function until max_trials is reached or the algorithm is done (some algorithms like random search are never done).

For informations on how to fetch results, see orion.client.experiment.ExperimentClient.

Note

Each call to this function will create a separate in-memory storage.

Parameters
name: str

Name of the experiment

version: int, optional

Version of the experiment. Defaults to last existing version for a given name or 1 for new experiment.

space: dict, optional

Optimization space of the algorithm. Should have the form dict(name=’<prior>(args)’).

algorithms: str or dict, optional

Algorithm used for optimization.

max_trials: int, optional

Maximum number or trials before the experiment is considered done.

max_broken: int, optional

Number of broken trials for the experiment to be considered broken.

Raises
NotImplementedError

If the algorithm specified is not properly installed.