Experiment client

Experiment wrapper client

Wraps the core Experiment object to provide further functionalities for the user

class orion.client.experiment.ExperimentClient(experiment, producer, executor=None, heartbeat=None)[source]

ExperimentClient providing all functionalities for the python API

Note that the ExperimentClient is not meant to be instantiated by the user. Look at orion.client.create_experiment to build an ExperimentClient.

Parameters
experiment: `orion.core.worker.experiment.Experiment`

Experiment object serving for interaction with storage

producer: `orion.core.worker.producer.Producer`

Producer object used to produce new trials.

Attributes
algorithms

Algorithms of the experiment.

configuration

Return a copy of an Experiment configuration as a dictionary.

id

Return the id of the experiment in the database.

is_broken

Return True, if this experiment is considered to be broken.

is_done

Return True, if this experiment is considered to be finished.

max_broken

Minimum number of broken trials before the experiment is considered broken.

max_trials

Max-trials to execute before stopping the experiment.

metadata

Metadata of the experiment.

mode

Return the access right of the experiment

name

Return the name of the experiment in the database.

node

Node of the experiment in the version control tree.

producer

Return the producer configuration of the experiment.

refers

References to the experiment version control

space

Return problem’s parameter orion.algo.space.Space.

stats

Calculate a stats dictionary for this particular experiment.

version

Version of the experiment.

working_dir

Working directory of the experiment.

Methods

close()

Verify that no reserved trials are remaining.

fetch_noncompleted_trials([with_evc_tree])

Fetch non-completed trials of this Experiment instance.

fetch_pending_trials([with_evc_tree])

Fetch all trials with status new, interrupted or suspended

fetch_trials([with_evc_tree])

Fetch all trials of the experiment

fetch_trials_by_status(status[, with_evc_tree])

Fetch all trials with the given status

get_trial([trial, uid])

Fetch a single trial

insert(params[, results, reserve])

Insert a new trial.

observe(trial, results)

Observe trial results

release(trial[, status])

Release a trial.

reserve(trial)

Reserve a trial.

suggest([pool_size])

Suggest a trial to execute.

tmp_executor(executor, **config)

Temporarily change the executor backend of the experiment client.

to_pandas([with_evc_tree])

Builds a dataframe with the trials of the experiment

workon(fct[, n_workers, pool_size, ...])

Optimize a given function

property algorithms

Algorithms of the experiment.

close()[source]

Verify that no reserved trials are remaining.

Experiment must be in executable (‘x’) mode.

Raises
orion.core.utils.exceptions.UnsupportedOperation

If the experiment was not loaded in executable mode.

property configuration

Return a copy of an Experiment configuration as a dictionary.

fetch_noncompleted_trials(with_evc_tree=False)[source]

Fetch non-completed trials of this Experiment instance.

Trials are sorted based on Trial.submit_time

Note

It will return all non-completed trials, including new, reserved, suspended, interrupted and broken ones.

Returns

list of non-completed orion.core.worker.trial.Trial objects

fetch_pending_trials(with_evc_tree=False)[source]

Fetch all trials with status new, interrupted or suspended

Trials are sorted based on Trial.submit_time

Returns

list of orion.core.worker.trial.Trial objects

fetch_trials(with_evc_tree=False)[source]

Fetch all trials of the experiment

Parameters
with_evc_tree: bool, optional

Fetch all trials from the EVC tree. Default: False

fetch_trials_by_status(status, with_evc_tree=False)[source]

Fetch all trials with the given status

Trials are sorted based on Trial.submit_time

Returns

list of orion.core.worker.trial.Trial objects

get_trial(trial=None, uid=None)[source]

Fetch a single trial

Parameters
trial: Trial, optional

trial object to retrieve from the database

uid: str, optional

trial id used to retrieve the trial object

Returns
return none if the trial is not found,
Raises
UndefinedCall

if both trial and uid are not set

AssertionError

if both trial and uid are provided and they do not match

property id

Return the id of the experiment in the database.

insert(params, results=None, reserve=False)[source]

Insert a new trial.

Experiment must be in writable (‘w’) or executable (‘x’) mode.

Parameters
params: dict

Parameters of the new trial to add to the database. These parameters must comply with the space definition otherwise a ValueError will be raised.

results: list, optional

Results to be set for the new trial. Results must have the format {name: <str>: type: <’objective’, ‘constraint’ or ‘gradient’>, value=<float>} otherwise a ValueError will be raised. Note that passing results will mark the trial as completed and therefore cannot be reserved. The returned trial will have status ‘completed’. If the results are invalid, the trial will still be inserted but reservation will be released.

reserve: bool, optional

If reserve=True, the inserted trial will be reserved. reserve cannot be True if results are given. Defaults to False.

Returns
orion.core.worker.trial.Trial

The trial inserted in storage. If reserve=True and no results are given, the returned trial will be in a reserved status.

Raises
ValueError
  • If results are given and reserve=True

  • If params have invalid format

  • If results have invalid format

orion.core.io.database.DuplicateKeyError
  • If a trial with identical params already exist for the current experiment.

orion.core.utils.exceptions.UnsupportedOperation

If the experiment was not loaded in writable mode.

property is_broken

Return True, if this experiment is considered to be broken.

Count how many trials are broken and return True if that number has reached as given threshold.

property is_done

Return True, if this experiment is considered to be finished.

  1. Count how many trials have been completed and compare with max_trials.

  2. Ask algorithms if they consider there is a chance for further improvement.

property max_broken

Minimum number of broken trials before the experiment is considered broken.

property max_trials

Max-trials to execute before stopping the experiment.

property metadata

Metadata of the experiment.

property mode

Return the access right of the experiment

{‘r’: read, ‘w’: read/write, ‘x’: read/write/execute}

property name

Return the name of the experiment in the database.

property node

Node of the experiment in the version control tree.

observe(trial, results)[source]

Observe trial results

Experiment must be in executable (‘x’) mode.

Parameters
trial: `orion.core.worker.trial.Trial`

Reserved trial to observe.

results: list

Results to be set for the new trial. Results must have the format {name: <str>: type: <’objective’, ‘constraint’ or ‘gradient’>, value=<float>} otherwise a ValueError will be raised. If the results are invalid, the trial will not be released.

Returns
orion.core.worker.trial.Trial

The trial inserted in storage. If reserve=True and no results are given, the returned trial will be in a reserved status.

Raises
ValueError
  • If results have invalid format

  • If the trial does not exist in storage.

RuntimeError

If reservation of the trial has been lost prior to releasing it.

orion.core.utils.exceptions.UnsupportedOperation

If the experiment was not loaded in executable mode.

property producer

Return the producer configuration of the experiment.

property refers

References to the experiment version control

release(trial, status='interrupted')[source]

Release a trial.

Release the reservation and stop the heartbeat.

Experiment must be in writable (‘w’) or executable (‘x’) mode.

Parameters
trial: `orion.core.worker.trial.Trial`

Trial to reserve.

status: str, optional

Set the trial to given status while releasing the reservation. Defaults to ‘interrupted’.

Raises
RuntimeError

If reservation of the trial has been lost prior to releasing it.

ValueError

If the trial does not exist in storage.

orion.core.utils.exceptions.UnsupportedOperation

If the experiment was not loaded in writable mode.

reserve(trial)[source]

Reserve a trial.

Experiment must be in executable (‘x’) mode.

Set a trial status to reserve to ensure that concurrent process cannot work on it. Trials can only be reserved with status ‘new’, ‘interrupted’ or ‘suspended’.

Parameters
trial: `orion.core.worker.trial.Trial`

Trial to reserve.

Raises
RuntimeError

If trial is reserved by another process

ValueError

If the trial does not exist in storage.

orion.core.utils.exceptions.UnsupportedOperation

If the experiment was not loaded in executable mode.

Notes

When reserved, a TrialPacemaker is started to update an heartbeat in storage. The frequency of the heartbeat is configurable at creation of experiment or with orion.core.config.worker.heartbeat. If the process terminates unexpectedly, the heartbeat will cease and remote processes may reset the status of the trial to ‘interrupted’ when the heartbeat has not been updated since twice the value of heartbeat.

property space

Return problem’s parameter orion.algo.space.Space.

property stats

Calculate a stats dictionary for this particular experiment.

Returns
statsdict
suggest(pool_size=0)[source]

Suggest a trial to execute.

Experiment must be in executable (‘x’) mode.

If any trial is available (new or interrupted), it selects one and reserves it. Otherwise, the algorithm is used to generate a new trial that is registered in storage and reserved.

Parameters
pool_size: int, optional

Number of trials to sample at a time. If 0, default to global config if defined, else 1. Increase it to improve the sampling speed if workers spend too much time waiting for algorithms to sample points. An algorithm will try sampling pool_size trials but may return less. Note: The method will still return only 1 trial even though if the pool size is larger than 1. This is because atomic reservation of trials can only be done one at a time.

Returns
orior.core.worker.trial.Trial

Reserved trial for execution.

Raises
orion.core.utils.exceptions.WaitingForTrials

if the experiment is not completed and algorithm needs to wait for some trials to complete before it can suggest new trials.

orion.core.utils.exceptions.BrokenExperiment

if too many trials failed to run and the experiment cannot continue. This is determined by max_broken in the configuration of the experiment.

orion.core.utils.exceptions.SampleTimeout

if the algorithm of the experiment could not sample new unique trials.

orion.core.utils.exceptions.CompletedExperiment

if the experiment was completed and algorithm could not sample new trials.

orion.core.utils.exceptions.UnsupportedOperation

If the experiment was not loaded in executable mode.

tmp_executor(executor, **config)[source]

Temporarily change the executor backend of the experiment client.

Parameters
executor: str or :class:`orion.executor.base.BaseExecutor`

The executor to use. If it is a str, the provided config will be used to create the executor with executor_factory.create(executor, **config).

**config:

Configuration to use if executor is a str.

to_pandas(with_evc_tree=False)[source]

Builds a dataframe with the trials of the experiment

Parameters
with_evc_tree: bool, optional

Fetch all trials from the EVC tree. Default: False

property version

Version of the experiment.

property working_dir

Working directory of the experiment.

workon(fct, n_workers=None, pool_size=0, max_trials=None, max_trials_per_worker=None, max_broken=None, trial_arg=None, on_error=None, **kwargs)[source]

Optimize a given function

Experiment must be in executable (‘x’) mode.

Parameters
fct: callable

Function to optimize. Must take arguments provided by trial.params. Additional constant parameter can be passed as **kwargs to workon. Function must return the final objective.

n_workers: int, optional

Number of workers to run in parallel. Defaults to value of global config.

pool_size: int, optional

Number of trials to sample at a time. If 0, defaults to n_workers or value of global config if defined. Increase it to improve the sampling speed if workers spend too much time waiting for algorithms to sample points. An algorithm will try sampling pool_size trials but may return less.

max_trials: int, optional

Maximum number of trials to execute within workon. If the experiment or algorithm reach status is_done before, the execution of workon terminates. Defaults to experiment’s max trial. If max_trials is larger than experiment.max_trials, the experiment will stop when reaching experiment.max_trials.

max_trials_per_worker: int, optional

Maximum number of trials to execute within each worker. max_trials and experiment.max_trials have precedence. Defaults to global config.

max_broken: int, optional

Maximum number of broken trials to accept during workon. When this threshold is reached the function will raise orion.core.utils.exceptions.BrokenExperiment. Defaults to value of global config.

trial_arg: str, optional

The function fct may support receiving the trial as an argument. This argument name can be specified with trial_arg. If not defined (None), then only the hyperparameters will be passed to fct.

on_error: callable, optional

Callback that is executed if an error occur during the execution of fct. The signature of the callback must be foo(ExperimentClient, Trial, Error, nb_of_worker_broken_trials). If the callblack returns False, the error will be ignored, otherwise it is counted for the threshold max_broken. In case of critical errors, you may also directly raise an error and force break out of workon.

**kwargs

Constant argument to pass to fct in addition to trial.params. If values in kwargs are present in trial.params, the latter takes precedence.

Raises
orion.core.utils.exceptions.InvalidResult

If results returned by fct have invalid format.

orion.core.utils.exceptions.WaitingForTrials

if the experiment is not completed and algorithm needs to wait for some trials to complete before it can suggest new trials.

orion.core.utils.exceptions.BrokenExperiment

if too many trials failed to run and the experiment cannot continue. This is determined by max_broken in the configuration of the experiment.

orion.core.utils.exceptions.SampleTimeout

if the algorithm of the experiment could not sample new unique points.

orion.core.utils.exceptions.UnsupportedOperation

If the experiment was not loaded in executable mode.

orion.client.experiment.reserve_trial(experiment, producer, pool_size, _depth=1)[source]

Reserve a new trial, or produce and reserve a trial if none are available.