Experiment client¶

Experiment wrapper client¶

Wraps the core Experiment object to provide further functionalities for the user

class orion.client.experiment.ExperimentClient(experiment, producer, heartbeat=None)[source]¶

ExperimentClient providing all functionalities for the python API

Note that the ExperimentClient is not meant to be instantiated by the user. Look at orion.client.create_experiment to build an ExperimentClient.

Parameters:

experiment: `orion.core.worker.experiment.Experiment`: Experiment object serving for interaction with storage
producer: `orion.core.worker.producer.Producer`: Producer object used to produce new trials.

Attributes:

algorithms: Algorithms of the experiment.
configuration: Return a copy of an Experiment configuration as a dictionary.
id: Return the id of the experiment in the database.
is_broken: Return True, if this experiment is considered to be broken.
is_done: Return True, if this experiment is considered to be finished.
max_broken: Minimum number of broken trials before the experiment is considered broken.
max_trials: Max-trials to execute before stopping the experiment.
metadata: Metadata of the experiment.
mode: Return the access right of the experiment
name: Return the name of the experiment in the database.
node: Node of the experiment in the version control tree.
producer: Return the producer configuration of the experiment.
refers: References to the experiment version control
space: Return problem’s parameter orion.algo.space.Space.
stats: Calculate a stats dictionary for this particular experiment.
version: Version of the experiment.
working_dir: Working directory of the experiment.

Methods

`close`()	Verify that no reserved trials are remaining and unregister atexit().
`fetch_noncompleted_trials`([with_evc_tree])	Fetch non-completed trials of this Experiment instance.
`fetch_trials`([with_evc_tree])	Fetch all trials of the experiment
`fetch_trials_by_status`(status[, with_evc_tree])	Fetch all trials with the given status
`get_trial`([trial, uid])	Fetch a single trial
`insert`(params[, results, reserve])	Insert a new trial.
`observe`(trial, results)	Observe trial results
`release`(trial[, status])	Release a trial.
`reserve`(trial)	Reserve a trial.
`suggest`()	Suggest a trial to execute.
`to_pandas`([with_evc_tree])	Builds a dataframe with the trials of the experiment
`workon`(fct[, max_trials])	Optimize a given function

algorithms¶: Algorithms of the experiment.

close()[source]¶

Verify that no reserved trials are remaining and unregister atexit().

Experiment must be in executable (‘x’) mode.

Raises:	`orion.core.utils.exceptions.UnsupportedOperation` If the experiment was not loaded in executable mode.

configuration¶: Return a copy of an Experiment configuration as a dictionary.

fetch_noncompleted_trials(with_evc_tree=False)[source]¶

Fetch non-completed trials of this Experiment instance.

Trials are sorted based on Trial.submit_time

Note

It will return all non-completed trials, including new, reserved, suspended, interrupted and broken ones.

Returns:	list of non-completed `orion.core.worker.trial.Trial` objects

fetch_trials(with_evc_tree=False)[source]¶

Fetch all trials of the experiment

Parameters:	with_evc_tree: bool, optional Fetch all trials from the EVC tree. Default: False

fetch_trials_by_status(status, with_evc_tree=False)[source]¶

Fetch all trials with the given status

Trials are sorted based on Trial.submit_time

Returns:	list of `orion.core.worker.trial.Trial` objects

get_trial(trial=None, uid=None)[source]¶

Fetch a single trial

Parameters:	trial: Trial, optional trial object to retrieve from the database uid: str, optional trial id used to retrieve the trial object
Returns:	return none if the trial is not found,
Raises:	UndefinedCall if both trial and uid are not set AssertionError if both trial and uid are provided and they do not match

id¶: Return the id of the experiment in the database.

insert(params, results=None, reserve=False)[source]¶

Insert a new trial.

Experiment must be in writable (‘w’) or executable (‘x’) mode.

Parameters:	params: dict Parameters of the new trial to add to the database. These parameters must comply with the space definition otherwise a ValueError will be raised. results: list, optional Results to be set for the new trial. Results must have the format {name: <str>: type: <’objective’, ‘constraint’ or ‘gradient’>, value=<float>} otherwise a ValueError will be raised. Note that passing results will mark the trial as completed and therefore cannot be reserved. The returned trial will have status ‘completed’. If the results are invalid, the trial will still be inserted but reservation will be released. reserve: bool, optional If reserve=True, the inserted trial will be reserved. `reserve` cannot be True if results are given. Defaults to False.
Returns:	`orion.core.worker.trial.Trial` The trial inserted in storage. If reserve=True and no results are given, the returned trial will be in a reserved status.
Raises:	ValueError If results are given and reserve=True If params have invalid format If results have invalid format `orion.core.io.database.DuplicateKeyError` If a trial with identical params already exist for the current experiment. `orion.core.utils.exceptions.UnsupportedOperation` If the experiment was not loaded in writable mode.

is_broken¶

Return True, if this experiment is considered to be broken.

Count how many trials are broken and return True if that number has reached as given threshold.

is_done¶

Return True, if this experiment is considered to be finished.

Count how many trials have been completed and compare with max_trials.
Ask algorithms if they consider there is a chance for further improvement.

max_broken¶: Minimum number of broken trials before the experiment is considered broken.

max_trials¶: Max-trials to execute before stopping the experiment.

metadata¶: Metadata of the experiment.

mode¶

Return the access right of the experiment

{‘r’: read, ‘w’: read/write, ‘x’: read/write/execute}

name¶: Return the name of the experiment in the database.

node¶: Node of the experiment in the version control tree.

observe(trial, results)[source]¶

Observe trial results

Experiment must be in executable (‘x’) mode.

Parameters:	trial: `orion.core.worker.trial.Trial` Reserved trial to observe. results: list Results to be set for the new trial. Results must have the format {name: <str>: type: <’objective’, ‘constraint’ or ‘gradient’>, value=<float>} otherwise a ValueError will be raised. If the results are invalid, the trial will not be released.
Returns:	`orion.core.worker.trial.Trial` The trial inserted in storage. If reserve=True and no results are given, the returned trial will be in a reserved status.
Raises:	ValueError If results have invalid format If the trial does not exist in storage. RuntimeError If reservation of the trial has been lost prior to releasing it. `orion.core.utils.exceptions.UnsupportedOperation` If the experiment was not loaded in executable mode.

producer¶: Return the producer configuration of the experiment.

refers¶: References to the experiment version control

release(trial, status='interrupted')[source]¶

Release a trial.

Release the reservation and stop the heartbeat.

Experiment must be in writable (‘w’) or executable (‘x’) mode.

Parameters:	trial: `orion.core.worker.trial.Trial` Trial to reserve. status: str, optional Set the trial to given status while releasing the reservation. Defaults to ‘interrupted’.
Raises:	RuntimeError If reservation of the trial has been lost prior to releasing it. ValueError If the trial does not exist in storage. `orion.core.utils.exceptions.UnsupportedOperation` If the experiment was not loaded in writable mode.

reserve(trial)[source]¶

Reserve a trial.

Experiment must be in executable (‘x’) mode.

Set a trial status to reserve to ensure that concurrent process cannot work on it. Trials can only be reserved with status ‘new’, ‘interrupted’ or ‘suspended’.

Parameters:	trial: `orion.core.worker.trial.Trial` Trial to reserve.
Raises:	RuntimeError If trial is reserved by another process ValueError If the trial does not exist in storage. `orion.core.utils.exceptions.UnsupportedOperation` If the experiment was not loaded in executable mode.

Notes

When reserved, a TrialPacemaker is started to update an heartbeat in storage. The frequency of the heartbeat is configurable at creation of experiment or with orion.core.config.worker.heartbeat. If the process terminates unexpectedly, the heartbeat will cease and remote processes may reset the status of the trial to ‘interrupted’ when the heartbeat has not been updated since twice the value of heartbeat.

space¶: Return problem’s parameter orion.algo.space.Space.

stats¶

Calculate a stats dictionary for this particular experiment.

Returns:	stats : dict

suggest()[source]¶

Suggest a trial to execute.

Experiment must be in executable (‘x’) mode.

If any trial is available (new or interrupted), it selects one and reserves it. Otherwise, the algorithm is used to generate a new trial that is registered in storage and reserved.

Returns:

orior.core.worker.trial.Trial or None: Reserved trial for execution. Will return None if experiment is done. of if the algorithm cannot suggest until other trials complete.

Raises:

orion.core.utils.exceptions.WaitingForTrials: if the experiment is not completed and algorithm needs to wait for some trials to complete before it can suggest new trials.
orion.core.utils.exceptions.BrokenExperiment: if too many trials failed to run and the experiment cannot continue. This is determined by max_broken in the configuration of the experiment.
orion.core.utils.exceptions.SampleTimeout: if the algorithm of the experiment could not sample new unique points.
orion.core.utils.exceptions.UnsupportedOperation: If the experiment was not loaded in executable mode.

to_pandas(with_evc_tree=False)[source]¶

Builds a dataframe with the trials of the experiment

Parameters:	with_evc_tree: bool, optional Fetch all trials from the EVC tree. Default: False

version¶: Version of the experiment.

working_dir¶: Working directory of the experiment.

workon(fct, max_trials=inf, **kwargs)[source]¶

Optimize a given function

Experiment must be in executable (‘x’) mode.

Parameters:

fct: callable: Function to optimize. Must take arguments provided by trial.params. Additional constant parameter can be passed as **kwargs to workon. Function must return the final objective.
max_trials: int, optional: Maximum number of trials to execute within workon. If the experiment or algorithm reach status is_done before, the execution of workon terminates.
**kwargs: Constant argument to pass to fct in addition to trial.params. If values in kwargs are present in trial.params, the latter takes precedence.

Raises:

ValueError: If results returned by fct have invalid format
orion.core.utils.exceptions.UnsupportedOperation: If the experiment was not loaded in executable mode.

orion.client.experiment.set_broken_trials(client)[source]¶: Release all trials with status broken if the process exits without releasing them.