Experiment client¶
orion.client.experiment
– Experiment wrapper client¶
-
class
orion.client.experiment.
ExperimentClient
(experiment, producer, heartbeat=None)[source]¶ ExperimentClient providing all functionalities for the python API
Note that the ExperimentClient is not meant to be instantiated by the user. Look at
orion.client.create_experiment
to build an ExperimentClient.Parameters: - experiment: `orion.core.worker.experiment.Experiment`
Experiment object serving for interaction with storage
- producer: `orion.core.worker.producer.Producer`
Producer object used to produce new trials.
Attributes: algorithms
Algorithms of the experiment.
configuration
Return a copy of an Experiment configuration as a dictionary.
id
Return the id of the experiment in the database.
is_broken
Return True, if this experiment is considered to be broken.
is_done
Return True, if this experiment is considered to be finished.
max_trials
Max-trials to execute before stopping the experiment.
metadata
Metadata of the experiment.
name
Return the name of the experiment in the database.
node
Node of the experiment in the version control tree.
producer
Return the producer configuration of the experiment.
space
Return problem’s parameter
orion.algo.space.Space
.stats
Calculate a stats dictionary for this particular experiment.
version
Version of the experiment.
working_dir
Working directory of the experiment.
Methods
close
()Verify that no reserved trials are remaining and unregister atexit(). fetch_noncompleted_trials
([with_evc_tree])Fetch non-completed trials of this Experiment instance. fetch_trials
([with_evc_tree])Fetch all trials of the experiment fetch_trials_by_status
(status[, with_evc_tree])Fetch all trials with the given status get_trial
([trial, uid])Fetch a single trial insert
(params[, results, reserve])Insert a new trial. observe
(trial, results)Observe trial results release
(trial[, status])Release a trial. reserve
(trial)Reserve a trial. suggest
()Suggest a trial to execute. workon
(fct[, max_trials])Optimize a given function -
algorithms
¶ Algorithms of the experiment.
-
configuration
¶ Return a copy of an Experiment configuration as a dictionary.
-
fetch_noncompleted_trials
(with_evc_tree=False)[source]¶ Fetch non-completed trials of this Experiment instance.
Trials are sorted based on
Trial.submit_time
Note
It will return all non-completed trials, including new, reserved, suspended, interrupted and broken ones.
Returns: list of non-completed Trial
objects
-
fetch_trials
(with_evc_tree=False)[source]¶ Fetch all trials of the experiment
Parameters: - with_evc_tree: bool, optional
Fetch all trials from the EVC tree. Default: False
-
fetch_trials_by_status
(status, with_evc_tree=False)[source]¶ Fetch all trials with the given status
Trials are sorted based on
Trial.submit_time
Returns: list of Trial
objects
-
get_trial
(trial=None, uid=None)[source]¶ Fetch a single trial
Parameters: - trial: Trial, optional
trial object to retrieve from the database
- uid: str, optional
trial id used to retrieve the trial object
Returns: - return none if the trial is not found,
Raises: - UndefinedCall
if both trial and uid are not set
- AssertionError
if both trial and uid are provided and they do not match
-
id
¶ Return the id of the experiment in the database.
-
insert
(params, results=None, reserve=False)[source]¶ Insert a new trial.
Parameters: - params: dict
Parameters of the new trial to add to the database. These parameters must comply with the space definition otherwise a ValueError will be raised.
- results: list, optional
Results to be set for the new trial. Results must have the format {name: <str>: type: <’objective’, ‘constraint’ or ‘gradient’>, value=<float>} otherwise a ValueError will be raised. Note that passing results will mark the trial as completed and therefore cannot be reserved. The returned trial will have status ‘completed’. If the results are invalid, the trial will still be inserted but reservation will be released.
- reserve: bool, optional
If reserve=True, the inserted trial will be reserved.
reserve
cannot be True if results are given. Defaults to False.
Returns: orion.core.worker.trial.Trial
The trial inserted in storage. If reserve=True and no results are given, the returned trial will be in a reserved status.
Raises: - ValueError
- If results are given and reserve=True
- If params have invalid format
- If results have invalid format
orion.core.io.database.DuplicateKeyError
- If a trial with identical params already exist for the current experiment.
-
is_broken
¶ Return True, if this experiment is considered to be broken.
Count how many trials are broken and return True if that number has reached as given threshold.
-
is_done
¶ Return True, if this experiment is considered to be finished.
- Count how many trials have been completed and compare with
max_trials
. - Ask
algorithms
if they consider there is a chance for further improvement.
- Count how many trials have been completed and compare with
-
max_trials
¶ Max-trials to execute before stopping the experiment.
-
metadata
¶ Metadata of the experiment.
-
name
¶ Return the name of the experiment in the database.
-
node
¶ Node of the experiment in the version control tree.
-
observe
(trial, results)[source]¶ Observe trial results
Parameters: - trial: `orion.core.worker.trial.Trial`
Reserved trial to observe.
- results: list
Results to be set for the new trial. Results must have the format {name: <str>: type: <’objective’, ‘constraint’ or ‘gradient’>, value=<float>} otherwise a ValueError will be raised. If the results are invalid, the trial will not be released.
Returns: orion.core.worker.trial.Trial
The trial inserted in storage. If reserve=True and no results are given, the returned trial will be in a reserved status.
Raises: - ValueError
- If results have invalid format
- If the trial does not exist in storage.
- RuntimeError
If reservation of the trial has been lost prior to releasing it.
-
producer
¶ Return the producer configuration of the experiment.
-
release
(trial, status='interrupted')[source]¶ Release a trial.
Release the reservation and stop the heartbeat.
Parameters: - trial: `orion.core.worker.trial.Trial`
Trial to reserve.
- status: str, optional
Set the trial to given status while releasing the reservation. Defaults to ‘interrupted’.
Raises: - RuntimeError
If reservation of the trial has been lost prior to releasing it.
- ValueError
If the trial does not exist in storage.
-
reserve
(trial)[source]¶ Reserve a trial.
Set a trial status to reserve to ensure that concurrent process cannot work on it. Trials can only be reserved with status ‘new’, ‘interrupted’ or ‘suspended’.
Parameters: - trial: `orion.core.worker.trial.Trial`
Trial to reserve.
Raises: - RuntimeError
If trial is reserved by another process
- ValueError
If the trial does not exist in storage.
Notes
When reserved, a
TrialPacemaker
is started to update an heartbeat in storage. The frequency of the heartbeat is configurable at creation of experiment or withorion.core.config.worker.heartbeat
. If the process terminates unexpectedly, the heartbeat will cease and remote processes may reset the status of the trial to ‘interrupted’ when the heartbeat has not been updated since twice the value of heartbeat.
-
space
¶ Return problem’s parameter
orion.algo.space.Space
.
-
stats
¶ Calculate a stats dictionary for this particular experiment.
Returns: - stats : dict
-
suggest
()[source]¶ Suggest a trial to execute.
If any trial is available (new or interrupted), it selects one and reserves it. Otherwise, the algorithm is used to generate a new trial that is registered in storage and reserved.
Returns: - orior.core.worker.trial.Trial or None
Reserved trial for execution. Will return None if experiment is done. of if the algorithm cannot suggest until other trials complete.
Raises: WaitingForTrials
if the experiment is not completed and algorithm needs to wait for some trials to complete before it can suggest new trials.
BrokenExperiment
if too many trials failed to run and the experiment cannot continue. This is determined by
max_broken
in the configuration of the experiment.SampleTimeout
if the algorithm of the experiment could not sample new unique points.
-
version
¶ Version of the experiment.
-
working_dir
¶ Working directory of the experiment.
-
workon
(fct, max_trials=inf, **kwargs)[source]¶ Optimize a given function
Parameters: - fct: callable
Function to optimize. Must take arguments provided by trial.params. Additional constant parameter can be passed as
**kwargs
toworkon
. Function must return the final objective.- max_trials: int, optional
Maximum number of trials to execute within
workon
. If the experiment or algorithm reach status is_done before, the execution ofworkon
terminates.- **kwargs
Constant argument to pass to fct in addition to trial.params. If values in kwargs are present in trial.params, the latter takes precedence.
Raises: - ValueError
If results returned by fct have invalid format