Experiment client¶

Experiment wrapper client¶

Wraps the core Experiment object to provide further functionalities for the user

class orion.client.experiment.ExperimentClient(experiment: Experiment[AlgoT], executor: BaseExecutor | None = None, heartbeat: int | None = None)[source]¶

ExperimentClient providing all functionalities for the python API

Note that the ExperimentClient is not meant to be instantiated by the user. Look at orion.client.create_experiment to build an ExperimentClient.

Parameters

experiment: `orion.core.worker.experiment.Experiment`: Experiment object serving for interaction with storage

Attributes

algorithm: Algorithm of the experiment.
configuration: Return a copy of an Experiment configuration as a dictionary.
executor: Returns the current executor to use to run jobs in parallel
id: Return the id of the experiment in the database.
is_broken: Return True, if this experiment is considered to be broken.
is_done: Return True, if this experiment is considered to be finished.
max_broken: Minimum number of broken trials before the experiment is considered broken.
max_trials: Max-trials to execute before stopping the experiment.
metadata: Metadata of the experiment.
mode: Return the access right of the experiment
name: Return the name of the experiment in the database.
node: Node of the experiment in the version control tree.
producer: Return the producer configuration of the experiment.
refers: References to the experiment version control
space: Return problem’s parameter orion.algo.space.Space.
stats: Calculate orion.core.worker.experiment.ExperimentStats for this particular experiment.
storage: Return the storage currently in use by this client
version: Version of the experiment.
working_dir: Working directory of the experiment.

Methods

`close`()	Verify that no reserved trials are remaining.
`fetch_noncompleted_trials`([with_evc_tree])	Fetch non-completed trials of this Experiment instance.
`fetch_pending_trials`([with_evc_tree])	Fetch all trials with status new, interrupted or suspended
`fetch_trials`([with_evc_tree])	Fetch all trials of the experiment
`fetch_trials_by_status`(status[, with_evc_tree])	Fetch all trials with the given status
`get_trial`([trial, uid])	Fetch a single trial
`insert`(params[, results, reserve])	Insert a new trial.
`observe`(trial, results[, name])	Observe trial results
`release`(trial[, status])	Release a trial.
`reserve`(trial)	Reserve a trial.
`suggest`([pool_size])	Suggest a trial to execute.
`tmp_executor`(executor, **config)	Temporarily change the executor backend of the experiment client.
`to_pandas`([with_evc_tree])	Builds a dataframe with the trials of the experiment
`workon`(fct[, n_workers, pool_size, ...])	Optimize a given function

property algorithm¶: Algorithm of the experiment.

close()[source]¶

Verify that no reserved trials are remaining.

Experiment must be in executable (‘x’) mode.

Raises

orion.core.utils.exceptions.UnsupportedOperation: If the experiment was not loaded in executable mode.

property configuration: ExperimentConfig¶: Return a copy of an Experiment configuration as a dictionary.

property executor¶: Returns the current executor to use to run jobs in parallel

fetch_noncompleted_trials(with_evc_tree=False)[source]¶

Fetch non-completed trials of this Experiment instance.

Trials are sorted based on Trial.submit_time

Note

It will return all non-completed trials, including new, reserved, suspended, interrupted and broken ones.

Returns: list of non-completed orion.core.worker.trial.Trial objects

fetch_pending_trials(with_evc_tree=False)[source]¶

Fetch all trials with status new, interrupted or suspended

Trials are sorted based on Trial.submit_time

Returns: list of orion.core.worker.trial.Trial objects

fetch_trials(with_evc_tree=False) → list[Trial][source]¶

Fetch all trials of the experiment

Parameters

with_evc_tree: bool, optional: Fetch all trials from the EVC tree. Default: False

fetch_trials_by_status(status, with_evc_tree=False)[source]¶

Fetch all trials with the given status

Trials are sorted based on Trial.submit_time

Returns: list of orion.core.worker.trial.Trial objects

get_trial(trial=None, uid=None)[source]¶

Fetch a single trial

Parameters

trial: Trial, optional: trial object to retrieve from the database
uid: str, optional: trial id used to retrieve the trial object

Returns

return none if the trial is not found,

Raises

UndefinedCall: if both trial and uid are not set
AssertionError: if both trial and uid are provided and they do not match

property id¶: Return the id of the experiment in the database.

insert(params, results=None, reserve=False)[source]¶

Insert a new trial.

Experiment must be in writable (‘w’) or executable (‘x’) mode.

Parameters

params: dict: Parameters of the new trial to add to the database. These parameters must comply with the space definition otherwise a ValueError will be raised.
results: list, optional: Results to be set for the new trial. Results must have the format {name: <str>: type: <’objective’, ‘constraint’ or ‘gradient’>, value=<float>} otherwise a ValueError will be raised. Note that passing results will mark the trial as completed and therefore cannot be reserved. The returned trial will have status ‘completed’. If the results are invalid, the trial will still be inserted but reservation will be released.
reserve: bool, optional: If reserve=True, the inserted trial will be reserved. reserve cannot be True if results are given. Defaults to False.

Returns

orion.core.worker.trial.Trial: The trial inserted in storage. If reserve=True and no results are given, the returned trial will be in a reserved status.

Raises

ValueError

If results are given and reserve=True
If params have invalid format
If results have invalid format

orion.core.io.database.DuplicateKeyError

If a trial with identical params already exist for the current experiment.

orion.core.utils.exceptions.UnsupportedOperation

If the experiment was not loaded in writable mode.

property is_broken¶

Return True, if this experiment is considered to be broken.

Count how many trials are broken and return True if that number has reached as given threshold.

property is_done: bool¶

Return True, if this experiment is considered to be finished.

Count how many trials have been completed and compare with max_trials.
Ask algorithms if they consider there is a chance for further improvement.

property max_broken¶: Minimum number of broken trials before the experiment is considered broken.

property max_trials¶: Max-trials to execute before stopping the experiment.

property metadata¶: Metadata of the experiment.

property mode¶

Return the access right of the experiment

{‘r’: read, ‘w’: read/write, ‘x’: read/write/execute}

property name¶: Return the name of the experiment in the database.

property node¶: Node of the experiment in the version control tree.

observe(trial: Trial, results: list[dict] | float, name: str = 'objective') → None[source]¶

Observe trial results

Experiment must be in executable (‘x’) mode.

Parameters

trial: `orion.core.worker.trial.Trial`: Reserved trial to observe.
results: list or float: Results to be set for the new trial. Results must have the format {name: <str>: type: <’objective’, ‘constraint’ or ‘gradient’>, value=<float>} otherwise a ValueError will be raised. If the results are invalid, the trial will not be released. If results is a float, the result type will be ‘objective’.
name: str: Name of the result if results is a float. Default: ‘objective’.

Returns

orion.core.worker.trial.Trial: The trial inserted in storage. If reserve=True and no results are given, the returned trial will be in a reserved status.

Raises

ValueError

If results have invalid format
If the trial does not exist in storage.

RuntimeError

If reservation of the trial has been lost prior to releasing it.

orion.core.utils.exceptions.UnsupportedOperation

If the experiment was not loaded in executable mode.

orion.core.utils.exceptions.InvalidResult

If the format of trial result is invalid.

property producer¶: Return the producer configuration of the experiment.

property refers¶: References to the experiment version control

release(trial, status='interrupted')[source]¶

Release a trial.

Release the reservation and stop the heartbeat.

Experiment must be in writable (‘w’) or executable (‘x’) mode.

Parameters

trial: `orion.core.worker.trial.Trial`: Trial to reserve.
status: str, optional: Set the trial to given status while releasing the reservation. Defaults to ‘interrupted’.

Raises

RuntimeError: If reservation of the trial has been lost prior to releasing it.
AlreadyReleased: If reservation of trial was already released
ValueError: If the trial does not exist in storage.
orion.core.utils.exceptions.UnsupportedOperation: If the experiment was not loaded in writable mode.

reserve(trial)[source]¶

Reserve a trial.

Experiment must be in executable (‘x’) mode.

Set a trial status to reserve to ensure that concurrent process cannot work on it. Trials can only be reserved with status ‘new’, ‘interrupted’ or ‘suspended’.

Parameters

trial: `orion.core.worker.trial.Trial`: Trial to reserve.

Raises

RuntimeError: If trial is reserved by another process
ValueError: If the trial does not exist in storage.
orion.core.utils.exceptions.UnsupportedOperation: If the experiment was not loaded in executable mode.

Notes

When reserved, a TrialPacemaker is started to update an heartbeat in storage. The frequency of the heartbeat is configurable at creation of experiment or with orion.core.config.worker.heartbeat. If the process terminates unexpectedly, the heartbeat will cease and remote processes may reset the status of the trial to ‘interrupted’ when the heartbeat has not been updated since twice the value of heartbeat.

property space: Space¶: Return problem’s parameter orion.algo.space.Space.

property stats¶: Calculate orion.core.worker.experiment.ExperimentStats for this particular experiment.

property storage¶: Return the storage currently in use by this client

suggest(pool_size=0)[source]¶

Suggest a trial to execute.

Experiment must be in executable (‘x’) mode.

If any trial is available (new or interrupted), it selects one and reserves it. Otherwise, the algorithm is used to generate a new trial that is registered in storage and reserved.

Parameters

pool_size: int, optional: Number of trials to sample at a time. If 0, default to global config if defined, else 1. Increase it to improve the sampling speed if workers spend too much time waiting for algorithms to sample points. An algorithm will try sampling pool_size trials but may return less. Note: The method will still return only 1 trial even though if the pool size is larger than 1. This is because atomic reservation of trials can only be done one at a time.

Returns

orior.core.worker.trial.Trial: Reserved trial for execution.

Raises

orion.core.utils.exceptions.WaitingForTrials: if the experiment is not completed and algorithm needs to wait for some trials to complete before it can suggest new trials.
orion.core.utils.exceptions.BrokenExperiment: if too many trials failed to run and the experiment cannot continue. This is determined by max_broken in the configuration of the experiment.
orion.core.utils.exceptions.ReservationRaceCondition: If a trial could not be reserved right after they were generated
orion.core.utils.exceptions.CompletedExperiment: if the experiment was completed and algorithm could not sample new trials.
orion.core.utils.exceptions.UnsupportedOperation: If the experiment was not loaded in executable mode.

tmp_executor(executor, **config)[source]¶

Temporarily change the executor backend of the experiment client.

Parameters

executor: str or :class:`orion.executor.base.BaseExecutor`: The executor to use. If it is a str, the provided config will be used to create the executor with executor_factory.create(executor, **config).
**config:: Configuration to use if executor is a str.

to_pandas(with_evc_tree=False)[source]¶

Builds a dataframe with the trials of the experiment

Parameters

with_evc_tree: bool, optional: Fetch all trials from the EVC tree. Default: False

property version¶: Version of the experiment.

property working_dir¶: Working directory of the experiment.

workon(fct: Callable, n_workers: int | None = None, pool_size: int = 0, max_trials: int | None = None, max_trials_per_worker: int | None = None, max_broken: int | None = None, trial_arg: str | None = None, on_error: Callable[[ExperimentClient, Exception, int], bool] | None = None, prepare_trial: Callable[[ExperimentClient, Trial], None] = <function prepare_trial_working_dir>, idle_timeout: int | None = None, **kwargs) → int[source]¶

Optimize a given function

Experiment must be in executable (‘x’) mode.

Parameters

fct: callable: Function to optimize. Must take arguments provided by trial.params. Additional constant parameter can be passed as **kwargs to workon. Function must return the final objective.
n_workers: int, optional: Number of workers to run in parallel. Defaults to value of global config.
pool_size: int, optional: Number of trials to sample at a time. If 0, defaults to n_workers or value of global config if defined. Increase it to improve the sampling speed if workers spend too much time waiting for algorithms to sample points. An algorithm will try sampling pool_size trials but may return less.
max_trials: int, optional: Maximum number of trials to execute within workon. If the experiment or algorithm reach status is_done before, the execution of workon terminates. Defaults to experiment’s max trial. If max_trials is larger than experiment.max_trials, the experiment will stop when reaching experiment.max_trials.
max_trials_per_worker: int, optional: Maximum number of trials to execute within each worker. max_trials and experiment.max_trials have precedence. Defaults to global config.
max_broken: int, optional: Maximum number of broken trials to accept during workon. When this threshold is reached the function will raise orion.core.utils.exceptions.BrokenExperiment. Defaults to value of global config.
trial_arg: str, optional: The function fct may support receiving the trial as an argument. This argument name can be specified with trial_arg. If not defined (None), then only the hyperparameters will be passed to fct.
on_error: callable, optional: Callback that is executed if an error occur during the execution of fct. The signature of the callback must be foo(ExperimentClient, Trial, Error, nb_of_worker_broken_trials). If the callblack returns False, the error will be ignored, otherwise it is counted for the threshold max_broken. In case of critical errors, you may also directly raise an error and force break out of workon.
prepare_trial: callable, optional: Callback that is executed before the trial is submitted to workers for execution. Default is orion.client.runner.prepare_trial_working_dir which will create working directory of trials if necessary.
idle_timeout: int, optional: Maximum time (seconds) allowed for idle workers. LazyWorkers will be raised if timeout is reached. Such timeout are generally caused when reaching the end of the optimization when no new trials can be sampled for the idle workers. Defaults to orion.core.config.worker.idle_timeout.
**kwargs: Constant argument to pass to fct in addition to trial.params. If values in kwargs are present in trial.params, the latter takes precedence.

Raises

orion.core.utils.exceptions.LazyWorkers: If workers stay idle for too long.
orion.core.utils.exceptions.InvalidResult: If results returned by fct have invalid format.
orion.core.utils.exceptions.WaitingForTrials: If the experiment is not completed and algorithm needs to wait for some trials to complete before it can suggest new trials.
orion.core.utils.exceptions.BrokenExperiment: If too many trials failed to run and the experiment cannot continue. This is determined by max_broken in the configuration of the experiment.
orion.core.utils.exceptions.ReservationRaceCondition: If a trial could not be reserved right after they were generated.
orion.core.utils.exceptions.UnsupportedOperation: If the experiment was not loaded in executable mode.

orion.client.experiment.reserve_trial(experiment: Experiment, producer: Producer, pool_size: int, timeout: int | None = None) → Trial[source]¶: Reserve a new trial, or produce and reserve a trial if none are available.