Experiment¶

Description of an optimization attempt¶

Manage history of trials corresponding to a black box process.

Represents an entry in database/experiments collection.

Notes

The following list represents possible entries in the metadata dict.

user: str: System user currently owning this running process, the one who invoked Oríon.
datetime: datetime.datetime: When was this particular configuration submitted to the database.
orion_version: str: Version of Oríon which suggested this experiment. user’s current Oríon version.
user_script: str: Full absolute path to user’s executable.
user_args: list of str: Contains separate arguments to be passed when invoking user_script, possibly templated for Oríon.
user_vcs: str, optional: User’s version control system for this executable’s code repository.
user_version: str, optional: Current user’s repository version.
user_commit_hash: str, optional: Current Experiment’s commit hash for Oríon’s invocation.

Attributes

name: str: Unique identifier for this experiment per user.
id: object: id of the experiment in the database if experiment is configured. Value is None if the experiment is not configured.
refers: dict or list of `Experiment` objects, after initialization is done.: A dictionary pointing to a past Experiment id, refers[parent_id], whose trials we want to add in the history of completed trials we want to re-use. For the purpose of convenience and database efficiency, all experiments of a common tree share a refers[root_id], with the root experiment referring to itself.
version: int: Current version of this experiment.
metadata: dict: Contains managerial information about this Experiment.
max_trials: int: How many trials must be evaluated, before considering this Experiment done. This attribute can be updated if the rest of the experiment configuration is the same. In that case, if trying to set to an already set experiment, it will overwrite the previous one.
max_broken: int: How many trials must be broken, before considering this Experiment broken. This attribute can be updated if the rest of the experiment configuration is the same. In that case, if trying to set to an already set experiment, it will overwrite the previous one.
space: Space: Object representing the optimization space.
algorithm: `BaseAlgorithm` object or a wrapper.: Complete specification of the optimization and dynamical procedures taking place in this Experiment.

Methods

`acquire_algorithm_lock`([timeout, retry_interval])	Acquire lock on algorithm
`duplicate_pending_trials`()	Find pending trials in EVC and duplicate them in current experiment.
`fetch_lost_trials`([with_evc_tree])	Fetch all reserved trials that are lost (old heartbeat)
`fetch_noncompleted_trials`([with_evc_tree])	Fetch non-completed trials of this `Experiment` instance.
`fetch_pending_trials`([with_evc_tree])	Fetch all trials with status new, interrupted or suspended
`fetch_trials`([with_evc_tree])	Fetch all trials of the experiment
`fetch_trials_by_status`(status[, with_evc_tree])	Fetch all trials with the given status
`fix_lost_trials`([with_evc_tree])	Find lost trials and set them to interrupted.
`get_trial`([trial, uid])	Fetch a single Trial, see `orion.storage.base.BaseStorageProtocol.get_trial()`
`register_trial`(trial[, status])	Register new trial in the database.
`reserve_trial`([score_handle])	Find new trials that exist currently in database and select one of them based on the highest score return from score_handle callable.
`retrieve_result`(trial, args, *kwargs)	See `orion.storage.base.BaseStorageProtocol.retrieve_result()`
`set_trial_status`(args, *kwargs)	See `orion.storage.base.BaseStorageProtocol.set_trial_status()`
`to_pandas`([with_evc_tree])	Builds a dataframe with the trials of the experiment
`update_completed_trial`(trial[, results_file])	Inform database about an evaluated trial with results.

acquire_algorithm_lock(timeout: int | float = 60, retry_interval: int | float = 1) → Generator[AlgoT, None, None][source]¶

Acquire lock on algorithm

This method should be called using a with-clause.

The context manager returns the algorithm object with its state updated based on the state loaded from storage.

Upon leaving the context manager, the new state of the algorithm is saved back to the storage before releasing the lock.

Parameters

timeout: int, optional: Timeout for the acquisition of the lock. If the lock is not obtained before timeout, then LockAcquisitionTimeout is raised. The timeout is only for the acquisition of the lock. Once the lock is obtained, it is valid until the context manager is closed. Default: 600.
retry_interval: int, optional: Sleep time between each attempts at acquiring the lock. Default: 1

Raises

RuntimeError: The algorithm configuration is different then the one during last execution of that same experiment.
orion.storage.base.LockAcquisitionTimeout: The lock could not be obtained in less than timeout seconds.

property configuration: ExperimentConfig¶: Return a copy of an Experiment configuration as a dictionary.

duplicate_pending_trials()[source]¶

Find pending trials in EVC and duplicate them in current experiment.

An experiment cannot execute trials from parent experiments otherwise some trials may have been executed in different environements of different experiment although they belong to the same experiment. Instead, trials that are pending in parent and child experiment are copied over to current experiment so that it can be reserved and executed. The parent or child experiment will only see their original copy of the trial, and the current experiment will only see the new copy of the trial.

fetch_lost_trials(with_evc_tree=False)[source]¶

Fetch all reserved trials that are lost (old heartbeat)

Trials are sorted based on Trial.submit_time

Returns: list of Trial objects

fetch_noncompleted_trials(with_evc_tree=False)[source]¶

Fetch non-completed trials of this Experiment instance.

Trials are sorted based on Trial.submit_time

Note

It will return all non-completed trials, including new, reserved, suspended, interrupted and broken ones.

Returns: list of non-completed Trial objects

fetch_pending_trials(with_evc_tree=False)[source]¶

Fetch all trials with status new, interrupted or suspended

Trials are sorted based on Trial.submit_time

Returns: list of Trial objects

fetch_trials(with_evc_tree=False) → list[Trial][source]¶: Fetch all trials of the experiment

fetch_trials_by_status(status, with_evc_tree=False)[source]¶

Fetch all trials with the given status

Trials are sorted based on Trial.submit_time

Returns: list of Trial objects

fix_lost_trials(with_evc_tree=True)[source]¶

Find lost trials and set them to interrupted.

A lost trial is defined as a trial whose heartbeat as not been updated since two times the wait time for monitoring. This usually means that the trial is stalling or has been interrupted in some way without its status being changed. This functions finds such trials and set them as interrupted so they can be launched again.

get_trial(trial=None, uid=None)[source]¶: Fetch a single Trial, see orion.storage.base.BaseStorageProtocol.get_trial()

property id¶

Id of the experiment in the database if configured.

Value is None if the experiment is not configured.

property is_broken¶

Return True, if this experiment is considered to be broken.

Count how many trials are broken and return True if that number has reached as given threshold.

property is_done¶

Return True, if this experiment is considered to be finished.

Count how many trials have been completed and compare with max_trials.
Ask algorithm if they consider there is a chance for further improvement, and verify is there is any pending trial.

Note

To be used as a terminating condition in a Worker.

property mode¶

Return the access right of the experiment

{‘r’: read, ‘w’: read/write, ‘x’: read/write/execute}

property node¶

Node of the experiment in the version control tree.

Value is None if the experiment is not connected to the version control tree.

property progress: float¶: Return a floating number between 0 and 1 representing experiment progress, or None if progress cannot be completed.

register_trial(trial, status='new')[source]¶

Register new trial in the database.

Inform database about new suggested trial with specific parameter values. Trials may only be registered one at a time to avoid registration of duplicates.

Parameters

trials: `Trial` object: Trial to register in the database

Raises

orion.core.io.database.DuplicateKeyError: If a trial with the same id already exist in the database. Since the id is computed based on a hashing of the trial, this should mean that an identical trial already exist in the database.

reserve_trial(score_handle=None) → Trial | None[source]¶

Find new trials that exist currently in database and select one of them based on the highest score return from score_handle callable.

Parameters

score_handle: callable object, optional: A way to decide which trial out of the new ones to to pick as reserved, defaults to a random choice. Deprecated

Returns

Selected Trial object, None if could not find any.

retrieve_result(trial, *args, **kwargs)[source]¶: See orion.storage.base.BaseStorageProtocol.retrieve_result()

set_trial_status(*args, **kwargs)[source]¶: See orion.storage.base.BaseStorageProtocol.set_trial_status()

property stats¶: Calculate orion.core.worker.experiment.ExperimentStats for this particular experiment.

property storage¶: Return the storage currently in use by this experiment

to_pandas(with_evc_tree=False)[source]¶

Builds a dataframe with the trials of the experiment

Parameters

with_evc_tree: bool, optional: Fetch all trials from the EVC tree. Default: False

update_completed_trial(trial, results_file=None)[source]¶

Inform database about an evaluated trial with results.

Parameters: trial (Trial) – Corresponds to a successful evaluation of a particular run.

Note

Change status from reserved to completed.

class orion.core.worker.experiment.ExperimentStats(trials_completed: int, best_trials_id: int, best_evaluation: float, start_time: ~datetime.datetime, finish_time: ~datetime.datetime, max_trials: int = 0, nb_trials: int = 0, progress: float = 0, trial_status_count: dict = <factory>, elapsed_time: ~datetime.timedelta = <factory>, sum_of_trials_time: ~datetime.timedelta = <factory>, eta: ~datetime.timedelta = <factory>, eta_milliseconds: float = 0)[source]¶

Parameters

trials_completed: int: Number of completed trials
best_trials_id: int: Unique identifier of the orion.core.worker.trial.Trial object in the database which achieved the best known objective result.
best_evaluation: float: Evaluation score of the best trial
start_time: `datetime.datetime`: When Experiment was first dispatched and started running.
finish_time: `datetime.datetime`: When Experiment reached terminating condition and stopped running.
elapsed_time: `datetime.timedelta`: Elapsed time.
max_trials: int: Experiment max_trials
nb_trials: int: Number of trials in experiment
progress: float: Experiment progression (between 0 and 1).
trial_status_count: Dict[str, int]: Dictionary mapping trial status to number of trials that have this status
sum_of_trials_time: `datetime.timedelta`: Sum of trial duration
eta: `datetime.timedelta`: Estimated remaining time
eta_milliseconds: float: ETA in milliseconds (used to get ETA in other programming languages, e.g. Javascript)

Methods

to_json()

Return a JSON-compatible dictionary of stats.

to_json()[source]¶: Return a JSON-compatible dictionary of stats.