Experiment¶
Description of an optimization attempt¶
Manage history of trials corresponding to a black box process.
- class orion.core.worker.experiment.Experiment(name, version=None, mode='r')[source]¶
Represents an entry in database/experiments collection.
- Attributes
- namestr
Unique identifier for this experiment per
user
.- id: object
id of the experiment in the database if experiment is configured. Value is
None
if the experiment is not configured.- refersdict or list of
Experiment
objects, after initialization is done. A dictionary pointing to a past
Experiment
id,refers[parent_id]
, whose trials we want to add in the history of completed trials we want to re-use. For convenience and database effiency purpose, all experiments of a common tree sharesrefers[root_id]
, with the root experiment refering to itself.- version: int
Current version of this experiment.
- metadatadict
Contains managerial information about this
Experiment
.- pool_sizeint
How many workers can participate asynchronously in this
Experiment
.- max_trialsint
How many trials must be evaluated, before considering this
Experiment
done. This attribute can be updated if the rest of the experiment configuration is the same. In that case, if trying to set to an already set experiment, it will overwrite the previous one.- max_broken: int
How many trials must be broken, before considering this
Experiment
broken. This attribute can be updated if the rest of the experiment configuration is the same. In that case, if trying to set to an already set experiment, it will overwrite the previous one.- space: Space
Object representing the optimization space.
- algorithmsPrimaryAlgo object.
Complete specification of the optimization and dynamical procedures taking place in this
Experiment
.
Methods
Find pending trials in EVC and duplicate them in current experiment.
fetch_lost_trials
([with_evc_tree])Fetch all reserved trials that are lost (old heartbeat)
fetch_noncompleted_trials
([with_evc_tree])Fetch non-completed trials of this
Experiment
instance.fetch_pending_trials
([with_evc_tree])Fetch all trials with status new, interrupted or suspended
fetch_trials
([with_evc_tree])Fetch all trials of the experiment
fetch_trials_by_status
(status[, with_evc_tree])Fetch all trials with the given status
fix_lost_trials
([with_evc_tree])Find lost trials and set them to interrupted.
get_trial
([trial, uid])Fetch a single Trial, see
orion.storage.base.BaseStorageProtocol.get_trial()
register_lie
(lying_trial)Register a fake trial created by the strategist.
register_trial
(trial[, status])Register new trial in the database.
reserve_trial
([score_handle])Find new trials that exist currently in database and select one of them based on the highest score return from score_handle callable.
retrieve_result
(trial, *args, **kwargs)See
orion.storage.base.BaseStorageProtocol.retrieve_result()
set_trial_status
(*args, **kwargs)See
orion.storage.base.BaseStorageProtocol.set_trial_status()
to_pandas
([with_evc_tree])Builds a dataframe with the trials of the experiment
update_completed_trial
(trial[, results_file])Inform database about an evaluated trial with results.
- property configuration¶
Return a copy of an
Experiment
configuration as a dictionary.
- duplicate_pending_trials()[source]¶
Find pending trials in EVC and duplicate them in current experiment.
An experiment cannot execute trials from parent experiments otherwise some trials may have been executed in different environements of different experiment although they belong to the same experiment. Instead, trials that are pending in parent and child experiment are copied over to current experiment so that it can be reserved and executed. The parent or child experiment will only see their original copy of the trial, and the current experiment will only see the new copy of the trial.
- fetch_lost_trials(with_evc_tree=False)[source]¶
Fetch all reserved trials that are lost (old heartbeat)
Trials are sorted based on Trial.submit_time
- Returns
list of Trial objects
- fetch_noncompleted_trials(with_evc_tree=False)[source]¶
Fetch non-completed trials of this
Experiment
instance.Trials are sorted based on Trial.submit_time
Note
It will return all non-completed trials, including new, reserved, suspended, interrupted and broken ones.
- Returns
list of non-completed Trial objects
- fetch_pending_trials(with_evc_tree=False)[source]¶
Fetch all trials with status new, interrupted or suspended
Trials are sorted based on Trial.submit_time
- Returns
list of Trial objects
- fetch_trials_by_status(status, with_evc_tree=False)[source]¶
Fetch all trials with the given status
Trials are sorted based on Trial.submit_time
- Returns
list of Trial objects
- fix_lost_trials(with_evc_tree=True)[source]¶
Find lost trials and set them to interrupted.
A lost trial is defined as a trial whose heartbeat as not been updated since two times the wait time for monitoring. This usually means that the trial is stalling or has been interrupted in some way without its status being changed. This functions finds such trials and set them as interrupted so they can be launched again.
- get_trial(trial=None, uid=None)[source]¶
Fetch a single Trial, see
orion.storage.base.BaseStorageProtocol.get_trial()
- property id¶
Id of the experiment in the database if configured.
Value is None if the experiment is not configured.
- property is_broken¶
Return True, if this experiment is considered to be broken.
Count how many trials are broken and return True if that number has reached as given threshold.
- property is_done¶
Return True, if this experiment is considered to be finished.
Count how many trials have been completed and compare with
max_trials
.Ask
algorithms
if they consider there is a chance for further improvement, and verify is there is any pending trial.
Note
To be used as a terminating condition in a
Worker
.
- property mode¶
Return the access right of the experiment
{‘r’: read, ‘w’: read/write, ‘x’: read/write/execute}
- property node¶
Node of the experiment in the version control tree.
Value is None if the experiment is not connected to the version control tree.
- register_lie(lying_trial)[source]¶
Register a fake trial created by the strategist.
The main difference between fake trial and orignal ones is the addition of a fake objective result, and status being set to completed. The id of the fake trial is different than the id of the original trial, but the original id can be computed using the hashcode on parameters of the fake trial. See
orion.core.worker.strategy
for more information and the Strategist object and generation of fake trials.- Parameters
- trials: `Trial` object
Fake trial to register in the database
- Raises
- orion.core.io.database.DuplicateKeyError
If a trial with the same id already exist in the database. Since the id is computed based on a hashing of the trial, this should mean that an identical trial already exist in the database.
- register_trial(trial, status='new')[source]¶
Register new trial in the database.
Inform database about new suggested trial with specific parameter values. Trials may only be registered one at a time to avoid registration of duplicates.
- Parameters
- trials: `Trial` object
Trial to register in the database
- Raises
- orion.core.io.database.DuplicateKeyError
If a trial with the same id already exist in the database. Since the id is computed based on a hashing of the trial, this should mean that an identical trial already exist in the database.
- reserve_trial(score_handle=None)[source]¶
Find new trials that exist currently in database and select one of them based on the highest score return from score_handle callable.
- Parameters
- score_handle: callable object, optional
A way to decide which trial out of the new ones to to pick as reserved, defaults to a random choice. Deprecated
- Returns
- Selected Trial object, None if could not find any.
- retrieve_result(trial, *args, **kwargs)[source]¶
See
orion.storage.base.BaseStorageProtocol.retrieve_result()
- set_trial_status(*args, **kwargs)[source]¶
See
orion.storage.base.BaseStorageProtocol.set_trial_status()
- property stats¶
Calculate a stats dictionary for this particular experiment.
- Returns
- statsdict