Adapters for trials compatibility between experiments

Adapters to connect experiments within the EVC system

Experiment branches because of changes in the configuration or the user’s code. This make experiments incompatible with one another unless we define adapters such that a branched experiment B can access trials from parent experiment A.

Adapters have two main methods, forward and backward. The method forward defines how trials from parent experiment A are filtered or adapted to be compatible with experiment B. Modifications are only applied at execution time and are not saved anywhere in the database. Adapters only provides a view on an experiment.

There is adapters for

  • Dimension addition

  • Dimension deletion

  • Dimension renaming

  • Change of dimension prior

  • Change of algorithm

  • Change of code

  • Combining different adapters

Adapters all have a to_dict method which provides sufficient information to rebuild the adapter. This is to facilitate save of adapters in a database and retrieval.

Adapters can be build using the factory class Adapter(**kwargs) or using Adapter.build(list_of_dicts).

class orion.core.evc.adapters.AlgorithmChange[source]

Adapter for changes in algorithm definition

Note

Current implementation does nothing

Methods

backward(trials)

Pass all trials from child experiment to parent experiment

forward(trials)

Pass all trials from parent experiment to child experiment

to_dict()

Provide the configuration of the adapter as a dictionary

backward(trials)[source]

Pass all trials from child experiment to parent experiment

forward(trials)[source]

Pass all trials from parent experiment to child experiment

to_dict()[source]

Provide the configuration of the adapter as a dictionary

class orion.core.evc.adapters.BaseAdapter[source]

Base class describing what an adapter can do.

Attributes
configuration

Provide the configuration of the adapter.

Methods

backward(trials)

Adapt trials of the child experiment such that they are compatible to the parent experiment

build(adapter_dicts)

Builder method for a list of adapters.

forward(trials)

Adapt trials of the parent experiment such that they are compatible to the child experiment

to_dict()

Provide the configuration of the adapter as a dictionary

abstract backward(trials)[source]

Adapt trials of the child experiment such that they are compatible to the parent experiment

Parameters
trials: list of :class:`orion.core.worker.trial.Trial`

List of orion.core.worker.trial.Trial coming from the child experiment

classmethod build(adapter_dicts)[source]

Builder method for a list of adapters.

Parameters
adapter_dicts: list of `dict`

List of adapter representation in dictionary form as expected to be saved in a database.

Returns
orion.core.evc.adapters.CompositeAdapter

An adapter which may contain many adapters

property configuration

Provide the configuration of the adapter.

For simplicity, the configuration is always returned as if the adapter is a CompositeAdapter, therefore no matter if the adapter is composite or not it will return a list of configurations.

The dictionaries of the list can be used to recreate adapters, for any adapter class.

Returns
dict

Configuration as a dictionary

Examples

1 configuration = a_dummy_adapter.configuration[0]
2 another_dummy_adapter = Adapter.build([configuration])
3 assert another_dummy_adapter.configuration[0] == configuration
abstract forward(trials)[source]

Adapt trials of the parent experiment such that they are compatible to the child experiment

Parameters
trials: list of :class:`orion.core.worker.trial.Trial`

List of orion.core.worker.trial.Trial coming from the parent experiment

abstract to_dict()[source]

Provide the configuration of the adapter as a dictionary

This method is intended for single adapters only. For coherence, method configuration is used by all adapters, which is a list of dictionary configurations.

class orion.core.evc.adapters.CodeChange(change_type)[source]

Adapter which filters parent’s trials based on the type of code change

This adapter let pass all trials if the code change didn’t break the compatibility with parent’s experiment. If the effect of the change is UNSURE, the trials may only pass from parent to child but not from child to parent. This is to ensure parent experiment does not get corrupted with possibly incompatible results.

On forward, the adapter filters out parent’s trials if type of code change is BREAK. On backward, the adapter filters out child’s trials if type of code change is UNSURE or BREAK.

Attributes
change_type: `str`

Type of change of the code. Can be one of CodeChange.NOEFFET, CodeChange.BREAK or CodeChange.UNSURE.

Methods

backward(trials)

Filter out child's trials if type of code change is UNSURE or BREAK.

forward(trials)

Filter out parent's trials if type of code change is BREAK.

to_dict()

Provide the configuration of the adapter as a dictionary

validate(change_type)

Validate change type and raise ValueError if invalid

backward(trials)[source]

Filter out child’s trials if type of code change is UNSURE or BREAK.

forward(trials)[source]

Filter out parent’s trials if type of code change is BREAK.

to_dict()[source]

Provide the configuration of the adapter as a dictionary

classmethod validate(change_type)[source]

Validate change type and raise ValueError if invalid

class orion.core.evc.adapters.CommandLineChange(change_type)[source]

Adapter which filters parent’s trials based on the type of command line change

This adapter let pass all trials if the command line change didn’t break the compatibility with parent’s experiment. If the effect of the change is UNSURE, the trials may only pass from parent to child but not from child to parent. This is to ensure parent experiment does not get corrupted with possibly incompatible results.

On forward, the adapter filters out parent’s trials if type of change is BREAK. On backward, the adapter filters out child’s trials if type of change is UNSURE or BREAK.

Attributes
change_type: `str`

Type of change of the command line. Can be one of CommandLineChange.NOEFFET, CommandLineChange.BREAK or CommandLineChange.UNSURE.

Methods

backward(trials)

Filter out child's trials if type of cli change is UNSURE or BREAK.

forward(trials)

Filter out parent's trials if type of cli change is BREAK.

to_dict()

Provide the configuration of the adapter as a dictionary

validate(change_type)

Validate change type and raise ValueError if invalid

backward(trials)[source]

Filter out child’s trials if type of cli change is UNSURE or BREAK.

forward(trials)[source]

Filter out parent’s trials if type of cli change is BREAK.

to_dict()[source]

Provide the configuration of the adapter as a dictionary

classmethod validate(change_type)[source]

Validate change type and raise ValueError if invalid

class orion.core.evc.adapters.CompositeAdapter(*adapters)[source]

Adapter which group many other adapters needed to connect two experiments

Attributes
adapters: instances of `orion.core.evc.adapters.BaseAdapter`

List of adaptors which are applied sequentially

Methods

backward(trials)

Apply the adaptors backward and in reverse order on the parent's trials

forward(trials)

Apply the adaptors on the parent's trials

to_dict()

Return nothing since it is not valid for CompositeAdapter

backward(trials)[source]

Apply the adaptors backward and in reverse order on the parent’s trials

property configuration

Provide the configuration of the adapter.

forward(trials)[source]

Apply the adaptors on the parent’s trials

to_dict()[source]

Return nothing since it is not valid for CompositeAdapter

class orion.core.evc.adapters.DimensionAddition(param)[source]

Adapter which adds a new dimension to parent’s trials

This adaptation is based on the assumption that the trials of the parent experiment are equivalent if we add the new dimension for a given default value.

On forward, the adapter add a dimension with provided default value to each trials. On backward, the adapter filters trials and only keep with with the default value.

Attributes
param: instances of `orion.core.worker.trial.Trial.Param`

A parameter object which defines the name and default value of the name dimension.

Methods

backward(trials)

Filter out trials which have values different than the default one.

forward(trials)

Add a dimension with provided default value to each trials of the parent

to_dict()

Provide the configuration of the adapter as a dictionary

backward(trials)[source]

Filter out trials which have values different than the default one.

forward(trials)[source]

Add a dimension with provided default value to each trials of the parent

to_dict()[source]

Provide the configuration of the adapter as a dictionary

class orion.core.evc.adapters.DimensionDeletion(param)[source]

Adapter which remove a dimension to parent’s trials

Note

This adapter is the opposite of orion.core.evc.adapters.DimensionAddition.

This adaptation is based on the assumption that the trials of the children experiment are equivalent if we remove the new dimension for a given default value.

On forward, the adapter filters trials and only keep with with the default value. On backward, the adapter add a dimension with provided default value to each trials.

Attributes
dimension_addition_adapter: instance of `orion.core.evc.adapters.DimensionAddition`

An adapter to add a new dimension, it is used by DimensionDeletion inversely to remove a dimension.

Methods

backward(trials)

Add a dimension with provided default value to each trials of the child.

forward(trials)

Filter out trials which have values different than the default one.

to_dict()

Provide the configuration of the adapter as a dictionary

backward(trials)[source]

Add a dimension with provided default value to each trials of the child.

forward(trials)[source]

Filter out trials which have values different than the default one.

property param

Parameter containing name and default value

to_dict()[source]

Provide the configuration of the adapter as a dictionary

class orion.core.evc.adapters.DimensionPriorChange(name, old_prior, new_prior)[source]

Adapter which filters parent’s trials based on a new prior

On forward, the adapter filters out trials based on the child’s prior. On backward, the adapter filters out trials based on the parent’s prior.

Attributes
name: `str`

Name of the dimension.

old_prior: `str`

string definition as parsable by orion.core.io.space_builder.

new_prior: `str`

string definition as parsable by orion.core.io.space_builder.

old_dimension: instance of `orion.algo.space.Dimension`

The dimension of the parent experiment

new_dimension: instance of `orion.algo.space.Dimension`

The dimension of the child experiment

Methods

backward(trials)

Filter out trials which have out of bound values based on the parent's prior.

forward(trials)

Filter out trials which have out of bound values based on the child's prior.

to_dict()

Provide the configuration of the adapter as a dictionary

backward(trials)[source]

Filter out trials which have out of bound values based on the parent’s prior.

forward(trials)[source]

Filter out trials which have out of bound values based on the child’s prior.

to_dict()[source]

Provide the configuration of the adapter as a dictionary

class orion.core.evc.adapters.DimensionRenaming(old_name, new_name)[source]

Adapter which change the name of a dimension in parent’s trials

On forward, the adapter change dimensions name from A to B. On backward, the adapter change dimensions name from B to A.

Attributes
old_name: `str`

Name of the parent’s dimension.

new_name: `str`

Name of the child’s dimension.

Methods

backward(trials)

Change name of dimension new_name to old_name.

forward(trials)

Change name of dimension old_name to new_name.

to_dict()

Provide the configuration of the adapter as a dictionary

backward(trials)[source]

Change name of dimension new_name to old_name.

forward(trials)[source]

Change name of dimension old_name to new_name.

to_dict()[source]

Provide the configuration of the adapter as a dictionary

class orion.core.evc.adapters.OrionVersionChange[source]

Adapter for changes of Oríon version

Note

Does nothing…

Methods

backward(trials)

Pass all trials from child experiment to parent experiment

forward(trials)

Pass all trials from parent experiment to child experiment

to_dict()

Provide the configuration of the adapter as a dictionary

backward(trials)[source]

Pass all trials from child experiment to parent experiment

forward(trials)[source]

Pass all trials from parent experiment to child experiment

to_dict()[source]

Provide the configuration of the adapter as a dictionary

class orion.core.evc.adapters.ScriptConfigChange(change_type)[source]

Adapter which filters parent’s trials based on the type of user’s script’s config change

This adapter let pass all trials if the change in the user’s script’s configuration file didn’t break the compatibility with parent’s experiment. If the effect of the change is UNSURE, the trials may only pass from parent to child but not from child to parent. This is to ensure parent experiment does not get corrupted with possibly incompatible results.

On forward, the adapter filters out parent’s trials if type of change is BREAK. On backward, the adapter filters out child’s trials if type of change is UNSURE or BREAK.

Attributes
change_type: `str`

Type of change of the command line. Can be one of ScriptConfigChange.NOEFFET, ScriptConfigChange.BREAK or ScriptConfigChange.UNSURE.

Methods

backward(trials)

Filter out child's trials if type of script's config change is UNSURE or BREAK.

forward(trials)

Filter out parent's trials if type of script's config change is BREAK.

to_dict()

Provide the configuration of the adapter as a dictionary

validate(change_type)

Validate change type and raise ValueError if invalid

backward(trials)[source]

Filter out child’s trials if type of script’s config change is UNSURE or BREAK.

forward(trials)[source]

Filter out parent’s trials if type of script’s config change is BREAK.

to_dict()[source]

Provide the configuration of the adapter as a dictionary

classmethod validate(change_type)[source]

Validate change type and raise ValueError if invalid

orion.core.evc.adapters.apply_if_valid(name, trial, callback=None, raise_if_not=True)[source]

Detect a parameter in trial and call a callback on it if provided

Parameters
name: str

Name of the param to look for

trial: `orion.core.worker.trial.Trial`

Instance of trial to investigate

callback: None of callable

Function to call with (trial, param) with a parameter is found. Defaults to None

raise_if_not: bool

raises RuntimeError if no parameter is found. Defaults to True.

Returns
bool

False if parameter is not found and raise_if_not is False. True if parameter is found and callback is None. Else, output of callback(trial, item).