Base definition of algorithms¶
-
class
orion.algo.base.
BaseAlgorithm
(space, **kwargs)[source]¶ Base class describing what an algorithm can do.
- Parameters
- space
orion.algo.space.Space
Definition of a problem’s parameter space.
- kwargsdict
Tunable elements of a particular algorithm, a dictionary from hyperparameter names to values.
- space
Notes
We are using the No Free Lunch theorem’s [R367dceedad15-1]_[R367dceedad15-3]_ formulation of an
BaseAlgorithm
. We treat it as a part of a procedure which in each iteration suggests a sample of the parameter space of the problem as a candidate solution and observes the results of its evaluation.Developer Note: Each algorithm’s complete specification, i.e. implementation of its methods and parameters of its own, lies in a separate concrete algorithm class, which must be an immediate subclass of
BaseAlgorithm
. [The reason for this is current implementation oforion.core.utils.Factory
metaclass which uses BaseAlgorithm.__subclasses__().] Second, one must declare an algorithm’s own parameters (tunable elements which could be set by configuration). This is done by passing them to BaseAlgorithm.__init__() by calling Python’s super with a Space object as a positional argument plus algorithm’s own parameters as keyword arguments. The keys of the keyword arguments passed to BaseAlgorithm.__init__() are interpreted as the algorithm’s parameter names. So for example, a subclass could be as simple as this (regarding the logistics, not an actual algorithm’s implementation):References
- 1
D. H. Wolpert and W. G. Macready, “No Free Lunch Theorems for Optimization,” IEEE Transactions on Evolutionary Computation, vol. 1, no. 1, pp. 67–82, Apr. 1997.
- 2
W. G. Macready and D. H. Wolpert, “What Makes An Optimization Problem Hard?,” Complexity, vol. 1, no. 5, pp. 40–46, 1996.
- 3
D. H. Wolpert and W. G. Macready, “No Free Lunch Theorems for Search,” Technical Report SFI-TR-95-02-010, Santa Fe Institute, 1995.
Examples
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
from orion.algo.base import BaseAlgorithm from orion.algo.space import (Integer, Space) class MySimpleAlgo(BaseAlgorithm): def __init__(self, space, multiplier=1, another_param="a string param"): super().__init__(space, multiplier=multiplier, another_param=another_param) def suggest(self, num=1): print(self.another_param) return list(map(lambda x: tuple(map(lambda y: self.multiplier * y, x)), self.space.sample(num))) def observe(self, points, results): pass dim = Integer('named_param', 'norm', 3, 2, shape=(2, 3)) s = Space() s.register(dim) algo = MySimpleAlgo(s, 2, "I am just sampling!") algo.suggest()
- Attributes
configuration
Return tunable elements of this algorithm in a dictionary form appropriate for saving.
fidelity_index
Compute the index of the point where fidelity is.
is_done
Whether the algorithm is done and will not make further suggestions.
n_observed
Number of completed trials observed by the algorithm
n_suggested
Number of trials suggested by the algorithm
- requires_dist
- requires_shape
- requires_type
should_suspend
Allow algorithm to decide whether a particular running trial is still worth to complete its evaluation, based on information provided by the
judge
method.space
Domain of problem associated with this algorithm’s instance.
state_dict
Return a state dict that can be used to reset the state of the algorithm.
Methods
format_point
(point)Format point based on space transformations
get_id
(point[, ignore_fidelity])Compute a unique hash for a point based on params
has_observed
(point)Whether the algorithm has observed a given point objective.
has_suggested
(point)Whether the algorithm has suggested a given point.
judge
(point, measurements)Inform an algorithm about online measurements of a running trial.
observe
(points, results)Observe the results of the evaluation of the points in the process defined in user’s script.
register
(point[, result])Save the point as one suggested or observed by the algorithm
score
(point)Allow algorithm to evaluate point based on a prediction about this parameter set’s performance.
seed_rng
(seed)Seed the state of the random number generator.
set_state
(state_dict)Reset the state of the algorithm based on the given state_dict
suggest
(num)Suggest a num of new sets of parameters.
-
property
configuration
¶ Return tunable elements of this algorithm in a dictionary form appropriate for saving.
-
property
fidelity_index
¶ Compute the index of the point where fidelity is.
Returns None if there is no fidelity dimension.
-
format_point
(point)[source]¶ Format point based on space transformations
This will apply the reverse transformation on the point and then transform it again.
Some transformations are lossy and thus the points suggested by the algorithm could be different when returned to
observe
. Usingformat_point
makes it possible for the algorithm to see the final version of the point after back and forth transformations. This way it can recognise the point inobserve
and also avoid duplicates that would have gone unnoticed during suggestion.- Parameters
- pointtuples of array-likes
Points from a
orion.algo.space.Space
.
-
get_id
(point, ignore_fidelity=False)[source]¶ Compute a unique hash for a point based on params
- Parameters
- pointtuples of array-likes
Points from a
orion.algo.space.Space
.- ignore_fidelity: bool, optional
If True, the fidelity dimension is ignored when computing a unique hash for the trial. Defaults to False.
-
has_observed
(point)[source]¶ Whether the algorithm has observed a given point objective.
This only counts observed completed trials.
- Parameters
- pointtuples of array-likes
Points from a
orion.algo.space.Space
.
- Returns
- bool
True if the point’s objective was observed by the algo, False otherwise.
-
has_suggested
(point)[source]¶ Whether the algorithm has suggested a given point.
- Parameters
- pointtuples of array-likes
Points from a
orion.algo.space.Space
.
- Returns
- bool
True if the point was suggested by the algo, False otherwise.
-
property
is_done
¶ Whether the algorithm is done and will not make further suggestions.
Return True, if an algorithm holds that there can be no further improvement. By default, the cardinality of the specified search space will be used to check if all possible sets of parameters has been tried.
-
judge
(point, measurements)[source]¶ Inform an algorithm about online measurements of a running trial.
- Parameters
point – A tuple which specifies the values of the (hyper)parameters used to execute user’s script with.
This method is to be used as a callback in a client-server communication between user’s script and a orion’s worker using a
BaseAlgorithm
. Data returned from this method must be serializable and will be used as a response to the running environment. Default response is None.Note
Calling algorithm to
judge
a point based on its online measurements will effectively change a state in the algorithm (like a reinforcement learning agent’s hidden state or an automatic early stopping mechanism’s regression), which it may change the value of the propertyshould_suspend
.- Returns
None or a serializable dictionary containing named data
-
property
n_observed
¶ Number of completed trials observed by the algorithm
-
property
n_suggested
¶ Number of trials suggested by the algorithm
-
observe
(points, results)[source]¶ Observe the results of the evaluation of the points in the process defined in user’s script.
- Parameters
- pointslist of tuples of array-likes
Points from a
orion.algo.space.Space
.- resultslist of dicts
Contains the result of an evaluation; partial information about the black-box function at each point in params.
-
register
(point, result=None)[source]¶ Save the point as one suggested or observed by the algorithm
- Parameters
- pointarray-likes
Point from a
orion.algo.space.Space
.- resultdict or None, optional
The result of an evaluation; partial information about the black-box function at each point in params. None is suggested and not yet completed.
-
score
(point)[source]¶ Allow algorithm to evaluate point based on a prediction about this parameter set’s performance.
By default, return the same score any parameter (no preference).
- Returns
A subjective measure of expected perfomance.
- Return type
-
seed_rng
(seed)[source]¶ Seed the state of the random number generator.
- Parameters
seed – Integer seed for the random number generator.
Note
This methods does nothing if the algorithm is deterministic.
-
set_state
(state_dict)[source]¶ Reset the state of the algorithm based on the given state_dict
- Parameters
state_dict – Dictionary representing state of an algorithm
-
property
should_suspend
¶ Allow algorithm to decide whether a particular running trial is still worth to complete its evaluation, based on information provided by the
judge
method.
-
property
space
¶ Domain of problem associated with this algorithm’s instance.
-
property
state_dict
¶ Return a state dict that can be used to reset the state of the algorithm.
-
abstract
suggest
(num)[source]¶ Suggest a num of new sets of parameters.
- Parameters
- num: int
Number of points to suggest. The algorithm may return less than the number of points requested.
- Returns
- list of points or None
A list of lists representing points suggested by the algorithm. The algorithm may opt out if it cannot make a good suggestion at the moment (it may be waiting for other trials to complete), in which case it will return None.
Notes
New parameters must be compliant with the problem’s domain
orion.algo.space.Space
.