Base definition of algorithms

class orion.algo.base.BaseAlgorithm(space, **kwargs)

Base class describing what an algorithm can do.

Parameters:

space : orion.algo.space.Space

Definition of a problem’s parameter space.

kwargs : dict

Tunable elements of a particular algorithm, a dictionary from hyperparameter names to values.

Notes

We are using the No Free Lunch theorem’s [R367dceedad15-1]_[R367dceedad15-3]_ formulation of an BaseAlgorithm. We treat it as a part of a procedure which in each iteration suggests a sample of the parameter space of the problem as a candidate solution and observes the results of its evaluation.

Developer Note: Each algorithm’s complete specification, i.e. implementation of its methods and parameters of its own, lies in a separate concrete algorithm class, which must be an immediate subclass of BaseAlgorithm. [The reason for this is current implementation of orion.core.utils.Factory metaclass which uses BaseAlgorithm.__subclasses__().] Second, one must declare an algorithm’s own parameters (tunable elements which could be set by configuration). This is done by passing them to BaseAlgorithm.__init__() by calling Python’s super with a Space object as a positional argument plus algorithm’s own parameters as keyword arguments. The keys of the keyword arguments passed to BaseAlgorithm.__init__() are interpreted as the algorithm’s parameter names. So for example, a subclass could be as simple as this (regarding the logistics, not an actual algorithm’s implementation):

References

[1]	D. H. Wolpert and W. G. Macready, “No Free Lunch Theorems for Optimization,” IEEE Transactions on Evolutionary Computation, vol. 1, no. 1, pp. 67–82, Apr. 1997.

[2]	W. G. Macready and D. H. Wolpert, “What Makes An Optimization Problem Hard?,” Complexity, vol. 1, no. 5, pp. 40–46, 1996.

[3]	D. H. Wolpert and W. G. Macready, “No Free Lunch Theorems for Search,” Technical Report SFI-TR-95-02-010, Santa Fe Institute, 1995.

Examples

from orion.algo.base import BaseAlgorithm
from orion.algo.space import (Integer, Space)

class MySimpleAlgo(BaseAlgorithm):

    def __init__(self, space, multiplier=1, another_param="a string param"):
        super().__init__(space, multiplier=multiplier, another_param=another_param)

    def suggest(self, num=1):
        print(self.another_param)
        return list(map(lambda x: tuple(map(lambda y: self.multiplier * y, x)),
                        self.space.sample(num)))

    def observe(self, points, results):
        pass

dim = Integer('named_param', 'norm', 3, 2, shape=(2, 3))
s = Space()
s.register(dim)

algo = MySimpleAlgo(s, 2, "I am just sampling!")
algo.suggest()

Attributes:

configuration

Return tunable elements of this algorithm in a dictionary form appropriate for saving.

is_done

Return True, if an algorithm holds that there can be no further improvement.

requires_dist

requires_shape

requires_type

should_suspend

Allow algorithm to decide whether a particular running trial is still worth to complete its evaluation, based on information provided by the judge method.

space

Domain of problem associated with this algorithm’s instance.

state_dict

Return a state dict that can be used to reset the state of the algorithm.

Methods

judge(point, measurements)	Inform an algorithm about online measurements of a running trial.
observe(points, results)	Observe the results of the evaluation of the points in the process defined in user’s script.
score(point)	Allow algorithm to evaluate point based on a prediction about this parameter set’s performance.
seed_rng(seed)	Seed the state of the random number generator.
set_state(state_dict)	Reset the state of the algorithm based on the given state_dict
suggest([num])	Suggest a num of new sets of parameters.

configuration

Return tunable elements of this algorithm in a dictionary form appropriate for saving.

is_done

Return True, if an algorithm holds that there can be no further improvement. By default, the cardinality of the specified search space will be used to check if all possible sets of parameters has been tried.

judge(point, measurements)

Inform an algorithm about online measurements of a running trial.

Parameters:point – A tuple which specifies the values of the (hyper)parameters used to execute user’s script with.

This method is to be used as a callback in a client-server communication between user’s script and a orion’s worker using a BaseAlgorithm. Data returned from this method must be serializable and will be used as a response to the running environment. Default response is None.

Note

Calling algorithm to judge a point based on its online measurements will effectively change a state in the algorithm (like a reinforcement learning agent’s hidden state or an automatic early stopping mechanism’s regression), which it may change the value of the property should_suspend.

Returns:None or a serializable dictionary containing named data

observe(points, results)

Observe the results of the evaluation of the points in the process defined in user’s script.

Parameters:

points : list of tuples of array-likes

Points from a orion.algo.space.Space. Evaluated problem parameters by a consumer.

results : list of dicts

Contains the result of an evaluation; partial information about the black-box function at each point in params.

score(point)

Allow algorithm to evaluate point based on a prediction about this parameter set’s performance.

By default, return the same score any parameter (no preference).

Returns:A subjective measure of expected perfomance. Return type:float

seed_rng(seed)

Seed the state of the random number generator.

Parameters:seed – Integer seed for the random number generator.

Note

This methods does nothing if the algorithm is deterministic.

set_state(state_dict)

Reset the state of the algorithm based on the given state_dict

Parameters:state_dict – Dictionary representing state of an algorithm

should_suspend

Allow algorithm to decide whether a particular running trial is still worth to complete its evaluation, based on information provided by the judge method.

space

Domain of problem associated with this algorithm’s instance.

state_dict

Return a state dict that can be used to reset the state of the algorithm.

suggest(num=1)

Suggest a num of new sets of parameters.

Parameters:

num: int, optional

Number of points to suggest. Defaults to 1.

Returns:

list of points or None

A list of lists representing points suggested by the algorithm. The algorithm may opt out if it cannot make a good suggestion at the moment (it may be waiting for other trials to complete), in which case it will return None.

Notes

New parameters must be compliant with the problem’s domain orion.algo.space.Space.