Algorithms

Generic tests for Algorithms

class orion.testing.algo.BaseAlgoTests[source]

Generic Test-suite for HPO algorithms.

This test-suite covers all typical cases for HPO algorithms. To use it for a new algorithm, the class inheriting from this one must redefine the attributes algo_name with the name of the algorithm used to create it with the algorithm factory orion.core.worker.primary_algo.SpaceTransform and config with a base configuration for the algorithm that contains all its arguments. The base space can be redefine if needed with the attribute space.

Most algorithms have different phases that should be tested. For instance TPE has a first phase of random search and a second of Bayesian Optimization. The random search and Bayesian optimization phases use different logic and should both be tested. The phases class attribute can be set to parametrize all tests with each phase. See tests/unittests/algo/test_tpe.py for an example.

Attributes
algo_name

Methods

assert_dim_type_supported(test_space)

Test that a given dimension type is properly supported by the algorithm

create_algo([config, space, seed, ...])

Create the algorithm based on config.

create_space([space])

Create the space object

force_observe(num, algo[, seed])

Force observe num trials.

get_num(num)

Force number of trials to suggest

observe_trials(trials, algo, rng)

Make the algorithm observe trials

set_phases(phases)

Parametrize the tests with different phases.

test_broken_trials()

Test that algorithm can handle broken trials

test_cat_data()

Test that algorithm supports categorical dimensions

test_configuration()

Test that configuration property attribute contains all class arguments.

test_get_id()

Test that the id hashing is valid

test_has_observed()

Verify that algorithm detects correctly if a trial was observed

test_has_observed_statedict()

Verify that algorithm detects correctly if a trial was observed even when state was restored.

test_has_suggested()

Verify that algorithm detects correctly if a trial was suggested

test_has_suggested_statedict()

Verify that algorithm detects correctly if a trial was suggested even when state was restored.

test_int_data()

Test that algorithm supports integer dimensions

test_is_done_cardinality()

Test that algorithm will stop when cardinality is reached

test_is_done_max_trials()

Test that algorithm will stop when max trials is reached

test_logint_data()

Test that algorithm supports loginteger dimensions

test_logreal_data()

Test that algorithm supports logreal dimensions

test_n_observed()

Verify that algorithm returns correct number of observed trials

test_n_suggested()

Verify that algorithm returns correct number of suggested trials

test_observe()

Verify that algorithm observes trial without any issues

test_optimize_branin()

Test that algorithm optimizes a simple task comparably to random search.

test_real_data()

Test that algorithm supports real dimensions

test_seed_rng(seed)

Test that the seeding gives reproducible results.

test_seed_rng_init()

Test that if the algo has a seed constructor argument and a value is passed, the suggested trials are reproducible.

test_shape_data()

Test that algorithm supports dimensions with shape

test_state_dict(seed, phase)

Verify that resetting state makes sampling deterministic.

test_suggest_n()

Verify that suggest returns correct number of trials if num is specified in suggest.

update_space(test_space)

Get complete space configuration with partial overwrite

first_phase

last_phase

algo_type: type[AlgoType]

The type of algorithm under test.

assert_dim_type_supported(test_space: dict)[source]

Test that a given dimension type is properly supported by the algorithm

This will test that the algorithm sample trials valid for the given type and that the algorithm can observe these trials.

Parameters
test_space: the search space of the test.
classmethod create_algo(config: dict | None = None, space: Space | None = None, seed: int | Sequence[int] | None = None, n_observed_trials: int | None = None, **kwargs)[source]

Create the algorithm based on config.

Also initializes the algorithm with the required number of random trials from the previous test phases before returning it.

Parameters
config: dict, optional

The configuration for the algorithm. cls.config will be used if config is None.

space: ``orion.algo.space.Space``, optional

Space object to pass to algo. The output of cls.create_space() will be used if space is None.

seed: int | Sequence[int], optional

When passed, seed_rng is called before observing anything.

n_observed_trials: int | None, optional

Number of trials that the algorithm should have already observed when returned. When None (default), observes the number of trials at which the current phase begins. When set to 0, the algorithm will be freshly initialized.

kwargs: dict

Values to override algorithm configuration.

classmethod create_space(space: dict | None = None)[source]

Create the space object

Parameters
space: dict, optional

Configuration of the search space. The default self.space will be used if space is None.

classmethod force_observe(num: int, algo: BaseAlgorithm, seed: int = 1)[source]

Force observe num trials.

Parameters
num: int

Number of trials to suggest and observe.

algo: ``orion.algo.base.BaseAlgorithm``

The algorithm that must suggest and observe.

seed: int, optional

The seed used to generate random objectives

Raises
RuntimeError
  • If the algorithm returns duplicates. Algorithms may return duplicates across workers, but in sequential scenarios as here, it should not happen.

  • If the algorithm fails to sample any trial at least 5 times.

classmethod get_num(num: int)[source]

Force number of trials to suggest

Some algorithms must be tested with specific number of suggests at a time (ex: ASHA). This method can be overridden to change num based on the special needs.

TODO: Remove this or give it a better name.

classmethod observe_trials(trials: list[Trial], algo: BaseAlgorithm, rng: numpy.random.RandomState)[source]

Make the algorithm observe trials

Parameters
trials: list of ``orion.core.worker.trial.Trial``
algo: ``orion.algo.base.BaseAlgorithm``
rng: ``numpy.random.RandomState``

Random number generator to generate random objectives.

phases: ClassVar[list[TestPhase]] = [TestPhase(name='default', n_trials=0, method_to_spy='sample')]

Test phases for the algorithms. Overwrte this if the algorithm has more than one phase.

classmethod set_phases(phases: Sequence[TestPhase])[source]

Parametrize the tests with different phases.

Some algorithms have different phases that should be tested. For instance TPE have a first phase of random search and a second of Bayesian Optimization. The random search and Bayesian optimization are different implementations and both should be tested.

Parameters
phases: list of tuples

The different phases to test. The format of the tuples should be (str(id of the test), int(number of trials before the phase begins), str(name of the algorithm’s attribute to spy (ex: “space.sample”)) )

test_broken_trials()[source]

Test that algorithm can handle broken trials

test_cat_data()[source]

Test that algorithm supports categorical dimensions

test_configuration()[source]

Test that configuration property attribute contains all class arguments.

test_get_id()[source]

Test that the id hashing is valid

test_has_observed()[source]

Verify that algorithm detects correctly if a trial was observed

test_has_observed_statedict()[source]

Verify that algorithm detects correctly if a trial was observed even when state was restored.

test_has_suggested()[source]

Verify that algorithm detects correctly if a trial was suggested

test_has_suggested_statedict()[source]

Verify that algorithm detects correctly if a trial was suggested even when state was restored.

test_int_data()[source]

Test that algorithm supports integer dimensions

test_is_done_cardinality()[source]

Test that algorithm will stop when cardinality is reached

test_is_done_max_trials()[source]

Test that algorithm will stop when max trials is reached

test_logint_data()[source]

Test that algorithm supports loginteger dimensions

test_logreal_data()[source]

Test that algorithm supports logreal dimensions

test_n_observed()[source]

Verify that algorithm returns correct number of observed trials

test_n_suggested()[source]

Verify that algorithm returns correct number of suggested trials

test_observe()[source]

Verify that algorithm observes trial without any issues

test_optimize_branin()[source]

Test that algorithm optimizes a simple task comparably to random search.

test_real_data()[source]

Test that algorithm supports real dimensions

test_seed_rng(seed: int)[source]

Test that the seeding gives reproducible results.

test_seed_rng_init()[source]

Test that if the algo has a seed constructor argument and a value is passed, the suggested trials are reproducible.

test_shape_data()[source]

Test that algorithm supports dimensions with shape

test_state_dict(seed: int, phase: TestPhase)[source]

Verify that resetting state makes sampling deterministic.

The “source” algo is initialized at the start of each phase. The “target” algo instance is set to different initial conditions. This checks that it always gives the same suggestion as the original algo after set_state is used.

test_suggest_n()[source]

Verify that suggest returns correct number of trials if num is specified in suggest.

update_space(test_space: dict) dict[source]

Get complete space configuration with partial overwrite

The values passed in test_space will override the default values in self.config.

Parameters
test_space: dict

The configuration for the space.

class orion.testing.algo.BaseParallelStrategyTests[source]

Generic Test-suite for parallel strategies.

This test-suite follow the same logic than BaseAlgoTests, but applied for ParallelStrategy classes.

Attributes
default_value
expected_value
parallel_strategy_name

Methods

create_strategy([config])

Create the parallel strategy based on config.

get_corrupted_trial()

Return a corrupted trial with results but status reserved

get_noncompleted_trial([status])

Return a single trial without results

get_trials()

10 objective observations

test_configuration()

Test that configuration property attribute contains all class arguments.

test_handle_corrupted_trials(caplog)

Test that strategy can handle trials that has objective but status is not properly set to completed.

test_infer_no_history()

Test that strategy can infer even without having seen trials

test_state_dict()

Verify state is restored properly

test_strategy_value()

Test that ParallelStrategy returns the expected value

test_handle_noncompleted_trials

create_strategy(config=None, **kwargs)[source]

Create the parallel strategy based on config.

Parameters
config: dict, optional

The configuration for the parallel strategy. self.config will be used if config is None.

kwargs: dict

Values to override strategy configuration.

get_corrupted_trial()[source]

Return a corrupted trial with results but status reserved

get_noncompleted_trial(status='reserved')[source]

Return a single trial without results

get_trials()[source]

10 objective observations

test_configuration()[source]

Test that configuration property attribute contains all class arguments.

test_handle_corrupted_trials(caplog)[source]

Test that strategy can handle trials that has objective but status is not properly set to completed.

test_infer_no_history()[source]

Test that strategy can infer even without having seen trials

test_state_dict()[source]

Verify state is restored properly

test_strategy_value()[source]

Test that ParallelStrategy returns the expected value

class orion.testing.algo.TestPhase(name: 'str', n_trials: 'int', method_to_spy: 'str | None' = None, prev: 'TestPhase | None' = None, next: 'TestPhase | int | None' = None)[source]
Attributes
end_n_trials

Returns the end of this test phase (either start of next phase or max_trials).

length

Returns the duration of this test phase, in number of trials.

method_to_spy
next
prev
property end_n_trials: int

Returns the end of this test phase (either start of next phase or max_trials).

property length: int

Returns the duration of this test phase, in number of trials.

method_to_spy: str | None = None

Name of the method or function that is supposed to create the trials during that test phase.

This is currently unused. Tests could potentially pass this as an argument to mocker.spy to check that the method is called the right number of times during each phase.

n_trials: int

Number of trials after which the phase should begin.

name: str

Name of the test phase.

orion.testing.algo.customized_mutate_example(search_space, rng, old_value, **kwargs)[source]

Define a customized mutate function example

orion.testing.algo.first_phase_only(test)[source]

Decorator to run a test only on the first phase of the algorithm.

orion.testing.algo.last_phase_only(test)[source]

Decorator to run a test only on the last test phase of the algorithm.