Algorithms¶
Generic tests for Algorithms
- class orion.testing.algo.BaseAlgoTests[source]¶
Generic Test-suite for HPO algorithms.
This test-suite covers all typical cases for HPO algorithms. To use it for a new algorithm, the class inheriting from this one must redefine the attributes
algo_name
with the name of the algorithm used to create it with the algorithm factoryorion.core.worker.primary_algo.SpaceTransformAlgoWrapper
andconfig
with a base configuration for the algorithm that contains all its arguments. The base space can be redefine if needed with the attributespace
.Most algorithms have different phases that should be tested. For instance TPE has a first phase of random search and a second of Bayesian Optimization. The random search and Bayesian optimization phases use different logic and should both be tested. The
phases
class attribute can be set to parametrize all tests with each phase. Seetests/unittests/algo/test_tpe.py
for an example.- Attributes
- algo_name
Methods
assert_dim_type_supported
(test_space)Test that a given dimension type is properly supported by the algorithm
create_algo
([config, space, seed, ...])Create the algorithm based on config.
create_space
([space])Create the space object
force_observe
(num, algo[, seed])Force observe
num
trials.get_num
(num)Force number of trials to suggest
observe_trials
(trials, algo, rng)Make the algorithm observe trials
set_phases
(phases)Parametrize the tests with different phases.
Test that algorithm can handle broken trials
Test that algorithm supports categorical dimensions
Test that configuration property attribute contains all class arguments.
Test that the id hashing is valid
Verify that algorithm detects correctly if a trial was observed
Verify that algorithm detects correctly if a trial was observed even when state was restored.
Verify that algorithm detects correctly if a trial was suggested
Verify that algorithm detects correctly if a trial was suggested even when state was restored.
Test that algorithm supports integer dimensions
Test that algorithm will stop when cardinality is reached
Test that algorithm will stop when max trials is reached
Test that algorithm supports loginteger dimensions
Test that algorithm supports logreal dimensions
Verify that algorithm returns correct number of observed trials
Verify that algorithm returns correct number of suggested trials
Verify that algorithm observes trial without any issues
Test that algorithm optimizes a simple task comparably to random search.
Test that algorithm supports real dimensions
test_seed_rng
(seed)Test that the seeding gives reproducible results.
Test that if the algo has a seed constructor argument and a value is passed, the suggested trials are reproducible.
Test that algorithm supports dimensions with shape
test_state_dict
(seed, phase)Verify that resetting state makes sampling deterministic.
Verify that suggest returns correct number of trials if
num
is specified insuggest
.update_space
(test_space)Get complete space configuration with partial overwrite
first_phase
last_phase
- assert_dim_type_supported(test_space: dict)[source]¶
Test that a given dimension type is properly supported by the algorithm
This will test that the algorithm sample trials valid for the given type and that the algorithm can observe these trials.
- Parameters
- test_space: the search space of the test.
- classmethod create_algo(config: dict | None = None, space: Space | None = None, seed: int | Sequence[int] | None = None, n_observed_trials: int | None = None, **kwargs) SpaceTransformAlgoWrapper[AlgoType] [source]¶
Create the algorithm based on config.
Also initializes the algorithm with the required number of random trials from the previous test phases before returning it.
- Parameters
- config: dict, optional
The configuration for the algorithm.
cls.config
will be used ifconfig
isNone
.- space: ``orion.algo.space.Space``, optional
Space object to pass to algo. The output of
cls.create_space()
will be used ifspace
isNone
.- seed: int | Sequence[int], optional
When passed, seed_rng is called before observing anything.
- n_observed_trials: int | None, optional
Number of trials that the algorithm should have already observed when returned. When
None
(default), observes the number of trials at which the current phase begins. When set to 0, the algorithm will be freshly initialized.- kwargs: dict
Values to override algorithm configuration.
- classmethod create_space(space: dict | None = None)[source]¶
Create the space object
- Parameters
- space: dict, optional
Configuration of the search space. The default
self.space
will be used ifspace
isNone
.
- classmethod force_observe(num: int, algo: BaseAlgorithm, seed: int = 1)[source]¶
Force observe
num
trials.- Parameters
- num: int
Number of trials to suggest and observe.
- algo: ``orion.algo.base.BaseAlgorithm``
The algorithm that must suggest and observe.
- seed: int, optional
The seed used to generate random objectives
- Raises
- RuntimeError
If the algorithm returns duplicates. Algorithms may return duplicates across workers, but in sequential scenarios as here, it should not happen.
If the algorithm fails to sample any trial at least 5 times.
- classmethod get_num(num: int)[source]¶
Force number of trials to suggest
Some algorithms must be tested with specific number of suggests at a time (ex: ASHA). This method can be overridden to change
num
based on the special needs.TODO: Remove this or give it a better name.
- classmethod observe_trials(trials: list[Trial], algo: BaseAlgorithm, rng: numpy.random.RandomState)[source]¶
Make the algorithm observe trials
- Parameters
- trials: list of ``orion.core.worker.trial.Trial``
Trials formatted as tuples of values
- algo: ``orion.algo.base.BaseAlgorithm``
The algorithm used to observe trials.
- rng: ``numpy.random.RandomState``
Random number generator to generate random objectives.
- phases: ClassVar[list[TestPhase]] = [TestPhase(name='default', n_trials=0, method_to_spy='sample')]¶
Test phases for the algorithms. Overwrte this if the algorithm has more than one phase.
- classmethod set_phases(phases: Sequence[TestPhase])[source]¶
Parametrize the tests with different phases.
Some algorithms have different phases that should be tested. For instance TPE have a first phase of random search and a second of Bayesian Optimization. The random search and Bayesian optimization are different implementations and both should be tested.
- Parameters
- phases: list of tuples
The different phases to test. The format of the tuples should be (str(id of the test), int(number of trials before the phase begins), str(name of the algorithm’s attribute to spy (ex: “space.sample”)) )
- test_configuration()[source]¶
Test that configuration property attribute contains all class arguments.
- test_has_observed_statedict()[source]¶
Verify that algorithm detects correctly if a trial was observed even when state was restored.
- test_has_suggested_statedict()[source]¶
Verify that algorithm detects correctly if a trial was suggested even when state was restored.
- test_optimize_branin()[source]¶
Test that algorithm optimizes a simple task comparably to random search.
- test_seed_rng_init()[source]¶
Test that if the algo has a seed constructor argument and a value is passed, the suggested trials are reproducible.
- test_state_dict(seed: int, phase: TestPhase)[source]¶
Verify that resetting state makes sampling deterministic.
The “source” algo is initialized at the start of each phase. The “target” algo instance is set to different initial conditions. This checks that it always gives the same suggestion as the original algo after set_state is used.
- class orion.testing.algo.BaseParallelStrategyTests[source]¶
Generic Test-suite for parallel strategies.
This test-suite follow the same logic than BaseAlgoTests, but applied for ParallelStrategy classes.
- Attributes
- default_value
- expected_value
- parallel_strategy_name
Methods
create_strategy
([config])Create the parallel strategy based on config.
Return a corrupted trial with results but status reserved
get_noncompleted_trial
([status])Return a single trial without results
10 objective observations
Test that configuration property attribute contains all class arguments.
test_handle_corrupted_trials
(caplog)Test that strategy can handle trials that has objective but status is not properly set to completed.
Test that strategy can infer even without having seen trials
Verify state is restored properly
Test that ParallelStrategy returns the expected value
test_handle_noncompleted_trials
- create_strategy(config=None, **kwargs)[source]¶
Create the parallel strategy based on config.
- Parameters
- config: dict, optional
The configuration for the parallel strategy.
self.config
will be used ifconfig
isNone
.- kwargs: dict
Values to override strategy configuration.
- test_configuration()[source]¶
Test that configuration property attribute contains all class arguments.
- class orion.testing.algo.TestPhase(name: 'str', n_trials: 'int', method_to_spy: 'str | None' = None, prev: 'TestPhase | None' = None, next: 'TestPhase | int | None' = None)[source]¶
- Attributes
end_n_trials
Returns the end of this test phase (either start of next phase or max_trials).
length
Returns the duration of this test phase, in number of trials.
- method_to_spy
- next
- prev
- property end_n_trials: int¶
Returns the end of this test phase (either start of next phase or max_trials).
- orion.testing.algo.customized_mutate_example(search_space, rng, old_value, **kwargs)[source]¶
Define a customized mutate function example