Algorithms¶

Generic tests for Algorithms

class orion.testing.algo.BaseAlgoTests[source]¶

Generic Test-suite for HPO algorithms.

This test-suite covers all typical cases for HPO algorithms. To use it for a new algorithm, the class inheriting from this one must redefine the attributes algo_name with the name of the algorithm used to create it with the algorithm factory orion.core.worker.primary_algo.SpaceTransformAlgoWrapper and config with a base configuration for the algorithm that contains all its arguments. The base space can be redefine if needed with the attribute space.

Many algorithms have different phases that should be tested. For instance TPE have a first phase of random search and a second of Bayesian Optimization. The random search and Bayesian optimization are different implementations and both should be tested. For this reason, the class method orion.testing.algo.BaseAlgoTests.set_phases must be called to parametrize the tests with phases. Failure to doing so will causes the tests to crash. See tests/unittests/algo/test_tpe.py for an example.

Attributes

algo_name

Methods

`assert_callbacks`(spy, num, algo)	Callback to make special asserts at end of tests
`assert_dim_type_supported`(mocker, num, attr, ...)	Test that a given dimension type is properly supported by the algorithm
`create_algo`([config, space])	Create the algorithm based on config.
`create_space`([space])	Create the space object
`force_observe`(num, algo)	Force observe `num` trials.
`get_num`(num)	Force number of trials to suggest
`observe_trials`(trials, algo[, objective])	Make the algorithm observe trials
`set_phases`(phases)	Parametrize the tests with different phases.
`spy_phase`(mocker, num, algo, attribute)	Force observe `num` trials and then mock a given method to count calls.
`test_cat_data`(mocker, num, attr)	Test that algorithm supports categorical dimesions
`test_configuration`()	Test that configuration property attribute contains all class arguments.
`test_get_id`()	Test that the id hashing is valid
`test_has_observed`(mocker, num, attr)	Verify that algorithm detects correctly if a trial was observed
`test_has_observed_statedict`(mocker, num, attr)	Verify that algorithm detects correctly if a trial was observed even when state was restored.
`test_has_suggested`(mocker, num, attr)	Verify that algorithm detects correctly if a trial was suggested
`test_has_suggested_statedict`(mocker, num, attr)	Verify that algorithm detects correctly if a trial was suggested even when state was restored.
`test_int_data`(mocker, num, attr)	Test that algorithm supports integer dimesions
`test_is_done_cardinality`()	Test that algorithm will stop when cardinality is reached
`test_is_done_max_trials`()	Test that algorithm will stop when max trials is reached
`test_logint_data`(mocker, num, attr)	Test that algorithm supports loginteger dimesions
`test_logreal_data`(mocker, num, attr)	Test that algorithm supports logreal dimesions
`test_n_observed`(mocker, num, attr)	Verify that algorithm returns correct number of observed trials
`test_n_suggested`(mocker, num, attr)	Verify that algorithm returns correct number of suggested trials
`test_observe`(mocker, num, attr)	Verify that algorithm observes trial without any issues
`test_optimize_branin`()	Test that algorithm optimizes somehow (this is on-par with random search)
`test_real_data`(mocker, num, attr)	Test that algorithm supports real dimesions
`test_seed_rng`(mocker, num, attr)	Test that the seeding gives reproducibile results.
`test_seed_rng_init`(mocker, num, attr)	Test that the seeding gives reproducibile results.
`test_shape_data`(mocker, num, attr)	Test that algorithm supports dimesions with shape
`test_state_dict`(mocker, num, attr)	Verify that resetting state makes sampling deterministic
`test_suggest_n`(mocker, num, attr)	Verify that suggest returns correct number of trials if `num` is specified in `suggest`.
`update_space`(test_space)	Get complete space configuration with partial overwrite

assert_callbacks(spy, num, algo)[source]¶

Callback to make special asserts at end of tests

Override this method in algorithm test-suite to customize verifications done at end of tests.

Parameters

spy: Mocked object: Object mocked by BaseAlgoTests.spy_phase.
num: int: number of trials of the phase.
algo: ``orion.algo.base.BaseAlgorithm``: The algorithm being tested.

assert_dim_type_supported(mocker, num, attr, test_space)[source]¶

Test that a given dimension type is properly supported by the algorithm

This will test that the algorithm sample trials valid for the given type and that the algorithm can observe these trials.

Parameters

mocker: ``pytest_mock.mocker``: Mocker from pytest_mock. Should be given by fixtures of the tests.
num: int: Number of trials to suggest and observe
algo: ``orion.algo.base.BaseAlgorithm``: The algorithm to test
attribute: str: The algorithm attribute or method to mock. The path is respective to the algorithm object. For example, a valid value would be ‘space.sample’ which will mock algo.algorithm.space.sample.

create_algo(config=None, space=None, **kwargs)[source]¶

Create the algorithm based on config.

Parameters

config: dict, optional: The configuration for the algorithm. self.config will be used if config is None.
space: ``orion.algo.space.Space``, optional: Space object to pass to algo. The output of self.create_space() will be used if space is None.
kwargs: dict: Values to override algorithm configuration.

create_space(space=None)[source]¶

Create the space object

Parameters

space: dict, optional: Configuration of the search space. The default self.space will be used if space is None.

force_observe(num, algo)[source]¶

Force observe num trials.

Parameters

num: int: Number of trials to suggest and observe.
algo: ``orion.algo.base.BaseAlgorithm``: The algorithm that must suggest and observe.

Raises

RuntimeError

If the algorithm returns duplicates. Algorithms may return duplicates across workers, but in sequential scenarios as here, it should not happen.
If the algorithm fails to sample any trial at least 5 times.

get_num(num)[source]¶

Force number of trials to suggest

Some algorithms must be tested with specific number of suggests at a time (ex: ASHA). This method can be overriden to change num based on the special needs.

observe_trials(trials, algo, objective=0)[source]¶

Make the algorithm observe trials

Parameters

trials: list of ``orion.core.worker.trial.Trial``: Trials formatted as tuples of values
algo: ``orion.algo.base.BaseAlgorithm``: The algorithm used to observe trials.
objective: int, optional: The base objective for the trials. All objectives will have value objective + i. Defaults to 0.

classmethod set_phases(phases)[source]¶

Parametrize the tests with different phases.

Some algorithms have different phases that should be tested. For instance TPE have a first phase of random search and a second of Bayesian Optimization. The random search and Bayesian optimization are different implementations and both should be tested.

Parameters

phases: list of tuples: The different phases to test. The format of the tuples should be (str(id of the test), int(number of trials before the phase begins), str(name of the algorithm’s attribute to spy (ex: “space.sample”)) )

spy_phase(mocker, num, algo, attribute)[source]¶

Force observe num trials and then mock a given method to count calls.

Parameters

mocker: ``pytest_mock.mocker``: Mocker from pytest_mock. Should be given by fixtures of the tests.
num: int: Number of trials to suggest and observe
algo: ``orion.algo.base.BaseAlgorithm``: The algorithm to test
attribute: str: The algorithm attribute or method to mock. The path is respective to the algorithm object. For example, a valid value would be ‘space.sample’ which will mock algo.algorithm.space.sample.

test_cat_data(mocker, num, attr)[source]¶

Test that algorithm supports categorical dimesions