Benchmark modules

Benchmark definition

class orion.benchmark.Benchmark(name, algorithms, targets, storage=None, executor=None)[source]

Benchmark definition

Parameters
name: str

Name of the benchmark

algorithms: list, optional

Algorithms used for benchmark, and for each algorithm, it can be formats as below:

  • A str of the algorithm name

  • A dict, with only one key and one value, where key is the algorithm name and value is a dict for the algorithm config.

  • A dict, with two keys.

    algorithm: str or dict

    Algorithm name in string or a dict with algorithm configure.

    deterministic: bool, optional

    True if it is a deterministic algorithm, then for each assessment, only one experiment will be run for this algorithm.

Examples:

>>> ["random", "tpe"]
>>> ["random", {"tpe": {"seed": 1}}]
>>> [{"algorithm": "random"}, {"algorithm": {"gridsearch": {"n_values": 50}}, "deterministic": True}]
targets: list, optional

Targets for the benchmark, each target will be a dict with two keys.

assess: list

Assessment objects

task: list

Task objects

storage: dict, optional

Configuration of the storage backend.

executor: `orion.executor.base.BaseExecutor`, optional

Executor to run the benchmark experiments

Attributes
configuration

Return a copy of an Benchmark configuration as a dictionary.

id

Id of the benchmark in the database if configured.

Methods

analysis()

Return all the assessment figures

experiments([silent])

Return all the experiments submitted in benchmark

process([n_workers])

Run studies experiment

setup_studies()

Setup studies to run for the benchmark.

status([silent])

Display benchmark status

analysis()[source]

Return all the assessment figures

property configuration

Return a copy of an Benchmark configuration as a dictionary.

experiments(silent=True)[source]

Return all the experiments submitted in benchmark

property id

Id of the benchmark in the database if configured.

Value is None if the benchmark is not configured.

process(n_workers=1)[source]

Run studies experiment

setup_studies()[source]

Setup studies to run for the benchmark. Benchmark algorithms, together with each task and assessment combination define a study.

status(silent=True)[source]

Display benchmark status

class orion.benchmark.Study(benchmark, algorithms, assessment, task)[source]

A study is one assessment and task combination in the Benchmark targets. It will build and run experiments for all the algorithms for that task.

Parameters
benchmark: A Benchmark instance
algorithms: list

Algorithms used for benchmark, each algorithm can be a string or dict, with same format as Benchmark algorithms.

assessment: list

Assessment instance

task: list

Task instance

Methods

analysis()

Return assessment figure

execute([n_workers])

Execute all the experiments of the study

experiments()

Return all the experiments of the study

setup_experiments()

Setup experiments to run of the study

status()

Return status of the study

analysis()[source]

Return assessment figure

execute(n_workers=1)[source]

Execute all the experiments of the study

experiments()[source]

Return all the experiments of the study

setup_experiments()[source]

Setup experiments to run of the study

status()[source]

Return status of the study