Benchmark modules

Benchmark definition

class orion.benchmark.Benchmark(name, algorithms, targets, storage=None, executor=None)[source]

Benchmark definition

Parameters
name: str

Name of the benchmark

algorithms: list, optional

Algorithms used for benchmark, and for each algorithm, it can be formats as below:

  • A str of the algorithm name

  • A dict, with only one key and one value, where key is the algorithm name and value is a dict for the algorithm config.

  • A dict, with two keys.

    algorithm: str or dict

    Algorithm name in string or a dict with algorithm configure.

    deterministic: bool, optional

    True if it is a deterministic algorithm, then for each assessment, only one experiment will be run for this algorithm.

Examples:

>>> ["random", "tpe"]
>>> ["random", {"tpe": {"seed": 1}}]
>>> [{"algorithm": "random"}, {"algorithm": {"gridsearch": {"n_values": 50}}, "deterministic": True}]
targets: list, optional

Targets for the benchmark, each target will be a dict with two keys.

assess: list

Assessment objects

task: list

Task objects

storage: dict, optional

Configuration of the storage backend.

executor: `orion.executor.base.BaseExecutor`, optional

Executor to run the benchmark experiments

Attributes
configuration

Return a copy of an Benchmark configuration as a dictionary.

executor

Returns the current executor to use to run jobs in parallel

id

Id of the benchmark in the database if configured.

Methods

analysis()

Return all the assessment figures with format as {assessment_name: {figure_name: figure_object}}

experiments([silent])

Return all the experiments submitted in benchmark

process([n_workers])

Run studies experiment

setup_studies()

Setup studies to run for the benchmark.

status([silent])

Display benchmark status

close

analysis()[source]

Return all the assessment figures with format as {assessment_name: {figure_name: figure_object}}

property configuration

Return a copy of an Benchmark configuration as a dictionary.

property executor

Returns the current executor to use to run jobs in parallel

experiments(silent=True)[source]

Return all the experiments submitted in benchmark

property id

Id of the benchmark in the database if configured.

Value is None if the benchmark is not configured.

process(n_workers=1)[source]

Run studies experiment

setup_studies()[source]

Setup studies to run for the benchmark. Benchmark algorithms, together with each task and assessment combination define a study.

status(silent=True)[source]

Display benchmark status

class orion.benchmark.Study(benchmark, algorithms, assessment, task)[source]

A study is one assessment and task combination in the Benchmark targets. It will build and run experiments for all the algorithms for that task.

Parameters
benchmark: A Benchmark instance
algorithms: list

Algorithms used for benchmark, each algorithm can be a string or dict, with same format as Benchmark algorithms.

assessment: list

Assessment instance

task: list

Task instance

Methods

analysis()

Return assessment figures

execute([n_workers])

Execute all the experiments of the study

experiments()

Return all the experiments of the study

setup_experiments()

Setup experiments to run of the study

status()

Return status of the study

analysis()[source]

Return assessment figures

execute(n_workers=1)[source]

Execute all the experiments of the study

experiments()[source]

Return all the experiments of the study

setup_experiments()[source]

Setup experiments to run of the study

status()[source]

Return status of the study