Benchmark modules

Benchmark definition

class orion.benchmark.Benchmark(storage, name, algorithms, targets, executor=None)[source]

Benchmark definition

Parameters
storage: Storage

Instance of the storage to use

name: str

Name of the benchmark

algorithms: list, optional

Algorithms used for benchmark, and for each algorithm, it can be formats as below:

  • A str of the algorithm name

  • A dict, with only one key and one value, where key is the algorithm name and value is a dict for the algorithm config.

Examples:

>>> ["random", "tpe"]
>>> ["random", {"tpe": {"seed": 1}}]
targets: list, optional

Targets for the benchmark, each target will be a dict with two keys.

assess: list

Assessment objects

task: list

Task objects

executor: `orion.executor.base.BaseExecutor`, optional

Executor to run the benchmark experiments

Attributes
configuration

Return a copy of an Benchmark configuration as a dictionary.

executor

Returns the current executor to use to run jobs in parallel

id

Id of the benchmark in the database if configured.

Methods

analysis([assessment, task, algorithms])

Return all assessment figures

get_experiments([silent])

Return all the experiments submitted in benchmark

process([n_workers])

Run studies experiment

setup_studies()

Setup studies to run for the benchmark.

status([silent])

Display benchmark status

close

validate_algorithms

validate_assessment

validate_task

analysis(assessment=None, task=None, algorithms=None)[source]

Return all assessment figures

Parameters
assessment: str or None, optional

Filter analysis and only return those for the given assessment name.

task: str or None, optional

Filter analysis and only return those for the given task name.

algorithms: list of str or None, optional

Compute analysis only on specified algorithms. Compute on all otherwise.

property configuration

Return a copy of an Benchmark configuration as a dictionary.

property executor

Returns the current executor to use to run jobs in parallel

get_experiments(silent=True)[source]

Return all the experiments submitted in benchmark

property id

Id of the benchmark in the database if configured.

Value is None if the benchmark is not configured.

process(n_workers=1)[source]

Run studies experiment

setup_studies()[source]

Setup studies to run for the benchmark. Benchmark algorithms, together with each task and assessment combination define a study.

status(silent=True)[source]

Display benchmark status

class orion.benchmark.Study(benchmark, algorithms, assessment, task)[source]

A study is one assessment and task combination in the Benchmark targets. It will build and run experiments for all the algorithms for that task.

Parameters
benchmark: A Benchmark instance
algorithms: list

Algorithms used for benchmark, each algorithm can be a string or dict, with same format as Benchmark algorithms.

assessment: list

Assessment instance

task: list

Task instance

Methods

analysis([algorithms])

Return assessment figure

execute([n_workers])

Execute all the experiments of the study

get_experiments([algorithms])

Return all the experiments of the study

setup_experiments()

Setup experiments to run of the study

status()

Return status of the study

analysis(algorithms=None)[source]

Return assessment figure

Parameters
algorithms: list of str or None, optional

Compute analysis only on specified algorithms. Compute on all otherwise.

execute(n_workers=1)[source]

Execute all the experiments of the study

get_experiments(algorithms=None)[source]

Return all the experiments of the study

Parameters
algorithms: list of str or None, optional

Return only experiments for specified algorithms. Return all otherwise.

setup_experiments()[source]

Setup experiments to run of the study

status()[source]

Return status of the study