Benchmark modules¶

Benchmark definition¶

class orion.benchmark.Benchmark(storage, name, algorithms, targets, executor=None)[source]¶

Benchmark definition

Parameters

storage: Storage

Instance of the storage to use

name: str

Name of the benchmark

algorithms: list, optional

Algorithms used for benchmark, and for each algorithm, it can be formats as below:

A str of the algorithm name
A dict, with only one key and one value, where key is the algorithm name and value is a dict for the algorithm config.

Examples:

>>> ["random", "tpe"]
>>> ["random", {"tpe": {"seed": 1}}]

targets: list, optional

Targets for the benchmark, each target will be a dict with two keys.

assess: list: Assessment objects
task: list: Task objects

executor: `orion.executor.base.BaseExecutor`, optional

Executor to run the benchmark experiments

Attributes

configuration: Return a copy of an Benchmark configuration as a dictionary.
executor: Returns the current executor to use to run jobs in parallel
id: Id of the benchmark in the database if configured.

Methods

`analysis`([assessment, task, algorithms])	Return all assessment figures
`get_experiments`([silent])	Return all the experiments submitted in benchmark
`process`([n_workers])	Run studies experiment
`setup_studies`()	Setup studies to run for the benchmark.
`status`([silent])	Display benchmark status

close
validate_algorithms
validate_assessment
validate_task

analysis(assessment=None, task=None, algorithms=None)[source]¶

Return all assessment figures

Parameters

assessment: str or None, optional: Filter analysis and only return those for the given assessment name.
task: str or None, optional: Filter analysis and only return those for the given task name.
algorithms: list of str or None, optional: Compute analysis only on specified algorithms. Compute on all otherwise.

property configuration¶: Return a copy of an Benchmark configuration as a dictionary.

property executor¶: Returns the current executor to use to run jobs in parallel

get_experiments(silent=True)[source]¶: Return all the experiments submitted in benchmark

property id¶

Id of the benchmark in the database if configured.

Value is None if the benchmark is not configured.

process(n_workers=1)[source]¶: Run studies experiment

setup_studies()[source]¶: Setup studies to run for the benchmark. Benchmark algorithms, together with each task and assessment combination define a study.

status(silent=True)[source]¶: Display benchmark status

class orion.benchmark.Study(benchmark, algorithms, assessment, task)[source]¶

A study is one assessment and task combination in the Benchmark targets. It will build and run experiments for all the algorithms for that task.

Parameters

benchmark: A Benchmark instance
algorithms: list: Algorithms used for benchmark, each algorithm can be a string or dict, with same format as Benchmark algorithms.
assessment: list: Assessment instance
task: list: Task instance

Methods

`analysis`([algorithms])	Return assessment figure
`execute`([n_workers])	Execute all the experiments of the study
`get_experiments`([algorithms])	Return all the experiments of the study
`setup_experiments`()	Setup experiments to run of the study
`status`()	Return status of the study

analysis(algorithms=None)[source]¶

Return assessment figure

Parameters

algorithms: list of str or None, optional: Compute analysis only on specified algorithms. Compute on all otherwise.

execute(n_workers=1)[source]¶: Execute all the experiments of the study

get_experiments(algorithms=None)[source]¶

Return all the experiments of the study

Parameters

algorithms: list of str or None, optional: Return only experiments for specified algorithms. Return all otherwise.

setup_experiments()[source]¶: Setup experiments to run of the study

status()[source]¶: Return status of the study