Benchmark modules

Benchmark definition

class orion.benchmark.Benchmark(name, algorithms, targets)[source]

Benchmark definition

Parameters:
name: str

Name of the benchmark

algorithms: list, optional

Algorithms used for benchmark, each algorithm can be a string or dict.

targets: list, optional

Targets for the benchmark, each target will be a dict with two keys.

assess: list

Assessment objects

task: list

Task objects

Attributes:
configuration

Return a copy of an Benchmark configuration as a dictionary.

id

Id of the benchmark in the database if configured.

Methods

analysis() Return all the assessment figures
experiments([silent]) Return all the experiments submitted in benchmark
process() Run studies experiment
setup_studies() Setup studies to run for the benchmark.
status([silent]) Display benchmark status
analysis()[source]

Return all the assessment figures

configuration

Return a copy of an Benchmark configuration as a dictionary.

experiments(silent=True)[source]

Return all the experiments submitted in benchmark

id

Id of the benchmark in the database if configured.

Value is None if the benchmark is not configured.

process()[source]

Run studies experiment

setup_studies()[source]

Setup studies to run for the benchmark. Benchmark algorithms, together with each task and assessment combination define a study.

status(silent=True)[source]

Display benchmark status

class orion.benchmark.Study(benchmark, algorithms, assessment, task)[source]

A study is one assessment and task combination in the Benchmark targets. It will build and run experiments for all the algorithms for that task.

Parameters:
benchmark: `Benchmark` instance
algorithms: list

Algorithms used for benchmark, each algorithm can be a string or dict.

assessment: list

Assessment instance

task: list

Task instance

Methods

analysis() Return assessment figure
execute() Execute all the experiments of the study
experiments() Return all the experiments of the study
setup_experiments() Setup experiments to run of the study
status() Return status of the study
analysis()[source]

Return assessment figure

execute()[source]

Execute all the experiments of the study

experiments()[source]

Return all the experiments of the study

setup_experiments()[source]

Setup experiments to run of the study

status()[source]

Return status of the study