Benchmark modules¶
Benchmark definition¶
-
class
orion.benchmark.
Benchmark
(name, algorithms, targets)[source]¶ Benchmark definition
Parameters: - name: str
Name of the benchmark
- algorithms: list, optional
Algorithms used for benchmark, each algorithm can be a string or dict.
- targets: list, optional
Targets for the benchmark, each target will be a dict with two keys.
- assess: list
Assessment objects
- task: list
Task objects
Attributes: configuration
Return a copy of an
Benchmark
configuration as a dictionary.id
Id of the benchmark in the database if configured.
Methods
analysis
()Return all the assessment figures experiments
([silent])Return all the experiments submitted in benchmark process
()Run studies experiment setup_studies
()Setup studies to run for the benchmark. status
([silent])Display benchmark status -
id
¶ Id of the benchmark in the database if configured.
Value is None if the benchmark is not configured.
-
setup_studies
()[source]¶ Setup studies to run for the benchmark. Benchmark algorithms, together with each
task
andassessment
combination define a study.
-
class
orion.benchmark.
Study
(benchmark, algorithms, assessment, task)[source]¶ A study is one assessment and task combination in the
Benchmark
targets. It will build and run experiments for all the algorithms for that task.Parameters: - benchmark: `Benchmark` instance
- algorithms: list
Algorithms used for benchmark, each algorithm can be a string or dict.
- assessment: list
Assessment instance
- task: list
Task instance
Methods
analysis
()Return assessment figure execute
()Execute all the experiments of the study experiments
()Return all the experiments of the study setup_experiments
()Setup experiments to run of the study status
()Return status of the study