Benchmark modules¶
Benchmark definition¶
- class orion.benchmark.Benchmark(name, algorithms, targets, storage=None, executor=None)[source]¶
Benchmark definition
- Parameters
- name: str
Name of the benchmark
- algorithms: list, optional
Algorithms used for benchmark, and for each algorithm, it can be formats as below:
A str of the algorithm name
A dict, with only one key and one value, where key is the algorithm name and value is a dict for the algorithm config.
A dict, with two keys.
- algorithm: str or dict
Algorithm name in string or a dict with algorithm configure.
- deterministic: bool, optional
True if it is a deterministic algorithm, then for each assessment, only one experiment will be run for this algorithm.
Examples:
>>> ["random", "tpe"] >>> ["random", {"tpe": {"seed": 1}}] >>> [{"algorithm": "random"}, {"algorithm": {"gridsearch": {"n_values": 50}}, "deterministic": True}]
- targets: list, optional
Targets for the benchmark, each target will be a dict with two keys.
- assess: list
Assessment objects
- task: list
Task objects
- storage: dict, optional
Configuration of the storage backend.
- executor: `orion.executor.base.BaseExecutor`, optional
Executor to run the benchmark experiments
- Attributes
configuration
Return a copy of an
Benchmark
configuration as a dictionary.executor
Returns the current executor to use to run jobs in parallel
id
Id of the benchmark in the database if configured.
Methods
analysis
()Return all the assessment figures with format as {assessment_name: {figure_name: figure_object}}
experiments
([silent])Return all the experiments submitted in benchmark
process
([n_workers])Run studies experiment
Setup studies to run for the benchmark.
status
([silent])Display benchmark status
close
- analysis()[source]¶
Return all the assessment figures with format as {assessment_name: {figure_name: figure_object}}
- property executor¶
Returns the current executor to use to run jobs in parallel
- property id¶
Id of the benchmark in the database if configured.
Value is None if the benchmark is not configured.
- setup_studies()[source]¶
Setup studies to run for the benchmark. Benchmark algorithms, together with each
task
andassessment
combination define a study.
- class orion.benchmark.Study(benchmark, algorithms, assessment, task)[source]¶
A study is one assessment and task combination in the
Benchmark
targets. It will build and run experiments for all the algorithms for that task.- Parameters
- benchmark: A Benchmark instance
- algorithms: list
Algorithms used for benchmark, each algorithm can be a string or dict, with same format as
Benchmark
algorithms.- assessment: list
Assessment instance
- task: list
Task instance
Methods
analysis
()Return assessment figures
execute
([n_workers])Execute all the experiments of the study
Return all the experiments of the study
Setup experiments to run of the study
status
()Return status of the study