Benchmark modules¶
Benchmark definition¶
- class orion.benchmark.Benchmark(name, algorithms, targets, storage=None, executor=None)[source]¶
Benchmark definition
- Parameters
- name: str
Name of the benchmark
- algorithms: list, optional
Algorithms used for benchmark, and for each algorithm, it can be formats as below:
A str of the algorithm name
A dict, with only one key and one value, where key is the algorithm name and value is a dict for the algorithm config.
A dict, with two keys.
- algorithm: str or dict
Algorithm name in string or a dict with algorithm configure.
- deterministic: bool, optional
True if it is a deterministic algorithm, then for each assessment, only one experiment will be run for this algorithm.
Examples:
>>> ["random", "tpe"] >>> ["random", {"tpe": {"seed": 1}}] >>> [{"algorithm": "random"}, {"algorithm": {"gridsearch": {"n_values": 50}}, "deterministic": True}]
- targets: list, optional
Targets for the benchmark, each target will be a dict with two keys.
- assess: list
Assessment objects
- task: list
Task objects
- storage: dict, optional
Configuration of the storage backend.
- executor: `orion.executor.base.BaseExecutor`, optional
Executor to run the benchmark experiments
- Attributes
configuration
Return a copy of an
Benchmark
configuration as a dictionary.executor
Returns the current executor to use to run jobs in parallel
id
Id of the benchmark in the database if configured.
Methods
analysis
()Return all the assessment figures
experiments
([silent])Return all the experiments submitted in benchmark
process
([n_workers])Run studies experiment
Setup studies to run for the benchmark.
status
([silent])Display benchmark status
close
- property executor¶
Returns the current executor to use to run jobs in parallel
- property id¶
Id of the benchmark in the database if configured.
Value is None if the benchmark is not configured.
- setup_studies()[source]¶
Setup studies to run for the benchmark. Benchmark algorithms, together with each
task
andassessment
combination define a study.
- class orion.benchmark.Study(benchmark, algorithms, assessment, task)[source]¶
A study is one assessment and task combination in the
Benchmark
targets. It will build and run experiments for all the algorithms for that task.- Parameters
- benchmark: A Benchmark instance
- algorithms: list
Algorithms used for benchmark, each algorithm can be a string or dict, with same format as
Benchmark
algorithms.- assessment: list
Assessment instance
- task: list
Task instance
Methods
analysis
()Return assessment figure
execute
([n_workers])Execute all the experiments of the study
Return all the experiments of the study
Setup experiments to run of the study
status
()Return status of the study