Benchmark modules¶
Benchmark definition¶
- class orion.benchmark.Benchmark(storage, name, algorithms, targets, executor=None)[source]¶
Benchmark definition
- Parameters
- storage: Storage
Instance of the storage to use
- name: str
Name of the benchmark
- algorithms: list, optional
Algorithms used for benchmark, and for each algorithm, it can be formats as below:
A str of the algorithm name
A dict, with only one key and one value, where key is the algorithm name and value is a dict for the algorithm config.
Examples:
>>> ["random", "tpe"] >>> ["random", {"tpe": {"seed": 1}}]
- targets: list, optional
Targets for the benchmark, each target will be a dict with two keys.
- assess: list
Assessment objects
- task: list
Task objects
- executor: `orion.executor.base.BaseExecutor`, optional
Executor to run the benchmark experiments
- Attributes
configuration
Return a copy of an
Benchmark
configuration as a dictionary.executor
Returns the current executor to use to run jobs in parallel
id
Id of the benchmark in the database if configured.
Methods
analysis
([assessment, task, algorithms])Return all assessment figures
get_experiments
([silent])Return all the experiments submitted in benchmark
process
([n_workers])Run studies experiment
Setup studies to run for the benchmark.
status
([silent])Display benchmark status
close
validate_algorithms
validate_assessment
validate_task
- analysis(assessment=None, task=None, algorithms=None)[source]¶
Return all assessment figures
- Parameters
- assessment: str or None, optional
Filter analysis and only return those for the given assessment name.
- task: str or None, optional
Filter analysis and only return those for the given task name.
- algorithms: list of str or None, optional
Compute analysis only on specified algorithms. Compute on all otherwise.
- property executor¶
Returns the current executor to use to run jobs in parallel
- property id¶
Id of the benchmark in the database if configured.
Value is None if the benchmark is not configured.
- setup_studies()[source]¶
Setup studies to run for the benchmark. Benchmark algorithms, together with each
task
andassessment
combination define a study.
- class orion.benchmark.Study(benchmark, algorithms, assessment, task)[source]¶
A study is one assessment and task combination in the
Benchmark
targets. It will build and run experiments for all the algorithms for that task.- Parameters
- benchmark: A Benchmark instance
- algorithms: list
Algorithms used for benchmark, each algorithm can be a string or dict, with same format as
Benchmark
algorithms.- assessment: list
Assessment instance
- task: list
Task instance
Methods
analysis
([algorithms])Return assessment figure
execute
([n_workers])Execute all the experiments of the study
get_experiments
([algorithms])Return all the experiments of the study
Setup experiments to run of the study
status
()Return status of the study
- analysis(algorithms=None)[source]¶
Return assessment figure
- Parameters
- algorithms: list of str or None, optional
Compute analysis only on specified algorithms. Compute on all otherwise.
Benchmark modules of Oríon