Assessment modules¶
Benchmark Assessments definition¶
-
class
orion.benchmark.assessment.AverageRank(task_num=1)[source]¶ Evaluate the average performance (objective value) between different search algorithms from the rank perspective at different time steps (trial number). The performance (objective value) used for a trial will the best result until the trial.
Methods
analysis(task, experiments)Generate a
plotly.graph_objects.Figureto display average rankings between different search algorithms.-
analysis(task, experiments)[source]¶ Generate a
plotly.graph_objects.Figureto display average rankings between different search algorithms.- task: str
Name of the task
- experiments: list
A list of (task_index, experiment), where task_index is the index of task to run for this assessment, and experiment is an instance of
orion.core.worker.experiment.
-
-
class
orion.benchmark.assessment.AverageResult(task_num=1)[source]¶ Evaluate the average performance (objective value) for each search algorithm at different time steps (trial number). The performance (objective value) used for a trial will the best result until the trial.
Methods
analysis(task, experiments)Generate a
plotly.graph_objects.Figureto display average performance for each search algorithm.-
analysis(task, experiments)[source]¶ Generate a
plotly.graph_objects.Figureto display average performance for each search algorithm.- task: str
Name of the task
- experiments: list
A list of (task_index, experiment), where task_index is the index of task to run for this assessment, and experiment is an instance of
orion.core.worker.experiment.
-
-
class
orion.benchmark.assessment.BaseAssess(task_num, **kwargs)[source]¶ Base class describing what an assessment can do.
- Parameters
- task_numint
Number of experiment the assessment ask to run the corresponding task
- kwargsdict
Configurable parameters of the assessment, a particular assessment implementation can have its own parameters.
- Attributes
configurationReturn the configuration of the assessment.
task_numReturn the task number to run for this assessment
Methods
analysis(task, experiments)Generate a
plotly.graph_objects.Figureto display the performance analysis based on the assessment purpose.-
abstract
analysis(task, experiments)[source]¶ Generate a
plotly.graph_objects.Figureto display the performance analysis based on the assessment purpose.- task: str
Name of the task
- experiments: list
A list of (task_index, experiment), where task_index is the index of task to run for this assessment, and experiment is an instance of
orion.core.worker.experiment.
-
property
configuration¶ Return the configuration of the assessment.
-
property
task_num¶ Return the task number to run for this assessment
Assessment modules of Oríon