Assessment modules

Benchmark Assessments definition

class orion.benchmark.assessment.BaseAssess(task_num, **kwargs)[source]

Base class describing what an assessment can do.

Parameters:
task_num : int

Number of experiment the assessment ask to run the corresponding task

kwargs : dict

Configurable parameters of the assessment, a particular assessment implementation can have its own parameters.

Attributes:
configuration

Return the configuration of the assessment.

task_num

Return the task number to run for this assessment

Methods

analysis(task, experiments) Generate a plotly.graph_objects.Figure to display the performance analysis based on the assessment purpose.
analysis(task, experiments)[source]

Generate a plotly.graph_objects.Figure to display the performance analysis based on the assessment purpose.

task: str
Name of the task
experiments: list
A list of (task_index, experiment), where task_index is the index of task to run for this assessment, and experiment is an instance of orion.core.worker.experiment.
configuration

Return the configuration of the assessment.

task_num

Return the task number to run for this assessment

class orion.benchmark.assessment.AverageRank(task_num=1)[source]

Evaluate the average performance (objective value) between different search algorithms from the rank perspective at different time steps (trial number). The performance (objective value) used for a trial will the best result until the trial.

Methods

analysis(task, experiments) Generate a plotly.graph_objects.Figure to display average rankings between different search algorithms.
analysis(task, experiments)[source]

Generate a plotly.graph_objects.Figure to display average rankings between different search algorithms.

task: str
Name of the task
experiments: list
A list of (task_index, experiment), where task_index is the index of task to run for this assessment, and experiment is an instance of orion.core.worker.experiment.
class orion.benchmark.assessment.AverageResult(task_num=1)[source]

Evaluate the average performance (objective value) for each search algorithm at different time steps (trial number). The performance (objective value) used for a trial will the best result until the trial.

Methods

analysis(task, experiments) Generate a plotly.graph_objects.Figure to display average performance for each search algorithm.
analysis(task, experiments)[source]

Generate a plotly.graph_objects.Figure to display average performance for each search algorithm.

task: str
Name of the task
experiments: list
A list of (task_index, experiment), where task_index is the index of task to run for this assessment, and experiment is an instance of orion.core.worker.experiment.