Base definition of assessments

Base definition of Assessment

class orion.benchmark.assessment.base.BenchmarkAssessment(repetitions, **kwargs)

Base class describing what an assessment can do.

Parameters

repetitionsint

Number of experiment the assessment ask to run the corresponding task

kwargsdict

Configurable parameters of the assessment, a particular assessment implementation can have its own parameters.

Attributes

configuration

Return the configuration of the assessment.

repetitions

Return the task number to run for this assessment

Methods

analysis(task, experiments)	Generate plotly.graph_objects.Figure objects to display the performance analysis based on the assessment purpose.
get_executor(task_index)	Return an instance of orion.executor.base.Executor based on the index of tasks that the assessment is asking to run.

abstract analysis(task, experiments)

Generate plotly.graph_objects.Figure objects to display the performance analysis based on the assessment purpose.

task: str

Name of the task

experiments: list

A list of (task_index, experiment), where task_index is the index of task to run for this assessment, and experiment is an instance of orion.core.worker.experiment.

Returns

Dict of plotly.graph_objects.Figure objects with a format as like

{“assessment name”: {“task name”: {“figure name”: plotly.graph_objects.Figure}}}

Examples

>>> {"AverageRank": {"RosenBrock": {"rankings": plotly.graph_objects.Figure}}}
    ..

property configuration

Return the configuration of the assessment.

get_executor(task_index)

Return an instance of orion.executor.base.Executor based on the index of tasks that the assessment is asking to run.

property repetitions

Return the task number to run for this assessment