Base definition of assessments

Base definition of Assessment

class orion.benchmark.assessment.base.BenchmarkAssessment(task_num, **kwargs)

Base class describing what an assessment can do.

Parameters

task_numint

Number of experiment the assessment ask to run the corresponding task

kwargsdict

Configurable parameters of the assessment, a particular assessment implementation can have its own parameters.

Attributes

configuration

Return the configuration of the assessment.

task_num

Return the task number to run for this assessment

Methods

analysis(task, experiments)	Generate plotly.graph_objects.Figure objects to display the performance analysis based on the assessment purpose.
get_executor(task_index)	Return an instance of orion.executor.base.Executor based on the index of tasks that the assessment is asking to run.

abstract analysis(task, experiments)

Generate plotly.graph_objects.Figure objects to display the performance analysis based on the assessment purpose.

task: str

Name of the task

experiments: list

A list of (task_index, experiment), where task_index is the index of task to run for this assessment, and experiment is an instance of orion.core.worker.experiment.

Returns

Dict of plotly.graph_objects.Figure objects with a format as like

{“assessment name”: {“task name”: {“figure name”: plotly.graph_objects.Figure}}}

Examples

>>> {"AverageRank": {"RosenBrock": {"rankings": plotly.graph_objects.Figure}}}
    ..

property configuration

Return the configuration of the assessment.

get_executor(task_index)

Return an instance of orion.executor.base.Executor based on the index of tasks that the assessment is asking to run.

property task_num

Return the task number to run for this assessment