Benchmark

You can benchmark the performance of search algorithms with different tasks at different assessment levels.

Benchmark can be created as below, refer to Benchmark Client for how to create and Benchmark modules for how to use benchmark.

from orion.benchmark.benchmark_client import get_or_create_benchmark
from orion.benchmark.assessment import AverageResult, AverageRank
from orion.benchmark.task import RosenBrock, EggHolder, CarromTable

benchmark = get_or_create_benchmark(name='benchmark',
        algorithms=['random', 'tpe'],
        targets=[
            {
                'assess': [AverageResult(2), AverageRank(2)],
                'task': [RosenBrock(25, dim=3), EggHolder(20, dim=4), CarromTable(20)]
            }
        ])

Beside out of box Task modules and Assessment modules, you can also extend benchmark to add new Tasks with Base definition of tasks and Assessments with Base definition of assessments,

Learn how to get start using benchmark in Orion with this sample notebook.