Hyperband Algorithm

A Novel Bandit-Based Approach to Hyperparameter Optimization

Implement Hyperband to exploit configurations with fixed resource efficiently

class orion.algo.hyperband.Bracket(hyperband, budgets, repetition_id)[source]

Bracket of rungs for the algorithm Hyperband.

Parameters:
hyperband: `Hyperband` algorithm

The hyperband algorithm object which this bracket will be part of.

budgets: list of tuple

Each tuple gives the (n_trials, resource_budget) for the respective rung.

repetition_id: int

The id of hyperband execution this bracket belongs to

Attributes:
is_done

Return True, if the last rung is filled.

is_filled

Return True if first rung with trials is filled

Methods

get_candidates(rung_id) Get a candidate for promotion
get_point_max_resource(point) Return the max resource value that has been tried for a point
has_rung_filled(rung_id) Return True, if the rung[rung_id] is filled.
is_ready([rung_id]) Return True, if the bracket is ready for next promote
promote() Promote the first candidate that is found and return it
register(point, objective) Register a point in the corresponding rung
sample() Sample a new trial with lowest fidelity
seed_rng(seed) Seed the state of the random number generator.
get_candidates(rung_id)[source]

Get a candidate for promotion

get_point_max_resource(point)[source]

Return the max resource value that has been tried for a point

has_rung_filled(rung_id)[source]

Return True, if the rung[rung_id] is filled.

is_done

Return True, if the last rung is filled.

is_filled

Return True if first rung with trials is filled

is_ready(rung_id=None)[source]

Return True, if the bracket is ready for next promote

promote()[source]

Promote the first candidate that is found and return it

The rungs are iterated over in reversed order, so that high rungs are prioritised for promotions. When a candidate is promoted, the loop is broken and the method returns the promoted point.

Note

All trials are part of the rungs, for any state. Only completed trials are eligible for promotion, i.e., only completed trials can be part of top-k. Lookup for promotion in rung l + 1 contains trials of any status.

register(point, objective)[source]

Register a point in the corresponding rung

sample()[source]

Sample a new trial with lowest fidelity

seed_rng(seed)[source]

Seed the state of the random number generator.

Parameters:seed – Integer seed for the random number generator.
class orion.algo.hyperband.Hyperband(space, seed=None, repetitions=inf)[source]

Hyperband formulates hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations.`

For more information on the algorithm, see original paper at http://jmlr.org/papers/v18/16-558.html.

Li, Lisha et al. “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization” Journal of Machine Learning Research, 18:1-52, 2018.

Parameters:
space: `orion.algo.space.Space`

Optimisation space with priors for each dimension.

seed: None, int or sequence of int

Seed for the random number generator used to sample new trials. Default: None

repetitions: int

Number of execution of Hyperband. Default is numpy.inf which means to run Hyperband until no new trials can be suggested.

Attributes:
fidelity_index

Compute the index of the point when fidelity is.

is_done

Return True, if all required execution been done.

state_dict

Return a state dict that can be used to reset the state of the algorithm.

Methods

get_id(point[, ignore_fidelity]) Compute a unique hash for a point based on params, without fidelity level by default.
observe(points, results) Observe evaluation results corresponding to list of points in space.
sample(num, bracket[, buffer]) Sample new points from bracket
seed_rng(seed) Seed the state of the random number generator.
set_state(state_dict) Reset the state of the algorithm based on the given state_dict
suggest([num]) Suggest a number of new sets of parameters.
fidelity_index

Compute the index of the point when fidelity is.

get_id(point, ignore_fidelity=True)[source]

Compute a unique hash for a point based on params, without fidelity level by default.

is_done

Return True, if all required execution been done.

observe(points, results)[source]

Observe evaluation results corresponding to list of points in space.

A simple random sampler though does not take anything into account.

sample(num, bracket, buffer=10)[source]

Sample new points from bracket

seed_rng(seed)[source]

Seed the state of the random number generator.

Parameters:seed – Integer seed for the random number generator.
set_state(state_dict)[source]

Reset the state of the algorithm based on the given state_dict

Parameters:state_dict – Dictionary representing state of an algorithm
state_dict

Return a state dict that can be used to reset the state of the algorithm.

suggest(num=1)[source]

Suggest a number of new sets of parameters.

Sample new points until first rung is filled. Afterwards waits for all trials to be completed before promoting trials to the next rung.

Parameters:
num: int, optional

Number of points to suggest. Defaults to 1.

Returns:
list of points or None

A list of lists representing points suggested by the algorithm. The algorithm may opt out if it cannot make a good suggestion at the moment (it may be waiting for other trials to complete), in which case it will return None.

orion.algo.hyperband.compute_budgets(max_resources, reduction_factor)[source]

Compute the budgets used for each execution of hyperband

orion.algo.hyperband.display_budgets(budgets_tab, max_resources, reduction_factor)[source]

Display hyperband budget as a table in debug log