Hyperband Algorithm¶

A Novel Bandit-Based Approach to Hyperparameter Optimization¶

Implement Hyperband to exploit configurations with fixed resource efficiently

class `orion.algo.hyperband.``Bracket`(hyperband, budgets, repetition_id)[source]

Bracket of rungs for the algorithm Hyperband.

Parameters: hyperband: `Hyperband` algorithm The hyperband algorithm object which this bracket will be part of. budgets: list of tuple Each tuple gives the (n_trials, resource_budget) for the respective rung. repetition_id: int The id of hyperband execution this bracket belongs to `is_done` Return True, if the last rung is filled. `is_filled` Return True if first rung with trials is filled

Methods

 `get_candidates`(rung_id) Get a candidate for promotion `get_point_max_resource`(point) Return the max resource value that has been tried for a point `has_rung_filled`(rung_id) Return True, if the rung[rung_id] is filled. `is_ready`([rung_id]) Return True, if the bracket is ready for next promote `promote`() Promote the first candidate that is found and return it `register`(point, objective) Register a point in the corresponding rung `sample`() Sample a new trial with lowest fidelity `seed_rng`(seed) Seed the state of the random number generator.
`get_candidates`(rung_id)[source]

Get a candidate for promotion

`get_point_max_resource`(point)[source]

Return the max resource value that has been tried for a point

`has_rung_filled`(rung_id)[source]

Return True, if the rung[rung_id] is filled.

`is_done`

Return True, if the last rung is filled.

`is_filled`

Return True if first rung with trials is filled

`is_ready`(rung_id=None)[source]

Return True, if the bracket is ready for next promote

`promote`()[source]

Promote the first candidate that is found and return it

The rungs are iterated over in reversed order, so that high rungs are prioritised for promotions. When a candidate is promoted, the loop is broken and the method returns the promoted point.

Note

All trials are part of the rungs, for any state. Only completed trials are eligible for promotion, i.e., only completed trials can be part of top-k. Lookup for promotion in rung l + 1 contains trials of any status.

`register`(point, objective)[source]

Register a point in the corresponding rung

`sample`()[source]

Sample a new trial with lowest fidelity

`seed_rng`(seed)[source]

Seed the state of the random number generator.

Parameters: seed – Integer seed for the random number generator.
class `orion.algo.hyperband.``Hyperband`(space, seed=None, repetitions=inf)[source]

Hyperband formulates hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations.`

Li, Lisha et al. “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization” Journal of Machine Learning Research, 18:1-52, 2018.

Parameters: space: `orion.algo.space.Space` Optimisation space with priors for each dimension. seed: None, int or sequence of int Seed for the random number generator used to sample new trials. Default: `None` repetitions: int Number of execution of Hyperband. Default is numpy.inf which means to run Hyperband until no new trials can be suggested. `fidelity_index` Compute the index of the point when fidelity is. `is_done` Return True, if all required execution been done. `state_dict` Return a state dict that can be used to reset the state of the algorithm.

Methods

 `get_id`(point[, ignore_fidelity]) Compute a unique hash for a point based on params, without fidelity level by default. `observe`(points, results) Observe evaluation results corresponding to list of points in space. `sample`(num, bracket[, buffer]) Sample new points from bracket `seed_rng`(seed) Seed the state of the random number generator. `set_state`(state_dict) Reset the state of the algorithm based on the given state_dict `suggest`([num]) Suggest a number of new sets of parameters.
`fidelity_index`

Compute the index of the point when fidelity is.

`get_id`(point, ignore_fidelity=True)[source]

Compute a unique hash for a point based on params, without fidelity level by default.

`is_done`

Return True, if all required execution been done.

`observe`(points, results)[source]

Observe evaluation results corresponding to list of points in space.

A simple random sampler though does not take anything into account.

`sample`(num, bracket, buffer=10)[source]

Sample new points from bracket

`seed_rng`(seed)[source]

Seed the state of the random number generator.

Parameters: seed – Integer seed for the random number generator.
`set_state`(state_dict)[source]

Reset the state of the algorithm based on the given state_dict

Parameters: state_dict – Dictionary representing state of an algorithm
`state_dict`

Return a state dict that can be used to reset the state of the algorithm.

`suggest`(num=1)[source]

Suggest a number of new sets of parameters.

Sample new points until first rung is filled. Afterwards waits for all trials to be completed before promoting trials to the next rung.

Parameters: num: int, optional Number of points to suggest. Defaults to 1. list of points or None A list of lists representing points suggested by the algorithm. The algorithm may opt out if it cannot make a good suggestion at the moment (it may be waiting for other trials to complete), in which case it will return None.
`orion.algo.hyperband.``compute_budgets`(max_resources, reduction_factor)[source]

Compute the budgets used for each execution of hyperband

`orion.algo.hyperband.``display_budgets`(budgets_tab, max_resources, reduction_factor)[source]

Display hyperband budget as a table in debug log