Hyperband Algorithm¶
A Novel Bandit-Based Approach to Hyperparameter Optimization¶
Implement Hyperband to exploit configurations with fixed resource efficiently
-
class
orion.algo.hyperband.
Bracket
(hyperband, budgets, repetition_id)[source]¶ Bracket of rungs for the algorithm Hyperband.
Parameters: - hyperband: `Hyperband` algorithm
The hyperband algorithm object which this bracket will be part of.
- budgets: list of tuple
Each tuple gives the (n_trials, resource_budget) for the respective rung.
- repetition_id: int
The id of hyperband execution this bracket belongs to
Attributes: Methods
get_candidates
(rung_id)Get a candidate for promotion get_point_max_resource
(point)Return the max resource value that has been tried for a point has_rung_filled
(rung_id)Return True, if the rung[rung_id] is filled. is_ready
([rung_id])Return True, if the bracket is ready for next promote promote
()Promote the first candidate that is found and return it register
(point, objective)Register a point in the corresponding rung sample
()Sample a new trial with lowest fidelity seed_rng
(seed)Seed the state of the random number generator. -
get_point_max_resource
(point)[source]¶ Return the max resource value that has been tried for a point
-
is_done
¶ Return True, if the last rung is filled.
-
is_filled
¶ Return True if first rung with trials is filled
-
promote
()[source]¶ Promote the first candidate that is found and return it
The rungs are iterated over in reversed order, so that high rungs are prioritised for promotions. When a candidate is promoted, the loop is broken and the method returns the promoted point.
Note
All trials are part of the rungs, for any state. Only completed trials are eligible for promotion, i.e., only completed trials can be part of top-k. Lookup for promotion in rung l + 1 contains trials of any status.
-
class
orion.algo.hyperband.
Hyperband
(space, seed=None, repetitions=inf)[source]¶ Hyperband formulates hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations.`
For more information on the algorithm, see original paper at http://jmlr.org/papers/v18/16-558.html.
Li, Lisha et al. “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization” Journal of Machine Learning Research, 18:1-52, 2018.
Parameters: - space: `orion.algo.space.Space`
Optimisation space with priors for each dimension.
- seed: None, int or sequence of int
Seed for the random number generator used to sample new trials. Default:
None
- repetitions: int
Number of execution of Hyperband. Default is numpy.inf which means to run Hyperband until no new trials can be suggested.
Attributes: fidelity_index
Compute the index of the point when fidelity is.
is_done
Return True, if all required execution been done.
state_dict
Return a state dict that can be used to reset the state of the algorithm.
Methods
get_id
(point[, ignore_fidelity])Compute a unique hash for a point based on params, without fidelity level by default. observe
(points, results)Observe evaluation results corresponding to list of points in space. sample
(num, bracket[, buffer])Sample new points from bracket seed_rng
(seed)Seed the state of the random number generator. set_state
(state_dict)Reset the state of the algorithm based on the given state_dict suggest
([num])Suggest a number of new sets of parameters. -
fidelity_index
¶ Compute the index of the point when fidelity is.
-
get_id
(point, ignore_fidelity=True)[source]¶ Compute a unique hash for a point based on params, without fidelity level by default.
-
is_done
¶ Return True, if all required execution been done.
-
observe
(points, results)[source]¶ Observe evaluation results corresponding to list of points in space.
A simple random sampler though does not take anything into account.
-
seed_rng
(seed)[source]¶ Seed the state of the random number generator.
Parameters: seed – Integer seed for the random number generator.
-
set_state
(state_dict)[source]¶ Reset the state of the algorithm based on the given state_dict
Parameters: state_dict – Dictionary representing state of an algorithm
-
state_dict
¶ Return a state dict that can be used to reset the state of the algorithm.
-
suggest
(num=1)[source]¶ Suggest a number of new sets of parameters.
Sample new points until first rung is filled. Afterwards waits for all trials to be completed before promoting trials to the next rung.
Parameters: - num: int, optional
Number of points to suggest. Defaults to 1.
Returns: - list of points or None
A list of lists representing points suggested by the algorithm. The algorithm may opt out if it cannot make a good suggestion at the moment (it may be waiting for other trials to complete), in which case it will return None.