Hyperband Algorithm¶
Hyperband hyperparameter optimizer
- class orion.algo.hyperband.BudgetTuple(n_trials: int, resource_budget: int | float)[source]¶
Buget Tuple
- Attributes
n_trialsAlias for field number 0
resource_budgetAlias for field number 1
- property n_trials¶
Alias for field number 0
- property resource_budget¶
Alias for field number 1
- class orion.algo.hyperband.Hyperband(space: Space, seed: int | Sequence[int] | None = None, repetitions: int | float = inf)[source]¶
Hyperband formulates hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations.`
For more information on the algorithm, see original paper at http://jmlr.org/papers/v18/16-558.html.
Li, Lisha et al. “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization” Journal of Machine Learning Research, 18:1-52, 2018.
- Parameters
- space: `orion.algo.space.Space`
Optimisation space with priors for each dimension.
- seed: None, int or sequence of int
Seed for the random number generator used to sample new trials. Default:
None- repetitions: int
Number of executions for Hyperband. A single execution of Hyperband takes a finite budget of
(log(R)/log(eta) + 1) * (log(R)/log(eta) + 1) * R, andrepetitionsallows you to run multiple executions of Hyperband. Default isnumpy.infwhich means to run Hyperband until no new trials can be suggested.
- Attributes
executed_timesCounter for how many times Hyperband been executed
fidelity_indexCompute the dimension name of the space where fidelity is.
is_doneReturn True, if all required execution has been done.
state_dictReturn a state dict that can be used to reset the state of the algorithm.
Methods
observe(trials)Observe evaluation results corresponding to list of trials in space.
sample_from_bracket(bracket, num)Sample new trials from bracket
seed_rng(seed)Seed the state of the random number generator.
set_state(state_dict)Reset the state of the algorithm based on the given state_dict
suggest(num)Suggest a number of new sets of parameters.
append_brackets
create_bracket
create_brackets
promote
register_samples
sample
seed_brackets
- property fidelity_index: str¶
Compute the dimension name of the space where fidelity is.
There is always a fidelity dimension in Hyperband. If there isn’t one, raises an exception.
- observe(trials: list[Trial]) None[source]¶
Observe evaluation results corresponding to list of trials in space.
- Parameters
- trials: list of ``orion.core.worker.trial.Trial``
Trials from a
orion.algo.space.Space.
- sample_from_bracket(bracket: BracketT, num: int) list[Trial][source]¶
Sample new trials from bracket
- seed_rng(seed: int | Sequence[int] | None) None[source]¶
Seed the state of the random number generator.
- Parameters
seed – Integer seed for the random number generator.
- set_state(state_dict: dict) None[source]¶
Reset the state of the algorithm based on the given state_dict
- Parameters
state_dict – Dictionary representing state of an algorithm
- property state_dict: dict¶
Return a state dict that can be used to reset the state of the algorithm.
- suggest(num: int) list[Trial][source]¶
Suggest a number of new sets of parameters.
Sample new points until first rung is filled. Afterwards waits for all trials to be completed before promoting trials to the next rung.
- Parameters
- num: int, optional
Number of points to suggest. Defaults to 1.
- Returns
- list of orion.core.worker.trial:Trial or None
A list of trials suggested by the algorithm. The algorithm may opt out if it cannot make a good suggestion at the moment (it may be waiting for other trials to complete), in which case it will return None.
- class orion.algo.hyperband.HyperbandBracket(owner: Owner, budgets: list[BudgetTuple], repetition_id: int)[source]¶
Bracket of rungs for the algorithm Hyperband.
- Parameters
- owner: `Hyperband` algorithm
The hyperband algorithm object which this bracket will be part of.
- budgets: list of tuple
Each tuple gives the (n_trials, resource_budget) for the respective rung.
- repetition_id: int
The id of hyperband execution this bracket belongs to
- Attributes
Methods
get_candidates(rung_id)Get a candidate for promotion
get_trial_max_resource(trial)Return the max resource value that has been tried for a trial
has_rung_filled(rung_id)Return True, if the rung[rung_id] is filled.
is_ready([rung_id])Return True, if the bracket is ready for next promote
promote(num)Promote the first candidate that is found and return it
register(trial)Register a trial in the corresponding rung
seed_rng(seed)Seed the state of the random number generator.
get_sample
set_state
- get_candidates(rung_id: int) list[Trial][source]¶
Get a candidate for promotion
- Raises
- TypeError
If get_candidates is called before the entire rung is completed.
- get_trial_max_resource(trial: Trial) int | float[source]¶
Return the max resource value that has been tried for a trial
- is_ready(rung_id: int | None = None) bool[source]¶
Return True, if the bracket is ready for next promote
- promote(num: int) list[Trial][source]¶
Promote the first candidate that is found and return it
The rungs are iterated over in reversed order, so that high rungs are prioritised for promotions. When a candidate is promoted, the loop is broken and the method returns the promoted trial.
Note
All trials are part of the rungs, for any state. Only completed trials are eligible for promotion, i.e., only completed trials can be part of top-k. Lookup for promotion in rung l + 1 contains trials of any status.