Hyperband Algorithm

A Novel Bandit-Based Approach to Hyperparameter Optimization

Implement Hyperband to exploit configurations with fixed resource efficiently

class orion.algo.hyperband.BudgetTuple(n_trials, resource_budget)[source]
Attributes
n_trials

Alias for field number 0

resource_budget

Alias for field number 1

property n_trials

Alias for field number 0

property resource_budget

Alias for field number 1

class orion.algo.hyperband.Hyperband(space: Space, seed: int | Sequence[int] | None = None, repetitions: int | float = inf)[source]

Hyperband formulates hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations.`

For more information on the algorithm, see original paper at http://jmlr.org/papers/v18/16-558.html.

Li, Lisha et al. “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization” Journal of Machine Learning Research, 18:1-52, 2018.

Parameters
space: `orion.algo.space.Space`

Optimisation space with priors for each dimension.

seed: None, int or sequence of int

Seed for the random number generator used to sample new trials. Default: None

repetitions: int

Number of executions for Hyperband. A single execution of Hyperband takes a finite budget of (log(R)/log(eta) + 1) * (log(R)/log(eta) + 1) * R, and repetitions allows you to run multiple executions of Hyperband. Default is numpy.inf which means to run Hyperband until no new trials can be suggested.

Attributes
executed_times

Counter for how many times Hyperband been executed

fidelity_index

Compute the dimension name of the space where fidelity is.

is_done

Return True, if all required execution has been done.

state_dict

Return a state dict that can be used to reset the state of the algorithm.

Methods

observe(trials)

Observe evaluation results corresponding to list of trials in space.

sample_from_bracket(bracket, num)

Sample new trials from bracket

seed_rng(seed)

Seed the state of the random number generator.

set_state(state_dict)

Reset the state of the algorithm based on the given state_dict

suggest(num)

Suggest a number of new sets of parameters.

append_brackets

create_bracket

create_brackets

promote

register_samples

sample

seed_brackets

property executed_times: int

Counter for how many times Hyperband been executed

property fidelity_index: str

Compute the dimension name of the space where fidelity is.

There is always a fidelity dimension in Hyperband. If there isn’t one, raises an exception.

property is_done: bool

Return True, if all required execution has been done.

observe(trials: list[Trial]) None[source]

Observe evaluation results corresponding to list of trials in space.

Parameters
trials: list of ``orion.core.worker.trial.Trial``

Trials from a orion.algo.space.Space.

sample_from_bracket(bracket: HyperbandBracket, num: int) list[Trial][source]

Sample new trials from bracket

seed_rng(seed: int | Sequence[int] | None) None[source]

Seed the state of the random number generator.

Parameters

seed – Integer seed for the random number generator.

set_state(state_dict: dict) None[source]

Reset the state of the algorithm based on the given state_dict

Parameters

state_dict – Dictionary representing state of an algorithm

property state_dict: dict

Return a state dict that can be used to reset the state of the algorithm.

suggest(num: int) list[Trial][source]

Suggest a number of new sets of parameters.

Sample new points until first rung is filled. Afterwards waits for all trials to be completed before promoting trials to the next rung.

Parameters
num: int, optional

Number of points to suggest. Defaults to 1.

Returns
list of orion.core.worker.trial:Trial or None

A list of trials suggested by the algorithm. The algorithm may opt out if it cannot make a good suggestion at the moment (it may be waiting for other trials to complete), in which case it will return None.

class orion.algo.hyperband.HyperbandBracket(owner: Owner, budgets: list[BudgetTuple], repetition_id: int)[source]

Bracket of rungs for the algorithm Hyperband.

Parameters
owner: `Hyperband` algorithm

The hyperband algorithm object which this bracket will be part of.

budgets: list of tuple

Each tuple gives the (n_trials, resource_budget) for the respective rung.

repetition_id: int

The id of hyperband execution this bracket belongs to

Attributes
is_done

Return True, if the last rung is filled.

is_filled

Return True if first rung with trials is filled

remainings
seed
state_dict

Methods

get_candidates(rung_id)

Get a candidate for promotion

get_trial_max_resource(trial)

Return the max resource value that has been tried for a trial

has_rung_filled(rung_id)

Return True, if the rung[rung_id] is filled.

is_ready([rung_id])

Return True, if the bracket is ready for next promote

promote(num)

Promote the first candidate that is found and return it

register(trial)

Register a trial in the corresponding rung

sample(num)

Sample a new trial with lowest fidelity

seed_rng(seed)

Seed the state of the random number generator.

get_sample

set_state

get_candidates(rung_id: int) list[Trial][source]

Get a candidate for promotion

Raises
TypeError

If get_candidates is called before the entire rung is completed.

get_trial_max_resource(trial: Trial) int | float[source]

Return the max resource value that has been tried for a trial

has_rung_filled(rung_id: int) bool[source]

Return True, if the rung[rung_id] is filled.

property is_done: bool

Return True, if the last rung is filled.

property is_filled: bool

Return True if first rung with trials is filled

is_ready(rung_id: int | None = None) bool[source]

Return True, if the bracket is ready for next promote

promote(num: int) list[Trial][source]

Promote the first candidate that is found and return it

The rungs are iterated over in reversed order, so that high rungs are prioritised for promotions. When a candidate is promoted, the loop is broken and the method returns the promoted trial.

Note

All trials are part of the rungs, for any state. Only completed trials are eligible for promotion, i.e., only completed trials can be part of top-k. Lookup for promotion in rung l + 1 contains trials of any status.

register(trial: orion.core.worker.trial.Trial) None[source]

Register a trial in the corresponding rung

sample(num: int) list[Trial][source]

Sample a new trial with lowest fidelity

seed_rng(seed: int | Sequence[int] | None) None[source]

Seed the state of the random number generator.

Parameters

seed – Integer seed for the random number generator.

class orion.algo.hyperband.RungDict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list.  For example:  dict(one=1, two=2)[source]
orion.algo.hyperband.compute_budgets(max_resources: float, reduction_factor: float) list[list[BudgetTuple]][source]

Compute the budgets used for each execution of hyperband

orion.algo.hyperband.display_budgets(budgets_tab: dict[int, list[BudgetTuple]], max_resources: Any, reduction_factor: Any) None[source]

Display hyperband budget as a table in debug log