Local Parameter Importance

Provide tools to calculate Local Parameter Importance

orion.analysis.lpi_utils.compute_variances(grid)[source]

Compute variance for each hyperparameters

orion.analysis.lpi_utils.lpi(trials, space, mode='best', model='RandomForestRegressor', n_points=20, n_runs=10, **kwargs)[source]

Calculates the Local Parameter Importance for a collection of orion.core.worker.trial.Trial.

For more information on the metric, see original paper at https://ml.informatik.uni-freiburg.de/papers/18-LION12-CAVE.pdf.

Biedenkapp, André, et al. “Cave: Configuration assessment, visualization and evaluation.” International Conference on Learning and Intelligent Optimization. Springer, Cham, 2018.

Parameters
trials: DataFrame or dict

A dataframe of trials containing, at least, the columns ‘objective’ and ‘id’. Or a dict equivalent.

space: Space object

A space object from an experiment.

mode: str

Mode to compute the LPI. - best: Take the best trial found as the anchor for the LPI - linear: Recompute LPI for all values on a grid

model: str

Name of the regression model to use. Can be one of - AdaBoostRegressor - BaggingRegressor - ExtraTreesRegressor - GradientBoostingRegressor - RandomForestRegressor (Default)

n_points: int

Number of points to compute the variances. Default is 20.

n_runs: int

Number of runs to compute the standard error of the LPI. Default is 10.

``**kwargs``

Arguments for the regressor model.

Returns
DataFrame

LPI value for each parameter. If mode is linear, then a list of param values and LPI metrics are returned in a DataFrame format.

orion.analysis.lpi_utils.make_grid(point, space, model, n_points)[source]

Build a grid based on point.

The shape of the grid will be
(number of hyperparameters,

number of points n_points, number of hyperparameters + 1)

Last column is the objective predicted by the model for a given point.

Parameters
point: numpy.ndarray

A tuple representation of the best trials, (hyperparameters + objective)

space: Space object

A space object from an experiment. It must be flattened and linearized.

model: `sklearn.base.RegressorMixin`

Trained regressor used to compute predictions on the grid

n_points: int

Number of points for each dimension on the grid.