Profet Task modules

Surrogate (simulated) tasks created using the Profet algorithm.

For a detailed description of Profet, see original paper at https://arxiv.org/abs/1905.12982 or source code at https://github.com/EmuKit/emukit/tree/main/emukit/examples/profet

Klein, Aaron, Zhenwen Dai, Frank Hutter, Neil Lawrence, and Javier Gonzalez. “Meta-surrogate benchmarking for hyperparameter optimization.” Advances in Neural Information Processing Systems 32 (2019): 6270-6280.

class orion.benchmark.task.profet.ProfetFcNetTask(max_trials: int = 100, input_dir: Union[Path, str] = 'profet_data', checkpoint_dir: Optional[Union[Path, str]] = None, model_config: Optional[MetaModelConfig] = None, device: Optional[Union[str, Any]] = None, with_grad: bool = False)[source]

Simulated Task consisting in training a fully-connected network.

Methods

ModelConfig([benchmark, task_id, seed, ...])

Config for training the Profet model on an FcNet task.

call(learning_rate, batch_size, ...)

Get the value of the sampled objective function at the given point (hyper-parameters).

get_search_space()

Return the search space for the task objective function

class ModelConfig(benchmark: str = 'fcnet', task_id: int = 0, seed: int = 123, num_burnin_steps: int = 50000, num_steps: int = 13000, mcmc_thining: int = 100, lr: float = 0.01, batch_size: int = 5, max_samples: Optional[int] = None, n_inducing_lvm: int = 50, max_iters: int = 10000, n_samples_task: int = 500)[source]

Config for training the Profet model on an FcNet task.

benchmark: str = 'fcnet'

Name of the benchmark.

c_max: ClassVar[float] = 14718.31848526001

The maximum of the C array.

c_min: ClassVar[float] = 0.0

The minimum of the C array.

hidden_space: ClassVar[int] = 5

Size of the hidden space for this benchmark.

json_file_name: ClassVar[str] = 'data_sobol_fcnet.json'

Name of the json file that contains the data of this benchmark.

log_cost: ClassVar[bool] = True

Whether to apply numpy.log onto the raw data for the cost of each point.

log_target: ClassVar[bool] = False

Whether to apply numpy.log onto the raw data for the y of each point.

normalize_targets: ClassVar[bool] = False

Whether to normalize the targets (y), by default False.

shapes: ClassVar[Tuple[Tuple[int, ...], Tuple[int, ...], Tuple[int, ...]]] = ((600, 6), (27, 600), (27, 600))

The shapes of the X, Y and C arrays of the dataset.

y_max: ClassVar[float] = 1.0

The maximum of the Y array.

y_min: ClassVar[float] = 0.0

The minimum of the Y array.

call(learning_rate: float, batch_size: int, units_layer1: int, units_layer2: int, dropout_rate_l1: float, dropout_rate_l2: float) List[Dict][source]

Get the value of the sampled objective function at the given point (hyper-parameters).

If self.with_grad is set, also returns the gradient of the objective function with respect to the inputs.

Parameters
**kwargs

Dictionary of hyper-parameters.

Returns
List[Dict]

Result dictionaries: objective and optionally gradient.

Raises
ValueError

If the input isn’t of a supported type.

get_search_space() Dict[str, str][source]

Return the search space for the task objective function

class orion.benchmark.task.profet.ProfetForresterTask(max_trials: int = 100, input_dir: Union[Path, str] = 'profet_data', checkpoint_dir: Optional[Union[Path, str]] = None, model_config: Optional[MetaModelConfig] = None, device: Optional[Union[str, Any]] = None, with_grad: bool = False)[source]

Simulated Task consisting in training a model on a variant of the Forrester function.

Methods

ModelConfig([benchmark, task_id, seed, ...])

Config for training the Profet model on a Forrester task.

call(x)

Get the value of the sampled objective function at the given point (hyper-parameters).

get_search_space()

Return the search space for the task objective function

class ModelConfig(benchmark: str = 'forrester', task_id: int = 0, seed: int = 123, num_burnin_steps: int = 50000, num_steps: int = 13000, mcmc_thining: int = 100, lr: float = 0.01, batch_size: int = 5, max_samples: Optional[int] = None, n_inducing_lvm: int = 50, max_iters: int = 10000, n_samples_task: int = 500)[source]

Config for training the Profet model on a Forrester task.

Methods

get_architecture

benchmark: str = 'forrester'

Name of the benchmark.

c_max: ClassVar[float] = 14718.31848526

The maximum of the C array.

c_min: ClassVar[float] = -18.049155413936802

The minimum of the C array.

get_architecture() Any

Callable that takes the input dimensionality and returns the network to be trained.

hidden_space: ClassVar[int] = 2

Size of the hidden space for this benchmark.

json_file_name: ClassVar[str] = 'data_sobol_forrester.json'

Name of the json file that contains the data of this benchmark.

log_cost: ClassVar[bool] = False

Whether to apply numpy.log onto the raw data for the cost of each point.

log_target: ClassVar[bool] = False

Whether to apply numpy.log onto the raw data for the y of each point.

normalize_targets: ClassVar[bool] = True

Whether to normalize the targets (y), by default False.

shapes: ClassVar[Tuple[Tuple[int, ...], Tuple[int, ...], Tuple[int, ...]]] = ((10, 1), (9, 10), (9, 10))

The shapes of the X, Y and C arrays of the dataset.

y_max: ClassVar[float] = 14718.31848526001

The maximum of the Y array.

y_min: ClassVar[float] = -18.049155413936802

The minimum of the Y array.

call(x: float) List[Dict][source]

Get the value of the sampled objective function at the given point (hyper-parameters).

If self.with_grad is set, also returns the gradient of the objective function with respect to the inputs.

Parameters
**kwargs

Dictionary of hyper-parameters.

Returns
List[Dict]

Result dictionaries: objective and optionally gradient.

Raises
ValueError

If the input isn’t of a supported type.

get_search_space() Dict[str, str][source]

Return the search space for the task objective function

class orion.benchmark.task.profet.ProfetSvmTask(max_trials: int = 100, input_dir: Union[Path, str] = 'profet_data', checkpoint_dir: Optional[Union[Path, str]] = None, model_config: Optional[MetaModelConfig] = None, device: Optional[Union[str, Any]] = None, with_grad: bool = False)[source]

Simulated Task consisting in training a Support Vector Machine.

Methods

ModelConfig([benchmark, task_id, seed, ...])

Config for training the Profet model on an SVM task.

call(C, gamma)

Get the value of the sampled objective function at the given point (hyper-parameters).

get_search_space()

Return the search space for the task objective function

class ModelConfig(benchmark: str = 'svm', task_id: int = 0, seed: int = 123, num_burnin_steps: int = 50000, num_steps: int = 13000, mcmc_thining: int = 100, lr: float = 0.01, batch_size: int = 5, max_samples: Optional[int] = None, n_inducing_lvm: int = 50, max_iters: int = 10000, n_samples_task: int = 500)[source]

Config for training the Profet model on an SVM task.

Methods

get_architecture(*[, classification, n_hidden])

benchmark: str = 'svm'

Name of the benchmark.

c_max: ClassVar[float] = 697154.4010462761

The maximum of the C array.

c_min: ClassVar[float] = 0.0

The minimum of the C array.

get_architecture(*, classification: bool = True, n_hidden: int = 500) Any

Callable that takes the input dimensionality and returns the network to be trained.

hidden_space: ClassVar[int] = 5

Size of the hidden space for this benchmark.

json_file_name: ClassVar[str] = 'data_sobol_svm.json'

Name of the json file that contains the data of this benchmark.

log_cost: ClassVar[bool] = True

Whether to apply numpy.log onto the raw data for the cost of each point.

log_target: ClassVar[bool] = False

Whether to apply numpy.log onto the raw data for the y of each point.

normalize_targets: ClassVar[bool] = False

Whether to normalize the targets (y), by default False.

shapes: ClassVar[Tuple[Tuple[int, ...], Tuple[int, ...], Tuple[int, ...]]] = ((200, 2), (26, 200), (26, 200))

The shapes of the X, Y and C arrays of the dataset.

y_max: ClassVar[float] = 1.0

The maximum of the Y array.

y_min: ClassVar[float] = 0.0

The minimum of the Y array.

call(C: float, gamma: float) List[Dict][source]

Get the value of the sampled objective function at the given point (hyper-parameters).

If self.with_grad is set, also returns the gradient of the objective function with respect to the inputs.

Parameters
**kwargs

Dictionary of hyper-parameters.

Returns
List[Dict]

Result dictionaries: objective and optionally gradient.

Raises
ValueError

If the input isn’t of a supported type.

get_search_space() Dict[str, str][source]

Return the search space for the task objective function

class orion.benchmark.task.profet.ProfetTask(max_trials: int = 100, input_dir: Union[Path, str] = 'profet_data', checkpoint_dir: Optional[Union[Path, str]] = None, model_config: Optional[MetaModelConfig] = None, device: Optional[Union[str, Any]] = None, with_grad: bool = False)[source]

Base class for Tasks that are generated using the Profet algorithm.

For more information on Profet, see original paper at https://arxiv.org/abs/1905.12982.

Klein, Aaron, Zhenwen Dai, Frank Hutter, Neil Lawrence, and Javier Gonzalez. “Meta-surrogate benchmarking for hyperparameter optimization.” Advances in Neural Information Processing Systems 32 (2019): 6270-6280.

Parameters
max_trialsint, optional

Max number of trials to run, by default 100

input_dirUnion[Path, str], optional

Input directory containing the data used to train the meta-model, by default None.

checkpoint_dirUnion[Path, str], optional

Directory used to save/load trained meta-models, by default None.

model_configMetaModelConfig, optional

Configuration options for the training of the meta-model, by default None

devicestr, optional

The device to use for training, by default None.

with_gradbool, optional

Whether the task should also return the gradients of the objective function with respect to the inputs. Defaults to False.

Attributes
configuration

Return the configuration of the task.

space

Methods

ModelConfig

alias of MetaModelConfig

call(**kwargs)

Get the value of the sampled objective function at the given point (hyper-parameters).

ModelConfig

alias of MetaModelConfig

call(**kwargs) List[Dict][source]

Get the value of the sampled objective function at the given point (hyper-parameters).

If self.with_grad is set, also returns the gradient of the objective function with respect to the inputs.

Parameters
**kwargs

Dictionary of hyper-parameters.

Returns
List[Dict]

Result dictionaries: objective and optionally gradient.

Raises
ValueError

If the input isn’t of a supported type.

property configuration

Return the configuration of the task.

class orion.benchmark.task.profet.ProfetXgBoostTask(max_trials: int = 100, input_dir: Union[Path, str] = 'profet_data', checkpoint_dir: Optional[Union[Path, str]] = None, model_config: Optional[MetaModelConfig] = None, device: Optional[Union[str, Any]] = None, with_grad: bool = False)[source]

Simulated Task consisting in fitting a Extreme-Gradient Boosting predictor.

Methods

ModelConfig([benchmark, task_id, seed, ...])

Config for training the Profet model on an XgBoost task.

call(learning_rate, gamma, ...)

Get the value of the sampled objective function at the given point (hyper-parameters).

get_search_space()

Return the search space for the task objective function

class ModelConfig(benchmark: str = 'xgboost', task_id: int = 0, seed: int = 123, num_burnin_steps: int = 50000, num_steps: int = 13000, mcmc_thining: int = 100, lr: float = 0.01, batch_size: int = 5, max_samples: Optional[int] = None, n_inducing_lvm: int = 50, max_iters: int = 10000, n_samples_task: int = 500)[source]

Config for training the Profet model on an XgBoost task.

benchmark: str = 'xgboost'

Name of the benchmark.

c_max: ClassVar[float] = 5485.541382551193

The maximum of the C array.

c_min: ClassVar[float] = 0.0

The minimum of the C array.

hidden_space: ClassVar[int] = 5

Size of the hidden space for this benchmark.

json_file_name: ClassVar[str] = 'data_sobol_xgboost.json'

Name of the json file that contains the data of this benchmark.

log_cost: ClassVar[bool] = True

Whether to apply numpy.log onto the raw data for the cost of each point.

log_target: ClassVar[bool] = True

Whether to apply numpy.log onto the raw data for the y of each point.

normalize_targets: ClassVar[bool] = True

Whether to normalize the targets (y), by default False.

shapes: ClassVar[Tuple[Tuple[int, ...], Tuple[int, ...], Tuple[int, ...]]] = ((800, 8), (11, 800), (11, 800))

The shapes of the X, Y and C arrays of the dataset.

y_max: ClassVar[float] = 3991387.335843141

The maximum of the Y array.

y_min: ClassVar[float] = 0.0

The minimum of the Y array.

call(learning_rate: float, gamma: float, l1_regularization: float, l2_regularization: float, nb_estimators: int, subsampling: float, max_depth: int, min_child_weight: int) List[Dict][source]

Get the value of the sampled objective function at the given point (hyper-parameters).

If self.with_grad is set, also returns the gradient of the objective function with respect to the inputs.

Parameters
**kwargs

Dictionary of hyper-parameters.

Returns
List[Dict]

Result dictionaries: objective and optionally gradient.

Raises
ValueError

If the input isn’t of a supported type.

get_search_space() Dict[str, str][source]

Return the search space for the task objective function