Search Space¶
The search space is defined by the name of the hyperparameters to optimize and their corresponding distribution priors. These priors are used by the optimization algorithms to sample values or adjust the exploration.
Priors¶
We support all of scipy distributions out of the box. With the exception of the functions
outlined below, every distribution in scipy.stats
can be used using the same original function
signature.
uniform(low, high)
¶
The uniform distribution is redefining scipy’s function signature as
uniform(low, high)
instead uniform(low, interval)
. This is to uniformize the interface with
numpy and python’s builtin random.uniform
.
loguniform(low, high)
¶
A logarithmic version of the uniform distribution.
normal(loc, scale)
¶
A wrapper for scipy.stats.norm
.
gaussian(loc, scale)
¶
An additional wrapper for scipy.stats.norm
.
choices(options)
¶
options
may be a list
of builtin python objects or a dict
of builtin python objects with
their corresponding probabilities. When passing a list
, each object has an equal probability of
being sampled choices(['all', 'equally', 'likely'])
.
To give different probabilities: choices({'likely': 0.8, 'unlikely': 0.2, 'impossible': 0})
.
You can pass strings, integers and floats, and all mixed together if needed:
choices([1.0, 2, 'three'])
.
fidelity(low, high, base=2)
¶
This prior is a special placeholder to define a Fidelity
dimension. The algorithms will not use
this prior to sample, but rather for multi-fidelity optimization. For an example of an algorithm
using multi-fidelity, you can look at the documentation of ASHA. To define the space of
multi-fidelity optimization, you must give the minimum (low
) and maximum (high
) values for
the fidelity and optionally the logarithmic base that should be used to explore this space.
Dimension Types¶
The dimensions are casted to special types according to their prior. This is critical to either allow algorithms to leverage type information (ex: some algorithms works better on integers) or automatically transform trial types to make them compatible with specific algorithms (ex: some algorithms cannot work on categorical values).
Real¶
All continous priors are automatically casted to Real
.
Integer¶
Discrete distributions of scipy are automatically casted to Integer
. All other
distributions can be casted to Integer
by setting discrete=True
(ex: uniform(low, high, discrete=True)
).
Warning
We recommend using continous priors with discrete=True
as there is an issue with scipy discrete distribution because of incorrect interval. Issue
is documented
here.
Categorical¶
Distribition of k possible categories, with no ordinal relationship. Only the prior
choices(options)
is casted to Categorical
.
Fidelity¶
Special placeholder to represent a fidelity dimension. Only the prior
fidelity(low, high, base=2) is casted to Fidelity
.
Special arguments¶
discrete
¶
ex: uniform(0, 10, discrete=True)
Argument to cast a continuous distribution into Integer. Defaults to False
.
default_value
¶
ex: uniform(0, 10, default_value=5)
Dimensions can be set to a default value so that commandline call insert can support insertion without specifing this hyperparameter, assigning it the default value. This is also usefull in when using the Experiment Version Control, so that experiments where an hyperparameter is deleted or added can adapt trials from other experiments by using the default value.
precision
¶
ex: loguniform(1e-5, 1e-2, precision=2)
Argument to sample a continuous distribution up to the requested precision. Defaults to 4
.
The above example would sample values such as 6.7e-4 or 2.9e-3 but not 6.789e-4.
shape
¶
ex: uniform(0, 10, shape=2)
Some hyper-parameters may have multiple dimensions. This can be set using shape
Configuration¶
You can configure the search space of your experiment on the commandline call directly or in a configuration file used by your script.
Commandline¶
Any argument in commandline with the form --arg~aprior(some, args)
will be detected as a search
space dimension by Oríon. You can also use the verbose format --arg 'orion~aprior(some, args)'
.
Note that some shells may not play nicely with the parenthesis. You can format your command in the
following way to avoid this problem --arg~'aprior(some, args)'
.
Configuration file¶
You can use configuration files to define search space with placeholder
'orion~dist(*args, **kwargs)'
in yaml and json files or
name~dist(*args, **kwargs)
in any other text-based file.
By default Oríon will only consider the file passed through the argument --config
as a
configuration file for the user script. To change this behavior, take a look at the documentation
here.
This should not be confused with the argument --config
of orion hunt
,
which is the configuration of Oríon. We are here referring the configuration of the user script,
represented with my_script_config.txt
in the following example.
orion hunt --config my_orion_config.yaml ./my_script --config my_script_config.txt
Here is an example of a configuration file with yaml
lr: 'orion~loguniform(1e-5, 1.0)'
model:
activation: "orion~choices(['sigmoid', 'relu'])"
hiddens: 'orion~randint(100, 1000)'
Here is another example with json
{
"lr": "orion~loguniform(1e-5, 1.0)"
"model": {
"activation": "orion~choices(['sigmoid', 'relu'])"
"hiddens": "orion~randint(100, 1000)"
}
}
And here is an example with python! Note that for other files than for json and yaml, the
placeholders must be defined as name~dist(*args, **kwargs)
. Also, note that the code cannot be executed as is,
but once Oríon makes the substitution it will.
def my_config():
lr = lr~loguniform(1e-5, 1.0)
activations = model/activations~choices(['sigmoid', 'relu'])
nhiddens = model/hiddens~randint(100, 1000)
layers = []
for layer in range(model/nlayers~randint(3, 10)):
nhiddens /= 2
layers.append(nhiddens)
return lr, layers
Oríon could generate a script like this one for instance.
def my_config():
lr = 0.001
activations = 'relu'
nhiddens = 100
layers = []
for layer in range(4):
nhiddens /= 2
layers.append(nhiddens)
return lr, layers
When a trial is executed, a copy of the configuration file is created inside trial.working_dir
and the corresponding path is passed to the user script instead of the original path.
Notes¶
Tranformations¶
Some algorithms only support limited types of dimensions. In such case, these algorithms define the type required, and then a wrapper transforms the space to make it compatible.
Real¶
Categorical
are casted toInteger
(low=0, high=number of categories) and then to one-hot (Real
with space=number of categories) to break ordinal relationship. (probabilities are lost if defined)
Integer¶
Categorical
are casted toInteger
(low=0, high=number of categories). (probabilities are lost if defined)
Conditional dependencies¶
There is currently no support for conditional dependencies between dimensions.
Conditional dependencies arises in situations where some hyperparameter defines which algorithm to
use and each algorithm have its own
set of different hyperparameter. We plan to support this in the future by replacing our current
Space
implementation by ConfigSpace. This should not change the current interface and
only add more special arguments. You can see the state of our plan in our Roadmap.