Developing algorithms¶
The documentation here explains how to create new algorithm plugins based on the cookiecutter.
Usage¶
Install Python requirements to use the template:
$ python -m pip install cookiecutter>=1.5 versioneer>=0.18 jinja2
Create a new project directly from the template on GitHub:
$ cookiecutter gh:Epistimio/cookiecutter-orion.algo
plugin_name []: skopt
author_name []: Xavier Bouthillier
author_short [Author Name]:
author_email []: xavier.bouthillier@umontreal.ca
github_username []: bouthilx
copyright [2019, Author Name]:
short_description [TODO]:
synopsis [TODO]:
algo_name []: BayesianOptimizer
algo_module_name [bayesianoptimizer]: bayes
Field |
Description |
|
|
Will be used for orion.algo.plugin_name |
|
|
For metadata of python package |
|
|
For metadata of python package |
|
|
For metadata of python package |
|
|
Username to build the url for installation |
|
|
For the BSD-3 license (You can change the license) |
|
|
For metadata of python package |
|
|
For documentation in algo module |
|
|
Name for the algorithm class |
|
|
Name of the algorithm module |
This will create the following package structure.
orion.algo.{plugin_name}
├── README.rst
├── setup.cfg
├── setup.py
├── MANIFEST.in
├── LICENSE (BSD License)
├── versioneer.py
├── tox.ini
├── dev-requirements.txt
├── doc
│ ├── requirements.txt
│ └── src
│ ├── conf.py
│ └── index.rst
├── tests
│ ├── requirements.txt
│ ├── integration_test.py
│ └── benchmark
│ ├── requirements.txt
│ ├── main.py
│ ├── rosenbrock.py
│ ├── {algoname}.yaml
│ ├── bayesopt.yaml
│ └── random_search.yaml
└── src
└── orion
└── algo
└── {plugin_name}
├── {algoname}.py
├── __init__.py
└── _version.py
The important files to modify are src/orion/algo/{plugin_name}/{module_name}.py
to implement the
algorithm and tests/benchmark/{algo_name}.yaml
to fill the arguments required for the algorithm
you implement.
LICENSE
Note that you are free to change the License, copyright is to your name.
versioneer.py
src/orion/algo/{plugin_name}/_version.py
This serves to version automatically your algo, just ignore these if you don’t plan to make releases.
tests
These are the automatically generated tests.
tests/benchmark
Automatically generated benchmark test using the yaml files created in the same folder.
doc
Automatically generated template for documentation
tox.ini
Tox file defining commands to run tests, build doc and publish code.
Implementation¶
requires = 'real'
Some algorithms requires the search space to be real. You can specify this requirement by adding the
attribute requires = 'real'
at the base of the class definition.In this case, the algorithm
wrapper in Orion’s core will convert the search space to real one before passing it to your
algorithm. This way the user can define discrete or categorital dimensions while using algorithms
that require a real space.
def __init__(self, space, seed=None):
The initialization of the algorithm must pass space and seed to super().__init__
, but must also
pass any other argument that must be part of the configuration of the algorithm. Any argument passed
to super()
will be assigned as an attribute to the algorithm and will be included in
algo.configuration
, which is used to save the configuration of the algorithm in the database.
def seed_rng(self, seed=None):
This method must seed the internal state of the algorithm so that it would always sample the same sequence of points.
@property
def state_dict(self):
The state dict is used to copy algorithms within the parallel strategy. All algorithms must provide a state dict to ensure that we reset it to a previous state.
def set_state(self, state_dict):
Stateful attributes of the algorithm are reset using the given state_dict
. Note that
set_state
must be compliant with state_dict
and use
the same structure.
def suggest(self, num=1):
The method to suggest new trials. The argument num=1
request the number of trials that the algorithm must sample. Note that it is possible to only
support num=1
and raise ValueError otherwise.
def observe(self, points, results):
The method to observe results of suggested trials. Note that observe may be called several times for
the same points. Make sure to handle this properly within your algorithm if this is problematic.
Points are passed as a list of lists, each list representing the value of the params in the order
defined in self.space
Tests¶
To test the freshly built package, you must first install the requirements. From within the new package, run
$ pip install -r tests/requirements.txt
You can then run the unit-tests with
$ pytest tests/integration_test.py
or using tox
$ tox -e py36
Note that the algorithm pre-built is random search so that you can start from a fully working environment and test your way through the modifications.
There is also the option of running the toy-benchmark to compare the performance of your algorithm with random search and bayesian optimization. First install the requirements.
$ pip install -r tests/benchmark/requirements.txt
And then execute the benchmark
$ pytest tests/benchmark/main.py
or using tox
$ tox -e benchmark
Finally, official plugins must follow the same code quality standards than orion.core
. Therefore
there is tests included in the pre-built package for flake8
and pylint
. You can execute them
with
$ tox -e flake8
and
$ tox -e pylint