PyTorch MNIST¶
This is a simple tutorial on running hyperparameter search with Oríon on Pytorch’s MNIST example
Installation and setup¶
Make sure Oríon is installed (Installing Oríon) and the database is configured (Database Setup).
Then install pytorch
and torchvision
and clone the
PyTorch examples repository:
$ pip3 install torch torchvision
$ git clone https://github.com/pytorch/examples.git
Adapting the code for Oríon¶
To use Oríon with any code we need to do two things
- import the
orion.client.report_objective
helper function - call report_objective on the final objective output to be minimized (e.g. final test error rate)
After cloning pytorch examples repository, cd to mnist folder:
$ cd examples/mnist
1. At the top of the file, below the imports, add one line of import for the helper function
orion.client.report_objective()
:
from orion.client import report_objective
- We need the test error rate so we’re going to add a line to the function
test()
to return it
return 1 - (correct / len(test_loader.dataset))
Finally, we get back this test error rate and call report_objective
to
return the final objective value to Oríon. Note that report_objective
is meant to
be called only once because Oríon only looks at 1 'objective'
value per run.
test_error_rate = test(args, model, device, test_loader)
report_objective(test_error_rate)
Execution¶
Once the script is adapted, optimizing the hyper-parameters with Oríon is rather simple. Normally you would call the script the following way.
$ python main.py --lr 0.01
To use it with Oríon, you simply need to prepend the call with
orion hunt -n <experiment name>
and specify the hyper-parameter prior
distributions.
$ orion hunt -n orion-tutorial python main.py --lr~'loguniform(1e-5, 1.0)'
This commandline call will sequentially execute python main.py --lr=<value>
with random
values sampled from the distribution loguniform(1e-5, 1.0)
. We support all
distributions from scipy.stats, plus choices()
for categorical
hyper-parameters (similar to numpy’s choice function).
Experiments are interruptible, meaning that you can stop them either with
<ctrl-c>
or with kill signals. If your script is not resumable automatically then resuming an
experiment will restart your script from scratch.
You can resume experiments using the same commandline or simply by specifying the name of the experiment.
$ orion hunt -n orion-tutorial
Note that experiment names are unique, you cannot create two different experiment with the same name.
You can also register experiments without executing them.
$ orion hunt --init-only -n orion-tutorial python main.py --lr~'loguniform(1e-5, 1.0)'
Results¶
When an experiment reaches its termination criterion, basically max-trials
, it will print the
following statistics if Oríon is called with -v
or -vv
.
RESULTS
=======
{'best_evaluation': 0.05289999999999995,
'best_trials_id': 'b7a741e70b75f074208942c1c2c7cd36',
'duration': datetime.timedelta(0, 49, 751548),
'finish_time': datetime.datetime(2018, 8, 30, 1, 8, 2, 562000),
'start_time': datetime.datetime(2018, 8, 30, 1, 7, 12, 810452),
'trials_completed': 5}
BEST PARAMETERS
===============
[{'name': '/lr', 'type': 'real', 'value': 0.012027705702344259}]
These results can be printed in terminal later on with the command info or fetched using the library API.
$ orion info -n orion-tutorial
Notes¶
We use test error rate for sake of simplicity, because the script does not contain validation dataset loader as-is, but we should never optimize our hyper-parameters on the test set and instead always use a validation set.
Oríon will always minimize the objective so make sure you never try to optimize something like the accuracy of the model unless you are looking for very very bad models.
You can also report results of types 'gradient'
and 'constraint'
for
algorithms which require those parameters as well, or 'statistic'
for metrics
to be saved with the trial. See
report_results()
for more details.
Debugging¶
When preparing a script for hyper-parameter optimization, we recommend first testing with debug
mode. This will use an in-memory database which will be flushed at the end of execution. If you
don’t use --debug
you will likely quickly fill your database with broken experiments.
$ orion --debug hunt -n orion-tutorial python main.py --lr~'loguniform(1e-5, 1.0)'
Hunting Options¶
$ orion hunt --help
Oríon arguments (optional):
These arguments determine orion's behaviour
-n stringID, --name stringID
experiment's unique name; (default: None - specified
either here or in a config)
-u USER, --user USER user associated to experiment's unique name; (default:
$USER - can be overriden either here or in a config)
-c path-to-config, --config path-to-config
user provided orion configuration file
--max-trials # number of trials to be completed for the experiment.
This value will be saved within the experiment
configuration and reused across all workers to
determine experiment's completion. (default: inf/until
preempted)
--worker-trials # number of trials to be completed for this worker. If
the experiment is completed, the worker will die even
if it did not reach its maximum number of trials
(default: inf/until preempted)
--working-dir WORKING_DIR
Set working directory for running experiment.
--pool-size # number of simultaneous trials the algorithm should
suggest. This is useful if many workers are executed
in parallel and the algorithm has a strategy to sample
non-independant trials simultaneously. Otherwise, it
is better to leave `pool_size` to 1 and set a Strategy
for Oríon's producer. Note that this option is not usefull useless you
know the algorithm have a strategy to produce multiple trials
simultaneously. If you have any doubt, leave it to 1.
(default: 1)
name
The unique name of the experiment.
user
Username used to identify the experiments of a user. The default value is the system’s username $USER.
config
Configuration file for Oríon which may define the database, the algorithm and all options of the
command hunt, including name
, pool-size
and max-trials
.
max-trials
The maximum number of trials tried during an experiment.
worker-trials
The maximum number of trials to be executed by a worker (a single call to orion hunt [...]
).
working-dir
The directory where configuration files are created. If not specified, Oríon will create a temporary directory that will be removed at end of execution of the trial.
pool-size
The number of trials which are generated by the algorithm each time it is interrogated. This is
useful if many workers are executed in parallel and the algorithm has a strategy to sample
non-independant trials simultaneously. Otherwise, it is better to leave pool_size
to its default
value 1. Note that this option is not usefull useless you know the algorithm have a strategy
to produce multiple trials simultaneously. If you have any doubt, leave it to 1. :)