Example with ikostrikov/pytorch-a2c-ppo-acktr

Note

If Oríon not installed: pip install orion

If the database is not setup, you can follow the instructions here: Setup Database.

Alternatively, you can test the example without setting up a database by using the option –debug, but note that all data gathered during an execution will be lost at the end of it.

Intro

Here, we are looking to update the ikostrikov/pytorch-a2c-ppo-acktr Reinforcement Learning algorithm implementations to use Oríon to find the best hyperparameters while trying to prevent overfitting via a validation set of random evaluation seeds in the environment.

What to change

To get the original repository of ikostrikov to work using Orion, we make a couple of changes.

First, we fork the original repo at commit hash: 4d95ec364c7303566c6a52fb0a254640e931609d

To the top of

main.py

we add:

#!/usr/bin/env python
from orion.client import report_results

and then we run

chmod +x main.py

To make it executable.

Then, we ensure that we evaluate on a separate set of hold out random seeds for the environment (which should be different than the test set and training seed). For MuJoCo environments where the random seed has an effect, we can simply set the random seed before a rollout. In Atari, we would have to create a new validation set of rollouts perhaps with different human starts.

The original repository doesn’t separate training/validation/testing so we add the required methods as follows. We create a file with functions for evaluation:

eval.py

Note

The execution with Oríon does not require the added evaluation methods for a validation set and could use the final training performance. However, for sake of adhering to best practices, we create a validation set method in eval.py.

And then simply add a registration for the evaluation after training our algorithm:

validation_returns = evaluate_with_seeds(eval_env,
                                         actor_critic,
                                         args.cuda,
                                         eval_env_seeds)

report_results([dict(
    name='validation_return',
    type='objective',
    value=np.mean(validation_returns))])

Now we’re ready to go to run orion’s hyperparameter optimization!

How to search for hyperparameters

orion -v hunt -n ppo_hopper \
  ./main.py --env-name "Hopper-v2" --algo ppo --use-gae --vis-interval 1 \
  --log-interval 1 --num-stack 1 --num-steps 2048 --num-processes 1 \
  --lr~'loguniform(1e-5, 1.0)' --entropy-coef 0 --value-loss-coef 1 \
  --ppo-epoch 10 --num-mini-batch 32 --gamma~'uniform(.95, .9995)' --tau 0.95 \
  --num-frames 1000000 --eval-env-seeds-file ./seeds.json --no-vis \
  --log-dir~trial.hash_name

Notice that this will search over the learning rates and gamma values, while setting the log directory name to be the hashed trial name provided in the orion database.

The full modified codebase for use with Oríon can be found on Gihub:

git clone https://github.com/Breakend/orion-pytorch-ppo-acktr-a2c