{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Local Parameter Importance\n\n.. hint:: \n\n   Conveys a very compact measure of the importance of the different hyperparameters\n   to achieve the best objective found so far.\n\nThe local parameter importance measures the variance of the results when\nvarying one hyperparameter and keeping all other fixes [Biedenkapp2018]_.\n\nGiven a best set of hyperparameters, we separately build a grid\nfor each hyperparameter and compute the variance of the results when keeping all other\nhyperparameter fixed to their values in the best set. In order to infer these results,\nwe train a regression model from scikit-learn (by default RandomForestRegressor)\non the trial history of the experiment, and use it to predict the objective.\nThe ratio of variance for one hyperparameter versus the sum of variances for all hyperparameters\nis used as the local parameter importance metric.\n\n.. autofunction:: orion.plotting.base.lpi\n    :noindex:\n\nThe local parameter importance plot can be executed directly from the ``experiment`` with\n``plot.lpi()`` as shown in the example below.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from orion.client import get_experiment\n\n# Specify the database where the experiments are stored. We use a local PickleDB here.\nstorage = dict(type=\"legacy\", database=dict(type=\"pickleddb\", host=\"../db.pkl\"))\n\n# Load the data for the specified experiment\nexperiment = get_experiment(\"2-dim-exp\", storage=storage)\nfig = experiment.plot.lpi()\nfig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "On this plot the x-axis shows the different hyperparameters while the y-axis gives the\nlocal parameter importance.\nThe error bars represent the standard deviation of the LPI based on 10 runs. Remember\nthat the LPI requires the training of a regression model. The initial state of this model can\ngreatly influence the results. This is why the computation of the LPI is done multiple times\n(10 times by default) using different random seeds. Here is an example\nsetting the number of points for the grids, the number of runs and the initial random seed\nfor the regression model.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "experiment.plot.lpi(n_points=10, n_runs=5, model_kwargs=dict(random_state=1))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can see that the learning rate had a larger impact than the dropout in achieving the best\nobjective.\nA search space of only 2 dimensions is easy to analyse visually however so the LPI\nhas little additional value in this example. We need to use an example with a larger search space\nto better show to utility of the LPI. We will load results from tutorial\n`sphx_glr_auto_tutorials_code_2_hyperband_checkpoint.py` for this.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Load the data for the specified experiment\nexperiment = get_experiment(\"hyperband-cifar10\", storage=storage)\nexperiment.plot.lpi()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "There is a large difference here between the most important hyperparameter (learning rate) and the\nleast important one (gamma).\n\nOne caveat of LPI is that the variance for each hyperparameters depends on the search space.\nIf the prior for one hyperparameter is narrow and fits the region of best values for this\nhyperparameter, then the variance will be low and this hyperparameter will be considered\nnon-important. It may be important, but it is not important to optimize it within this narrow\nsearch space. Another related issue, is that if one hyperparameter have a dramatic effect, it will\nlead to a variance so large that the other hyperparameters will seem unrelevant in comparison.\nThis is what we observe here with the learning rate. If we branch from the experiment and define\na narrowed prior for the learning rate, we will see that it becomes an unimportant\nhyperparameter.\nSee documentation on `EVC system` for more information on branching, or\n:py:func:`orion.client.build_experiment` for informations on ``branching`` arguments.\nOriginal learning rate prior was ``loguniform(1e-5, 0.1)``. We will narrow it to\n``loguniform(1e-3, 0.1)``.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from orion.client import build_experiment\n\n# Branch from \"hyperband-cifar10\" with a narrower search space.\nexperiment = build_experiment(\n    \"narrow-hyperband-cifar10\",\n    branching={\"branch_from\": \"hyperband-cifar10\"},\n    space={\n        \"epochs\": \"fidelity(1, 120, base=4)\",\n        \"learning_rate\": \"loguniform(1e-3, 0.1)\",\n        \"momentum\": \"uniform(0, 0.9)\",\n        \"weight_decay\": \"loguniform(1e-10, 1e-2)\",\n        \"gamma\": \"loguniform(0.97, 1)\",\n    },\n    storage=storage,\n)\n\nexperiment.plot.lpi()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The prior of the learning rate is arguably large, spanning over 3 orders of magnitude\n`(0.001, 0.1)`. Nevertheless, for this problem, most learning rates within this range\nleads to optimal results whenever the other hyperparameters are optimal. What you must remember\nis that defining to narrow search spaces may lead to misleading local parameter importance.\nSee `sphx_glr_auto_examples_plot_4_partial_dependencies.py` for a visualization to verify if\nthe search space you defined may be too narrow.\n\n## Special cases\n\n### Logarithmic scale\n\nDimensions with a logarithmic prior `search-space-prior-loguniform` are linearized before\nbeing passed to the regression model (using log(dim) instead of dim directly). This means the\nmodel is trained and will be making predictions in the linearized space.\n\n### Dimension with shape\n\nIf some dimensions have a `search-space-shape` larger than 1, they will be flattened so that\neach subdimension can be represented in the bar plot.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Load the data for the specified experiment\nexperiment = get_experiment(\"2-dim-shape-exp\", storage=storage)\nexperiment.plot.lpi()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In the example above, the dimension ``learning_rate~loguniform(1e-5, 1e-2, shape=3)``\nis flattened and represented with ``learning_rate[i]``. If the shape would be or more dimensions\n(ex: ``(3, 2)``), the indices would be ``learning_rate[i,j]`` with i=0..2 and j=0..1.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Categorical dimension\n\nCategorical dimensions are converted into integer values, so that the regression model\ncan handle them. The integers are simply indices that are assigned to each category in arbitrary\norder. Here is an example where dimension ``mt-join`` has the prior\n``choices(['mean', 'max', 'concat'])``.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Load the data for the specified experiment\nexperiment = get_experiment(\"3-dim-cat-shape-exp\", storage=storage)\nexperiment.plot.lpi()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Finally we save the image to serve as a thumbnail for this example. See\nthe guide\n`How to save <sphx_glr_auto_examples_how-tos_code_2_how_to_save.py>`\nfor more information on image saving.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "fig.write_image(\"../../docs/src/_static/lpi_thumbnail.png\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        ".. [Biedenkapp2018] Biedenkapp, Andr\u00e9, et al.\n   \"Cave: Configuration assessment, visualization and evaluation.\"\n   International Conference on Learning and Intelligent Optimization.\n   Springer, Cham, 2018.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}