{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n# Parallel Coordinates\n\n.. hint:: \n\n Conveys a dense overview of the trial objectives in a multi-dimensional space.\n Helps identifying trends of best or worst hyperparameter values.\n\nThe parallel coordinates plot decomposes a search space of `n` dimensions into `n`\naxis so that the entire space can be visualized simultaneously. Each dimension\nis represented as a vertical axis and trials are represented as lines crossing each \naxis at the corresponding value of the hyperparameters. There is no obvious optimal ordering\nfor the vertical axis, and you will often find that changing the order helps better understanding\nthe data. Additionaly, the lines are plotted with graded colors based on the objective. The\ngradation is shown in a color bar on the right of the plot. Note that the objectives are added\nas the last axis is the plot as well.\n\n.. autofunction:: orion.plotting.base.parallel_coordinates\n :noindex:\n\nThe parallel coordinates plot can be executed directly from the ``experiment`` with\n``plot.parallel_coordinates()`` as shown in the example below.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from orion.client import get_experiment\n\n# Specify the database where the experiments are stored. We use a local PickleDB here.\nstorage = dict(type=\"legacy\", database=dict(type=\"pickleddb\", host=\"../db.pkl\"))\n\n# Load the data for the specified experiment\nexperiment = get_experiment(\"2-dim-exp\", storage=storage)\nfig = experiment.plot.parallel_coordinates()\nfig" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this basic example the parallel coordinates plot is marginally useful as there are only\n2 dimensions. It is possible however to identify the best performing values of ``dropout`` and\n``learning_rate``. The GIF below demonstrates how to select subsets of the\naxis to highlight the trials that corresponds to the best objectives.\n\n\n\n

Note

Hover is not supported by plotly at the moment.\n Feature request can be tracked `here `_.

\n\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Lets now load the results from tutorial\n`sphx_glr_auto_tutorials_code_2_hyperband_checkpoint.py` for an example with a larger search\nspace.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Load the data for the specified experiment\nexperiment = get_experiment(\"hyperband-cifar10\", storage=storage)\nfig = experiment.plot.parallel_coordinates()\nfig" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the large number of trials until trained for a few epochs is cluttering the entire\nplot. You can first select the trials with 120 epochs to clear the plot. Once that is done,\nWe can see that gamma and momentum had limited influence. Good trials can be found\nfor almost any values of gamma and momentum. On the other hand, learning rate\nand weight decay are clearly more optimal in lower values. You can try re-ordering the columns as\nshown in the animation below to see the connections between one hyperparameter and the objective.\n\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n\n\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also select a subset of hyperparameters to help with the visualization.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Load the data for the specified experiment\nfig = experiment.plot.parallel_coordinates(\n order=[\"epochs\", \"learning_rate\", \"weight_decay\"]\n)\nfig" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Special cases\n\n### Logarithmic scale\n\n

Note

Logarithmic scales are not supported yet. Contributions are welcome. :)\n See `issue `_.

\n\n### Dimension with shape\n\nIf some dimensions have a `search-space-shape` larger than 1, they will be flattened so that\neach subdimension can be represented in the parallel coordinates plot.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Load the data for the specified experiment\nexperiment = get_experiment(\"2-dim-shape-exp\", storage=storage)\nfig = experiment.plot.parallel_coordinates()\nfig" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the example above, the dimension ``learning_rate~loguniform(1e-5, 1e-2, shape=3)``\nis flattened and represented with ``learning_rate[i]``. If the shape would be or more dimensions\n(ex: ``(3, 2)``), the indices would be ``learning_rate[i,j]`` with i=0..2 and j=0..1.\n\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The flattened hyperparameters can be fully selected with ``params=['']``.\n\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "experiment.plot.parallel_coordinates(order=[\"/learning_rate\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Or a subset of the flattened hyperparameters can be selected with ``params=['[index]']``.\n\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "experiment.plot.parallel_coordinates(order=[\"/learning_rate[0]\", \"/learning_rate[1]\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Categorical dimension\n\nParallel coordinates plots can also render categorical dimensions, in which case the\ncategories are shown in an arbitrary order on the axis.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Load the data for the specified experiment\nexperiment = get_experiment(\"3-dim-cat-shape-exp\", storage=storage)\nfig = experiment.plot.parallel_coordinates()\nfig" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally we save the image to serve as a thumbnail for this example. See\nthe guide\n`How to save `\nfor more information on image saving.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "fig.write_image(\"../../docs/src/_static/pcp_thumbnail.png\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" } }, "nbformat": 4, "nbformat_minor": 0 }