{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Regret curves\n\n.. hint:: \n\n   Conveys how quickly the algorithm found the best hyperparameters.\n    \nThe regret is the difference between the achieved objective and the optimal objective.\nPlotted as a time series, it shows how fast an optimization algorithm approaches the best\nobjective. An equivalent way to visualize it is using the cumulative minimums instead of the\ndifferences. Or\u00edon plots the cumulative minimums so that the objective can easily be read\nfrom the y-axis.\n\n.. autofunction:: orion.plotting.base.regret\n    :noindex:\n\nThe regret plot can be executed directly from the ``experiment`` with ``plot.regret()`` as\nshown in the example below.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from orion.client import get_experiment\n\n# Specify the database where the experiments are stored. We use a local PickleDB here.\nstorage = dict(type=\"legacy\", database=dict(type=\"pickleddb\", host=\"../db.pkl\"))\n\n# Load the data for the specified experiment\nexperiment = get_experiment(\"random-rosenbrock\", storage=storage)\nfig = experiment.plot.regret()\nfig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The objective of the trials is overlayed as a scatter plot under the regret curve.\nThanks to this we can see whether the algorithm focused its optimization close to the\noptimum (if all points are close to the regret curve near the end) or if it explored far\nfrom it (if many points are far from the regret curve near the end). We can see in this example\nwith random search that the algorithm unsurpringly randomly explored the space.\nIf we plot the results from the algorithm TPE applied on the same task, we will see a very\ndifferent pattern\n(see how the results were generated in tutorial `sphx_glr_auto_tutorials_code_1_python_api.py`).\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "experiment = get_experiment(\"tpe-rosenbrock\", storage=storage)\nfig = experiment.plot.regret()\nfig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can hover your mouse over the trials to see the configuration of the corresponding trials.\nThe configuration of the trials is following this format:\n\n::\n\n   ID: <trial id>\n   value: <objective>\n   time: <time x-axis is ordered by>\n   parameters\n     <name>: <value>\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "For now, the only option to customize the regret plot is ``order_by``, the sorting order\nof the trials on the x-axis. Contributions are more than welcome to increase the customizability\nof the plot!\nBy default the sorting order is ``suggested``, the order the trials were suggested by the\noptimization algorithm. Other options are ``reserved``, the time the trials started being executed\nand ``completed``, the time the trials were completed.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "fig = experiment.plot.regret(order_by=\"completed\")\nfig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In this example, the order by ``suggested`` or by ``completed`` is the same,\nbut parallelized experiments can lead to different order of completed trials.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# TODO: Add documentation for `regrets()`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Finally we save the image to serve as a thumbnail for this example. See\nthe guide\n`How to save <sphx_glr_auto_examples_how-tos_code_2_how_to_save.py>`\nfor more information on image saving.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "fig.write_image(\"../../docs/src/_static/regret_thumbnail.png\")"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}