{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n\n# Scaling workers\n\nThis tutorial shows simple examples to use the different backends of Or\u00edon to scale execution to\nmultiple workers in parallel.\nThe parallelization of Or\u00edon workers is explained in more details in section `parallelism`.\n\nWe will start with a basic example using scikit-learn.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import joblib\nimport numpy\nfrom sklearn import datasets\nfrom sklearn.model_selection import cross_validate\nfrom sklearn.svm import SVC\n\n\ndef main(C, gamma, tol, class_weight, joblib_backend=\"loky\"):\n\n digits = datasets.load_digits()\n\n X = digits.data\n y = digits.target\n\n model = SVC(kernel=\"rbf\", C=C, gamma=gamma, tol=tol, class_weight=class_weight)\n\n # Single metric evaluation using cross_validate\n with joblib.parallel_backend(joblib_backend):\n cv_results = cross_validate(model, X, y, cv=5)\n\n accuracy = numpy.mean(cv_results[\"test_score\"])\n error_rate = 1 - accuracy\n\n return [{\"name\": \"test_error_rate\", \"type\": \"objective\", \"value\": error_rate}]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We create a ``main()`` function that takes as arguments the hyperparameters for an SVM for\nclassification. We add the argument ``joblib_backend`` so that we can control which backend\nis used to paralellize the cross-validation. By default we will use ``loky`` for multi-processing.\nWe load the dataset ``digits`` and divide it in features ``X`` and targets ``y``.\nWe then create the model with the given hyperparameter values.\nWe use ``joblib.parallel_backend`` to execute the cross-validation on 5 folds in parallel\nfor more effiency. Finaly we compute the average accuracy, and convert it to test error\nrate since Or\u00edon is minimizing the objective and we would not want to minimize the accuracy.\nThe results is returned in the format required by Or\u00edon.\n\nWe will now create an experiment with Or\u00edon to optimize this function.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Specify the database where the experiments are stored. We use a local PickleDB here.\nstorage = {\n \"type\": \"legacy\",\n \"database\": {\n \"type\": \"pickleddb\",\n \"host\": \"./db.pkl\",\n },\n}\n\n# Specify optimization space for the SVM\nspace = {\n \"C\": \"loguniform(1e-6, 1e6)\",\n \"gamma\": \"loguniform(1e-8, 1e8)\",\n \"tol\": \"loguniform(1e-4, 1e-1)\",\n \"class_weight\": \"choices([None, 'balanced'])\",\n}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Joblib\n\n`Joblib`_ is a lightweight library for task parallel execution in Python. It is the default\nbackend used by Or\u00edon to spawn multiple workers.\n\nWe first build the experiment and limit it to 200 trials.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from orion.client import build_experiment\n\nexperiment = build_experiment(\n name=\"joblib_example\",\n max_trials=200,\n space=space,\n storage=storage,\n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since joblib is the default backend, we do not need to do anything special to use it.\nWe can simply call\n:meth:`ExperimentClient.workon() `\nand specify the number of workers that we want.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "experiment.workon(main, n_workers=4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is as simple as this.\n\nThe experiment backend is by default the one configured in the global configuration\n(`config_worker_executor` and `config_worker_executor_configuration`).\nIf you want to use a different backend while executing\n:meth:`ExperimentClient.workon() `, you can use\n:meth:`ExperimentClient.tmp_executor() `\nlike the following.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "with experiment.tmp_executor(\"joblib\", n_workers=10):\n experiment.workon(main, n_workers=2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that you must specify ``n_workers`` for both the backend and for\n:meth:`ExperimentClient.workon() `.\nFor the backend, it\nrefers to the number of workers you would like to have running in parallel.\nFor :meth:`ExperimentClient.workon() `,\nit is the number of\nOrion workers to run in parallel. You may want to have more backend workers than\nOr\u00edon workers if for instance your task is also parallelizing tasks. You can\nsee such an example here, because we are parallelizing the cross-validation inside\nthe function Or\u00edon is optimizing. Each worker will create 5 tasks that can be run in parallel.\n\n\n\n## Dask\n\nUsing Dask is similar to joblib. Unless you have configured\nOr\u00edon to use Dask by default\n(`config_worker_executor` and `config_worker_executor_configuration`),\nyou will want to use\n:meth:`ExperimentClient.tmp_executor() `\nto change the backend. Make sure to only run Dask with if ``__name__ == \"__main__\"``\notherwise you will run into a RuntimeError\n(`see here `_).\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "experiment = build_experiment(\n name=\"dask_example\",\n max_trials=200,\n space=space,\n storage=storage,\n)\n\nif __name__ == \"__main__\":\n with experiment.tmp_executor(\"dask\", n_workers=10):\n experiment.workon(main, n_workers=2, joblib_backend=\"dask\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" } }, "nbformat": 4, "nbformat_minor": 0 }