The Oríon Web API is a RESTful service that provides a way to retrieve and visualize information about your experiments and trials.
The API requests are handled by the Oríon itself.
The REST server is started by the command line $ orion serve
. The service is hosted through a
gunicorn container.
The database and host details are configured through the option --config <file>
.
Storage options are expressed in the configuration file in the same fashion as
for other commands. Gunicorn options are
specified under the top-level node gunicorn
. An example is included below:
storage:
database:
host: 'database.pkl'
type: 'pickleddb'
gunicorn:
bind: '127.0.0.1:8000'
workers: 4
threads: 2
Authentication¶
No authentication is necessary at the moment to use the API.
Runtime¶
The runtime resource represents the runtime information for the API server.
- GET /¶
Example response:
HTTP/1.1 200 OK Content-Type: text/javascript
{ "orion": "v0.1.7", "server": "gunicorn", "database": "pickleddb" }
- Response JSON Object
orion – The version of Oríon running the API server.
server – The WSGI HTTP Server hosting the API server.
database – The type of database where the HPO data is stored.
Experiments¶
The experiment resource permits the retrieval of in-progress and completed experiments. You can retrieve individual experiments as well as a list of all your experiments.
- GET /experiments¶
Return an unordered list of your experiments. Only the latest version of your experiments are returned.
Example response
HTTP/1.1 200 OK Content-Type: text/javascript
[ { "name":"JCZY5", "version":2 }, { "name":"UGH3", "version":1 } ]
- Response JSON Array of Objects
name – Name of the experiment.
version – Latest version of the experiment.
- GET /experiments/:name¶
Retrieve the details of the existing experiment named
name
.Example response
HTTP/1.1 200 OK Content-Type: text/javascript
{ "name": "JCZY5", "version": 2, "status": "done", "trialsCompleted": 8, "startTime": "2020-01-21T16:29:33.73701", "endTime": "2020-01-22 14:43:42.02448", "user": "your username", "orionVersion": "0.1.7", "config": { "maxTrials": 10, "algorithm": { "name": "hyperband", "seed": 42, "repetitions": 1 }, "space": { "epsilon":"~uniform(1,5)", "lr":"~uniform(0.1,1)" } }, "bestTrial": { "id": "f70277", "submitTime": "2020-01-22 14:19:42.02448", "startTime": "2020-01-22 14:20:42.02448", "endTime": "2020-01-22 14:20:42.0248", "parameters": { "epsilon": 1, "lr": 0.1 }, "objective": -0.7865584361152724, "statistics": { "low": 1, "high": 42 } } }
- Query Parameters
version – Optional version of the experiment to retrieve. If unspecified, the latest version of the experiment is retrieved.
- Response JSON Object
name – The name of the experiment.
version – The version of the experiment.
status – The status of the experiment. Can be one of ‘done’ or ‘not done’ if there is trials remaining.
trialsCompleted – The number of trials completed.
startTime – The timestamp when the experiment started.
endTime – The timestamp when the experiment finished.
user – The name of the user that registered the experiment.
orionVersion – The version of Oríon that carried out the experiment.
config – The configuration of the experiment.
config.maxTrials – The trial budget for the experiment.
config.algorithm – The algorithm settings for the experiment.
config.space – The dictionary of priors as
"prior-name":"prior-value"
.bestTrial – The result of the optimization process in the form of the best trial. See the specification of
GET /trials/:experiment/:id
.
- Status Codes
400 Bad Request – When an invalid query parameter is passed in the request.
404 Not Found – When the specified experiment doesn’t exist in the database.
- GET /experiments/status/:name¶
Retrieve the stats of the existing experiment named
name
.Example response
HTTP/1.1 200 OK Content-Type: text/javascript
{ "trials_completed": 40, "best_trials_id": "955c77e7f567c2625f48546188a6cda1", "best_evaluation": -0.788720013597263, "start_time": "2019-11-25 16:02:02.872583", "finish_time": "2019-11-27 21:13:27.043519", "max_trials": 40, "nb_trials": 40, "progress": 1, "trial_status_count": { "completed": 40 }, "elapsed_time": "2 days, 5:11:24.006755", "sum_of_trials_time": "8 days, 23:15:15.594405", "eta": "0:00:00", "eta_milliseconds": 0 }
- Query Parameters
version – Optional version of the experiment to retrieve. If unspecified, the latest version of the experiment is retrieved.
- Response JSON Object
trials_completed – The number of trials completed.
best_trial_id – The best trial ID.
best_evaluation – Best evaluation.
start_time – The timestamp when the experiment started.
finish_time – The timestamp when the experiment finished.
max_trials – The number of max trials for this experiment.
nb_trials – The current number of trials in this experiment.
progress – Floating value between 0 and 1 representing experiment progression.
trial_status_count – A dictionary mapping trial status to number of trials with this status in the experiment.
elapsed_time – The time elapsed since experiment started.
sum_of_trials_time – The sum of trials execution times.
eta – The estimation of remaining time for experiment to finish.
eta_milliseconds – The ETA in milliseconds (convenient for usages in Javascript).
- Status Codes
400 Bad Request – When an invalid query parameter is passed in the request.
404 Not Found – When the specified experiment doesn’t exist in the database.
Trials¶
The trials resource permits the retrieval of your trials regardless of their status. You can retrieve individual trials as well as a list of all your trials per experiment. You can also edit status for an individual trial.
- GET /trials/:experiment¶
Return an unordered list of the trials for the experiment
experiment
.Example response
HTTP/1.1 200 OK Content-Type: text/javascript
[ {"id": "f70277"}, {"id": "a5f7e1b"} ]
- Query Parameters
ancestors – Optionally include the trials from all the experiment’s parents. If unspecified, only the trials for this experiment version are retrieved.
status – Optionally filter the trials by their status. See the available statuses in
orion.core.worker.trial.validate_status()
.version – Optional version of the experiment to retrieve. If unspecified, the latest version of the experiment is retrieved.
- Response JSON Array of Objects
id – The ID of one trial for this experiment’s version.
- Status Codes
400 Bad Request – When an invalid query parameter is passed in the request.
404 Not Found – When the specified experiment doesn’t exist in the database.
- GET /trials/:experiment/:id¶
Return the details of an existing trial with id
id
from the experimentexperiment
.Example response
HTTP/1.1 200 OK Content-Type: text/javascript
{ "id": "f70277", "submitTime": "2020-01-22 14:19:42.02448" "startTime": "2020-01-22 14:20:42.02448", "endTime": "2020-01-22 14:20:42.0248", "parameters": { "epsilon": 1, "lr": 0.1 }, "objective": -0.7865584361152724, "statistics": { "low": 1, "high": 42 }, "status": "completed" }
- Response JSON Object
id – The ID of the trial.
submitTime – The timestamp when the trial was created
startTime – The timestamp when the trial started to be executed.
endTime – The timestamp when the trial finished its execution.
parameters – The dictionary of hyper-parameters as
"parameter-name":"parameter-value"
for this trial.objective – The objective found for this trial with the given hyper-parameters.
statistics – The dictionary of statistics recorded during the trial as
"statistic-name":"statistic-value"
.status – The status of the trial.
- Status Codes
400 Bad Request – When an invalid query parameter is passed in the request.
404 Not Found – When the specified experiment doesn’t exist in the database.
404 Not Found – When the specified trial doesn’t exist for the specified experiment.
- GET /trials/:experiment/:id/set-status/:status¶
Set status to value
status
for an existing trial with idid
from the experimentexperiment
.Return the details of trial with updated status. See the specification of
GET /trials/:experiment/:id
for a description of returned fields.Example response
HTTP/1.1 200 OK Content-Type: text/javascript
{ "id": "f70277", "submitTime": "2020-01-22 14:19:42.02448" "startTime": "2020-01-22 14:20:42.02448", "endTime": "2020-01-22 14:20:42.0248", "parameters": { "epsilon": 1, "lr": 0.1 }, "objective": -0.7865584361152724, "statistics": { "low": 1, "high": 42 }, "status": "completed" }
Plots¶
The plot resource permits the generation and retrieval of Plotly plots to visualize your experiments and their results.
- GET /plots/lpi/:experiment¶
Return a lpi plot for the specified experiment.
Example response
HTTP/1.1 200 OK Content-Type: text/javascript
The JSON output is generated automatically according to the Plotly.js schema reference.
- Status Codes
404 Not Found – When the specified experiment doesn’t exist in the database.
- GET /plots/parallel_coordinates/:experiment¶
Return a parallel coordinates plot for the specified experiment.
Example response
HTTP/1.1 200 OK Content-Type: text/javascript
The JSON output is generated automatically according to the Plotly.js schema reference.
- Status Codes
404 Not Found – When the specified experiment doesn’t exist in the database.
- GET /plots/partial_dependencies/:experiment¶
Return a partial dependency plot for the specified experiment.
Example response
HTTP/1.1 200 OK Content-Type: text/javascript
The JSON output is generated automatically according to the Plotly.js schema reference.
- Status Codes
404 Not Found – When the specified experiment doesn’t exist in the database.
- GET /plots/regret/:experiment¶
Return a regret plot for the specified experiment.
Example response
HTTP/1.1 200 OK Content-Type: text/javascript
The JSON output is generated automatically according to the Plotly.js schema reference.
- Status Codes
404 Not Found – When the specified experiment doesn’t exist in the database.
Benchmarks¶
The benchmark resource permits the retrieval of in-progress and completed benchmarks. You can retrieve individual benchmarks as well as a list of all your benchmarks.
- GET /benchmarks¶
Return an unordered list of your benchmarks.
Example response
HTTP/1.1 200 OK Content-Type: text/javascript
[ { "name": "branin_baselines", "algorithms": ["gridsearch", "random"], "assessments": {"AverageResult": {"repetitions": 2}}, "tasks": {"Branin": {"max_trials": 10}}, }, { "name": "another_benchmark", "algorithms": ["gridsearch", {"random": {"seed": 1}}], "assessments": { "AverageRank": {"repetitions": 2}, "AverageResult": {"repetitions": 2}, }, "tasks": { "Branin": {"max_trials": 10}, "CarromTable": {"max_trials": 20}, "EggHolder": {"dim": 4, "max_trials": 20}, }, }, ]
- Response JSON Array of Objects
name – Name of the benchmark.
algorithms – Algorithms of the benchmark.
assessments – Assessments used in the benchmark.
tasks – Tasks covered by the benchmark.
- GET /benchmarks/:name¶
Retrieve the details of the existing benchmark named
name
.Example response
HTTP/1.1 200 OK Content-Type: text/javascript
{ "name": "all_algos_webapi", "algorithms": ["gridsearch", {"random": {'seed': 1}}], "tasks": [{"Branin": {"max_trials": 5}}, {"RosenBrock": {"dim": 3, "max_trials": 5}}], "assessments": [{"AverageResult": {"repetitions": 3}}], "analysis": { "AverageResult": { "Branin": <plotly json encoding> "RosenBrock": <plotly json encoding> } } }
- Query Parameters
asssessment – Optional, specific assessment to analyse. All assessments will appear as part of the benchmark configuration, but the analysis dictionary will only contain the specified assessment.
task – Optional, specific task to analyse. All tasks will appear as part of the benchmark configuration, but the analysis dictionary will only contain the specified task.
algorithms – Optional, specific algorithms to include in the analyse. Multiple values may be passed. All algorithms will appear as part of the benchmark configuration, but the analysis will be executed on the specified algorithms only.
- Response JSON Array of Objects
name – Name of the benchmark.
algorithms – Algorithms of the benchmark.
assessments – Assessments used in the benchmark.
tasks – Tasks covered by the benchmark.
analysis – Dictionary of format {assessment_name: {task: <plotly json>}}
- Status Codes
400 Bad Request – When an invalid query parameter is passed in the request.
404 Not Found – When the specified benchmark does not exist in the database, or assessment, task or algorithms are not part of the existing benchmark configuration.
Database dumping¶
The database dumping resource allows to dump database content into a PickledDB and download it as PKL file.
- GET /dump¶
Return a PKL file containing database content.
- Query Parameters
name – Optional name of experiment to export. It unspecified, whole database is dumped.
version – Optional version of the experiment to retrieve. If unspecified and name is specified, the latest version of the experiment is exported. If both name and version are unspecified, whole database is dumped.
- Status Codes
404 Not Found – When an error occurred during dumping.
Database loading¶
The database loading resource allow to import data from a PKL file
- POST /load¶
Import data into database from a PKL file. This is a POST request, as a file must be uploaded. Launch an import task in a separate process in backend and return task ID which may be used to get task progress.
- Query Parameters
file – PKL file to import
resolve –
policy to resolve conflicts during import. Either:
ignore
: ignore imported data on conflictoverwrite
: overwrite ancient data on conflictbump
: bump version of imported data before insertion on conflict
name – Optional name of experiment to import. If unspecified, whole data from PKL file is imported.
version – Optional version of experiment to import. If unspecified and name is specified, the latest version of the experiment is imported. If both name and version are unspceified, whole data from PKL file is imported.
Example response
HTTP/1.1 200 OK Content-Type: text/javascript
{ "task": "e453679d-e36b-427a-a14d-58fe5e42ca19" }
- Response JSON Object
task – The ID of the running task that are importing data.
- Status Codes
400 Bad Request – When an invalid query parameter is passed in the request.
403 Forbidden – When an import task is already running.
Import progression¶
The import progression resource allows to monitor an import task launched by /load
entry.
- GET /import-status/:name¶
Returns status of a running import task identified by given
name
.name
is the task ID returned by/load
entry.Example response
HTTP/1.1 200 OK Content-Type: text/javascript
{ "messages": ["latest", "logging", "lines", "from", "import", "process"], "progress_message": "description of current import step", "progress_value": 0.889, "status": "active" }
- Response JSON Object
messages – Latest logging lines printed in import process since last call to
/import-status
entry.progress_message – Description of current import process step.
progress_value – Floating value (between 0 and 1 included) representing current import progression.
status – Import process status. Either: “active”: still running “error”: terminated with an error (see latest messages for error info) “finished”: successfully terminated
- Status Codes
400 Bad Request – When an invalid query parameter is passed in the request.
Errors¶
Oríon uses conventional HTTP response codes to indicate the success or failure of an API request. In general, 2xx result codes indicate success where 4xx indicate an error that failed given the information provided such as an unknown resource or invalid parameters. 5xx codes indicate a server side error.
200 OK |
The request succeeded |
400 Bad Request |
Missing or invalid parameter |
404 Not Found |
Resource unavailable or non-existent |
500 Server Error |
Internal server error |
- Response JSON Object
title (string) - The type of error. Can be one of
Experiment not found
,Invalid parameter
, andTrial not found
.description (string) - The human-readable description of the error.