Monitoring¶

You can monitor the experiments and their results using either the command line, the python API or the Web API. The command line tools help monitoring the status of experiments and inspect available experiments in a database. The library API can also be used to query the database within python code. Finally, the Web API facilitates the integration of the experiments’ results with third party tools.

Commands for terminal¶

`list` Overview of all experiments¶

Once you have launched a certain amount of experiments, you might start to lose track of some of them. You might forget their name and whether or not they are the children of some other experiment you have also forgotten. In any cases, the list command for Oríon will help you visualize the experiments inside your database by printing them in a easy-to-understand tree-like structure.

Configuration¶

As per usual with Oríon, if no configuration file is provided to the list command, the default configuration will be used. You can however provide a particular configuration file through the usual -c or --config argument. This configuration file needs only to contain a valid database configuration.

Basic Usage¶

The most basic usage of list is to use it without any arguments. This will simply print out every experiments inside the database in a tree-like fashion, so that the children of the experiments are easily identifiable. Here is a sample output to serve as an example:

       ┌child_2-v1
root-v1┤
       └child_1-v1┐
                  └grand_child_1-v1
other_root-v1

Here, you can see we have five experiments. Two of them are roots, which mean they do not have any parents. One of them, other_root, does not have any children and so, it does not have any branches coming out of it. On the other hand, the root experiment has multiple children, child_1 and child_2, which are printed on the same tree level, and one grand-child, grand_child_1 which branches from child_1.

The `--name` argument¶

The last example showed you how to print every experiments inside the database in a tree. However, if you wish to have an overview of the tree of a single experiment, you can add the --name argument to the call to list and only the experiment with the provided name and its children will be shown. Here’s two examples using the same set of experiments as above:

orion list --name root

Output

       ┌child_2-v1
root-v1┤
       └child_1-v1┐
                  └grand_child_1-v1

Here, the other_root experiment is not showned because it is not inside the root experiment tree.

orion list --name child_1

Output

child_1-v1┐
          └grand_child_1-v1

Here, the root and child_2 experiments are not present because they are not children of child_1.

`status` Overview of trials for experiments¶

When you reach a certain amount of trials, it becomes hard to keep track of them. This is where the status command comes into play. The status command outputs the status of the different trials inside every experiment or a specific EVC tree. It can either give you an overview of the different trials status, i.e., the number of currently completed trials and so on, or, it can give you a deeper view of the experiments by outlining every single trial, its status and its objective.

Basic Usage¶

The most basic of usages is to simply run status without any other arguments, except for a local configuration file if needed. This will then output a list of all the experiments inside your database with the count of every type of trials related to them. If an experiment has at least one completed trial associated with it, the objective value of the best one will be printed as well. Children experiments are printed below their parent and are indicated through a different tab alignment than their parent, mainly, one tab further. This continues on for grand-children, and so on and so forth. We provide an example output to illustrate this:

root-v1
=======
status       quantity    min example_objective
---------  ----------  -----------------------
completed           5                  4534.95


  child_1-v1
  ==========
  status       quantity    min example_objective
  ---------  ----------  -----------------------
  completed           5                  4547.28


other_root-v1
=============
status       quantity    min example_objective
---------  ----------  -----------------------
completed           5                  4543.73

The `--all` Argument¶

The basic status command combines statistics of all trials for each status. However, if you want to see every individual trial, with its id and its status, you can use the --all argument which will print out every single trial for each experiment with their full information. Here is a sample output using the same experiments and trials as before:

orion status --all

root-v1
=======
id                                status       min example_objective
--------------------------------  ---------  -----------------------
bc222aa1705b3fe3a266fd601598ac41  completed                  4555.13
59bf4c85305b7ba065fa770805e93cb1  completed                  4653.49
da5159d9d36ef44879e72cbe7955347c  completed                  4788.08
2c4c6409d3beeb179fce3c83b4f6d8f8  completed                  4752.45
6a82a8a55d2241d989978cf3a7ebbba0  completed                  4534.95


  child_1-v1
  ==========
  id                                status       min example_objective
  --------------------------------  ---------  -----------------------
  a3395b7192eee3ca586e93ccf4f12f59  completed                  4600.98
  e2e2e96e8e9febc33efb17b9de0920d1  completed                  4786.43
  7e0b4271f2972f539cf839fbd1b5430d  completed                  4602.58
  568acbcb2fa3e00c8607bdc2d2bda5e3  completed                  4748.09
  5f9743e88a29d0ee87b5c71246dbd2fb  completed                  4547.28


other_root-v1
=============
id                                status       min example_objective
--------------------------------  ---------  -----------------------
aaa16658770abd3516a027918eb91be5  completed                  4761.33
68233ce61ee5edfb6fb029ab7daf2db7  completed                  4543.73
cc0b0532c56c56fde63ad06fd73df63f  completed                  4753.5
b5335589cb897bbea2b58c5d4bd9c0c1  completed                  4751.15
a4a711389844567ac1b429eff96964e4  completed                  4790.87

The `--collapse` Argument¶

On the other hand, if you wish to only get an overview of the experiments and the amount of trials linked to them without looking through the whole EVC tree, you can use the --collapse option. As its name indicates, it will collapse every children into the root experiment and make a total count of the amount of trials in that EVC tree. As always, we provide an output to give you an example:

orion status --collapse

root-v1
=======
status       quantity    min example_objective
---------  ----------  -----------------------
completed          10                  4534.95


other_root-v1
=============
status       quantity    min example_objective
---------  ----------  -----------------------
completed           5                  4543.73

The `--name` Argument¶

If you wish to isolate a single EVC tree and look at their trials instead of listing every single experiments, you can use the --name argument by itself or combine it with the ones above to obtain the same results, but constrained. Once again, some examples for each type of scenrario is given:

orion status --name root

root-v1
=======
status       quantity    min example_objective
---------  ----------  -----------------------
completed          10                  4534.95


  child_1-v1
  ==========
  status       quantity    min example_objective
  ---------  ----------  -----------------------
  completed          10                  4547.28

orion status --name root --all

root-v1
=======
id                                status       min example_objective
--------------------------------  ---------  -----------------------
bc222aa1705b3fe3a266fd601598ac41  completed                  4555.13
59bf4c85305b7ba065fa770805e93cb1  completed                  4653.49
da5159d9d36ef44879e72cbe7955347c  completed                  4788.08
2c4c6409d3beeb179fce3c83b4f6d8f8  completed                  4752.45
6a82a8a55d2241d989978cf3a7ebbba0  completed                  4534.95


  child_1-v1
  ==========
  id                                status       min example_objective
  --------------------------------  ---------  -----------------------
  a3395b7192eee3ca586e93ccf4f12f59  completed                  4600.98
  e2e2e96e8e9febc33efb17b9de0920d1  completed                  4786.43
  7e0b4271f2972f539cf839fbd1b5430d  completed                  4602.58
  568acbcb2fa3e00c8607bdc2d2bda5e3  completed                  4748.09
  5f9743e88a29d0ee87b5c71246dbd2fb  completed                  4547.28

orion status --name root --collapse

root-v1
=======
status       quantity    min example_objective
---------  ----------  -----------------------
completed          10                  4534.95

`status` and the experiment tree¶

The status command handles the experiment tree in a particular fashion. Since most users will simply use the incrementing version mechanism instead of constantly renaming their experiments, the experiment tree can grow large in depth but not in breadth. This leads to a very hard to read output if the command was to print such a tree in the same way as presented above. Instead, if a root experiment does not have any children named differently, i.e. its tree only contains different version of itself, status will only print the latest version. However, if any of its children is named differently, than the whole tree will be printed just like above.

To illustrate the first case, suppose we have an experiment named test with three different versions: version 1, 2 and 3. Then running status as usual will only output version 3.

orion status --name test

test-v3
=======
empty

The `--version` argument¶

The --version argument allows you to specify a version to print instead of getting the latest one. Suppose we have the same setup as above with three experiments named test but with different versions. Then running the following command will output the second version instead of the latest.

orion status --name test --version 2

test-v2
=======
empty

It should be noted that using --version with any of --collapse or --expand-versions will lead to a RuntimeError.

The `--expand-versions` argument¶

As specified above, if there are no children of a root experiment with a different name then the experiment tree will not be printed in its entirety. The --expand-versions allows you to get the full output of the experiment tree, regardless if it only contains different versions. Once again, suppose we have the same setup with experiment test, then running the following command will print the experiment tree.

orion status --name test --expand-versions

test-v1
=======
empty


  test-v2
  =======
  empty


    test-v3
    =======
    empty

`info` Detailed information about experiments¶

This command gives a detailed description of a given experiment. Here is an example of all the sections provided by the command.

orion info --name orion-tutorial

Identification
==============
name: orion-tutorial
version: 1
user: <username>

Commandline
===========
python main.py --lr~loguniform(1e-5, 1.0)


Config
======
pool size: 1
max trials: inf


Algorithm
=========
random:
    seed: None


Space
=====
/lr: reciprocal(1e-05, 1.0)


Meta-data
=========
user: <username>
datetime: 2019-07-18 18:57:57.840000
orion version: v0.1.5


Parent experiment
=================
root:
parent:
adapter:


Stats
=====
trials completed: 1
best trial:
  /lr: 0.03543491957849911
best evaluation: 0.9626
start time: 2019-07-18 18:57:57.840000
finish time: 2019-07-18 18:58:08.565000
duration: 0:00:10.725000

The last section contains information about the best trial so far, providing its hyperparameter values and the corresponding objective.

The `--name` argument¶

To get information on an experiment, you need to call info with the –name or -n argument like shown in the previous example. This will fetch the latest version of the experiment with that name inside the database and display its content.

The `--version` argument¶

To specify which version of an experiment you wish to observe, you can use the –version argument. If provided, this will fetch the experiment with a version number corresponding to that version instead of fetching the latest one. Note that the –version argument cannot be used alone and that an invalid version number, i.e. a version number greater than the latest version, will simply fetch the latest one instead.

For example, suppose we have two experiments named orion-tutorial inside the database, one with version 1 and one with version 2. Then running the following command would simply give us the latest version, so version 2.

orion info --name orion-tutorial

Whereas, running the next command will give us the first version instead:

orion info --name orion-tutorial --version 1

Library API¶

Results¶

You can fetch experiments and trials using python code. There is no need to understand the specific database backend used (such as MongoDB) since you can fetch results using the orion.client.experiment.ExperimentClient object. The helper function orion.client.get_experiment() provides a simple way to fetch experiments using their unique names. You do not need to explicitly open a connection to the database since it will automatically infer its configuration from the global configuration file as when calling Oríon in commandline. Otherwise you can specify the configuration directly to get_experiment(). Take a look at the documentation for more details on all configuration arguments that are supported.

# Database automatically inferred
get_experiment(name="orion-tutorial")

# Database manually set
get_experiment(
    name="orion-tutorial",
    storage={
        'type': 'legacy',
        'database': {
            'type': 'mongodb',
            'name': 'myother',
            'host': 'localhost'}})

For a complete example, here is how you can fetch trials from a given experiment.

import pprint

from orion.client import get_experiment

experiment = get_experiment(name="orion-tutorial")

pprint.pprint(experiment.stats)

for trial in experiment.fetch_trials():
    print(trial.id)
    print(trial.status)
    print(trial.params)
    print(trial.results)
    print()
    pprint.pprint(trial.to_dict())

# Fetches only the completed trials
for trial in experiment.fetch_trials_by_status('completed'):
    print(trial.objective)

ExperimentClient has many methods that allows you to query for different trials. You can find them in the code reference section.

Iterative Results with EVC¶

When using the experiment version control (described here), the experiments are connected in a tree structure which we call the EVC tree. You can retrieve results from different experiments with the EVC tree similarly as described in previous section. All trials of the tree can be fetched with option fetch_trials(with_evc_tree=True), with_evc_tree=False` will only fetch the trials of the specific experiment.

import pprint

from orion.client import get_experiment

experiment = get_experiment(name="orion-tutorial-with-momentum")

print(experiment.name)
pprint.pprint(experiment.stats)

parent_experiment = experiment.node.parent.item
print(parent_experiment.name)
pprint.pprint(parent_experiment.stats)

for child in experiment.node.children:
    child_experiment = child.item
    print(child_experiment.name)
    pprint.pprint(child_experiment.stats)

Web API¶

The Oríon Web API is a RESTful service that provides a way to retrieve and visualize information about your experiments and trials.

The API requests are handled by the Oríon itself. The REST server is started by the command line $ orion serve. The service is hosted through a gunicorn container. The database and host details are configured through the option --config <file>. Storage options are expressed in the configuration file in the same fashion as for other commands. Gunicorn options are specified under the top-level node gunicorn. An example is included below:

storage:
   database:
      host: 'database.pkl'
      type: 'pickleddb'

gunicorn:
   bind: '127.0.0.1:8000'
   workers: 4
   threads: 2

Authentication¶

No authentication is necessary at the moment to use the API.

Runtime¶

The runtime resource represents the runtime information for the API server.

GET /¶

Example response:

HTTP/1.1 200 OK
Content-Type: text/javascript

{
  "orion": "v0.1.7",
  "server": "gunicorn",
  "database": "pickleddb"
}

Response JSON Object

orion – The version of Oríon running the API server.
server – The WSGI HTTP Server hosting the API server.
database – The type of database where the HPO data is stored.

Experiments¶

The experiment resource permits the retrieval of in-progress and completed experiments. You can retrieve individual experiments as well as a list of all your experiments.

GET /experiments¶

Return an unordered list of your experiments. Only the latest version of your experiments are returned.

Example response

HTTP/1.1 200 OK
Content-Type: text/javascript

[
  {
   "name":"JCZY5",
   "version":2
  },
  {
   "name":"UGH3",
   "version":1
  }
]

Response JSON Array of Objects

name – Name of the experiment.
version – Latest version of the experiment.

GET /experiments/:name¶

Retrieve the details of the existing experiment named name.

Example response

HTTP/1.1 200 OK
Content-Type: text/javascript

{
   "name": "JCZY5",
   "version": 2,
   "status": "done",
   "trialsCompleted": 8,
   "startTime": "2020-01-21T16:29:33.73701",
   "endTime": "2020-01-22 14:43:42.02448",
   "user": "your username",
   "orionVersion": "0.1.7",
   "config": {
      "maxTrials": 10,
      "algorithm": {
         "name": "hyperband",
         "seed": 42,
         "repetitions": 1
      },
      "space": {
         "epsilon":"~uniform(1,5)",
         "lr":"~uniform(0.1,1)"
      }
   },
   "bestTrial": {
      "id": "f70277",
      "submitTime": "2020-01-22 14:19:42.02448",
      "startTime": "2020-01-22 14:20:42.02448",
      "endTime": "2020-01-22 14:20:42.0248",
      "parameters": {
         "epsilon": 1,
         "lr": 0.1
      },
     "objective": -0.7865584361152724,
     "statistics": {
         "low": 1,
         "high": 42
     }
   }
}

Query Parameters

version – Optional version of the experiment to retrieve. If unspecified, the latest version of the experiment is retrieved.

Response JSON Object

name – The name of the experiment.
version – The version of the experiment.
status – The status of the experiment. Can be one of ‘done’ or ‘not done’ if there is trials remaining.
trialsCompleted – The number of trials completed.
startTime – The timestamp when the experiment started.
endTime – The timestamp when the experiment finished.
user – The name of the user that registered the experiment.
orionVersion – The version of Oríon that carried out the experiment.
config – The configuration of the experiment.
config.maxTrials – The trial budget for the experiment.
config.algorithm – The algorithm settings for the experiment.
config.space – The dictionary of priors as "prior-name":"prior-value".
bestTrial – The result of the optimization process in the form of the best trial. See the specification of GET /trials/:experiment/:id.

Status Codes

400 Bad Request – When an invalid query parameter is passed in the request.
404 Not Found – When the specified experiment doesn’t exist in the database.

GET /experiments/status/:name¶

Retrieve the stats of the existing experiment named name.

Example response

HTTP/1.1 200 OK
Content-Type: text/javascript

{
  "trials_completed": 40,
  "best_trials_id": "955c77e7f567c2625f48546188a6cda1",
  "best_evaluation": -0.788720013597263,
  "start_time": "2019-11-25 16:02:02.872583",
  "finish_time": "2019-11-27 21:13:27.043519",
  "max_trials": 40,
  "nb_trials": 40,
  "progress": 1,
  "trial_status_count": {
    "completed": 40
  },
  "elapsed_time": "2 days, 5:11:24.006755",
  "sum_of_trials_time": "8 days, 23:15:15.594405",
  "eta": "0:00:00",
  "eta_milliseconds": 0
}

Query Parameters

version – Optional version of the experiment to retrieve. If unspecified, the latest version of the experiment is retrieved.

Response JSON Object

trials_completed – The number of trials completed.
best_trial_id – The best trial ID.
best_evaluation – Best evaluation.
start_time – The timestamp when the experiment started.
finish_time – The timestamp when the experiment finished.
max_trials – The number of max trials for this experiment.
nb_trials – The current number of trials in this experiment.
progress – Floating value between 0 and 1 representing experiment progression.
trial_status_count – A dictionary mapping trial status to number of trials with this status in the experiment.
elapsed_time – The time elapsed since experiment started.
sum_of_trials_time – The sum of trials execution times.
eta – The estimation of remaining time for experiment to finish.
eta_milliseconds – The ETA in milliseconds (convenient for usages in Javascript).

Status Codes

400 Bad Request – When an invalid query parameter is passed in the request.
404 Not Found – When the specified experiment doesn’t exist in the database.

Trials¶

The trials resource permits the retrieval of your trials regardless of their status. You can retrieve individual trials as well as a list of all your trials per experiment.

GET /trials/:experiment¶

Return an unordered list of the trials for the experiment experiment.

Example response

HTTP/1.1 200 OK
Content-Type: text/javascript

[
   {"id": "f70277"},
   {"id": "a5f7e1b"}
]

Query Parameters

ancestors – Optionally include the trials from all the experiment’s parents. If unspecified, only the trials for this experiment version are retrieved.
status – Optionally filter the trials by their status. See the available statuses in orion.core.worker.trial.validate_status().
version – Optional version of the experiment to retrieve. If unspecified, the latest version of the experiment is retrieved.

Response JSON Array of Objects

id – The ID of one trial for this experiment’s version.

Status Codes

400 Bad Request – When an invalid query parameter is passed in the request.
404 Not Found – When the specified experiment doesn’t exist in the database.

GET /trials/:experiment/:id¶

Return the details of an existing trial with id id from the experiment experiment.

Example response

HTTP/1.1 200 OK
Content-Type: text/javascript

{
   "id": "f70277",
   "submitTime": "2020-01-22 14:19:42.02448"
   "startTime": "2020-01-22 14:20:42.02448",
   "endTime": "2020-01-22 14:20:42.0248",
   "parameters": {
      "epsilon": 1,
      "lr": 0.1
   },
   "objective": -0.7865584361152724,
   "statistics": {
      "low": 1,
      "high": 42
   }
}

Response JSON Object

id – The ID of the trial.
submitTime – The timestamp when the trial was created
startTime – The timestamp when the trial started to be executed.
endTime – The timestamp when the trial finished its execution.
parameters – The dictionary of hyper-parameters as "parameter-name":"parameter-value" for this trial.
objective – The objective found for this trial with the given hyper-parameters.
statistics – The dictionary of statistics recorded during the trial as "statistic-name":"statistic-value".

Status Codes

400 Bad Request – When an invalid query parameter is passed in the request.
404 Not Found – When the specified experiment doesn’t exist in the database.
404 Not Found – When the specified trial doesn’t exist for the specified experiment.

Plots¶

The plot resource permits the generation and retrieval of Plotly plots to visualize your experiments and their results.

GET /plots/lpi/:experiment¶

Return a lpi plot for the specified experiment.

Example response

HTTP/1.1 200 OK
Content-Type: text/javascript

The JSON output is generated automatically according to the Plotly.js schema reference.

Status Codes

404 Not Found – When the specified experiment doesn’t exist in the database.

GET /plots/parallel_coordinates/:experiment¶

Return a parallel coordinates plot for the specified experiment.

Example response

HTTP/1.1 200 OK
Content-Type: text/javascript

The JSON output is generated automatically according to the Plotly.js schema reference.

Status Codes

404 Not Found – When the specified experiment doesn’t exist in the database.

GET /plots/partial_dependencies/:experiment¶

Return a partial dependency plot for the specified experiment.

Example response

HTTP/1.1 200 OK
Content-Type: text/javascript

The JSON output is generated automatically according to the Plotly.js schema reference.

Status Codes

404 Not Found – When the specified experiment doesn’t exist in the database.

GET /plots/regret/:experiment¶

Return a regret plot for the specified experiment.

Example response

HTTP/1.1 200 OK
Content-Type: text/javascript

The JSON output is generated automatically according to the Plotly.js schema reference.

Status Codes

404 Not Found – When the specified experiment doesn’t exist in the database.

Benchmarks¶

The benchmark resource permits the retrieval of in-progress and completed benchmarks. You can retrieve individual benchmarks as well as a list of all your benchmarks.

GET /benchmarks¶

Return an unordered list of your benchmarks.

Example response

HTTP/1.1 200 OK
Content-Type: text/javascript

[
  {
    "name": "branin_baselines",
    "algorithms": ["gridsearch", "random"],
    "assessments": {"AverageResult": {"repetitions": 2}},
    "tasks": {"Branin": {"max_trials": 10}},
  },
  {
    "name": "another_benchmark",
    "algorithms": ["gridsearch", {"random": {"seed": 1}}],
    "assessments": {
      "AverageRank": {"repetitions": 2},
      "AverageResult": {"repetitions": 2},
    },
    "tasks": {
      "Branin": {"max_trials": 10},
      "CarromTable": {"max_trials": 20},
      "EggHolder": {"dim": 4, "max_trials": 20},
    },
  },
]

Response JSON Array of Objects

name – Name of the benchmark.
algorithms – Algorithms of the benchmark.
assessments – Assessments used in the benchmark.
tasks – Tasks covered by the benchmark.

GET /benchmarks/:name¶

Retrieve the details of the existing benchmark named name.

Example response

HTTP/1.1 200 OK
Content-Type: text/javascript

{
  "name": "all_algos_webapi",
  "algorithms": ["gridsearch", {"random": {'seed': 1}}],
  "tasks": [{"Branin": {"max_trials": 5}}, {"RosenBrock": {"dim": 3, "max_trials": 5}}],
  "assessments": [{"AverageResult": {"repetitions": 3}}],
  "analysis": {
    "AverageResult": {
      "Branin": <plotly json encoding>
      "RosenBrock": <plotly json encoding>
    }
  }
}

Query Parameters

asssessment – Optional, specific assessment to analyse. All assessments will appear as part of the benchmark configuration, but the analysis dictionary will only contain the specified assessment.
task – Optional, specific task to analyse. All tasks will appear as part of the benchmark configuration, but the analysis dictionary will only contain the specified task.
algorithms – Optional, specific algorithms to include in the analyse. Multiple values may be passed. All algorithms will appear as part of the benchmark configuration, but the analysis will be executed on the specified algorithms only.

Response JSON Array of Objects

name – Name of the benchmark.
algorithms – Algorithms of the benchmark.
assessments – Assessments used in the benchmark.
tasks – Tasks covered by the benchmark.
analysis – Dictionary of format {assessment_name: {task: <plotly json>}}

Status Codes

400 Bad Request – When an invalid query parameter is passed in the request.
404 Not Found – When the specified benchmark does not exist in the database, or assessment, task or algorithms are not part of the existing benchmark configuration.

Errors¶

Oríon uses conventional HTTP response codes to indicate the success or failure of an API request. In general, 2xx result codes indicate success where 4xx indicate an error that failed given the information provided such as an unknown resource or invalid parameters. 5xx codes indicate a server side error.

HTTP Codes Summary¶
200 OK	The request succeeded
400 Bad Request	Missing or invalid parameter
404 Not Found	Resource unavailable or non-existent
500 Server Error	Internal server error

Response JSON Object

title (string) - The type of error. Can be one of Experiment not found, Invalid parameter, and Trial not found.
description (string) - The human-readable description of the error.

Monitoring¶

Commands for terminal¶

list Overview of all experiments¶

Configuration¶

Basic Usage¶

The --name argument¶

status Overview of trials for experiments¶

Basic Usage¶

The --all Argument¶

The --collapse Argument¶

The --name Argument¶

status and the experiment tree¶

The --version argument¶

The --expand-versions argument¶

info Detailed information about experiments¶

The --name argument¶

The --version argument¶

Library API¶

Results¶

Iterative Results with EVC¶

Web API¶

Authentication¶

Runtime¶

Experiments¶

Trials¶

Plots¶

Benchmarks¶

Errors¶

`list` Overview of all experiments¶

The `--name` argument¶

`status` Overview of trials for experiments¶

The `--all` Argument¶

The `--collapse` Argument¶

The `--name` Argument¶

`status` and the experiment tree¶

The `--version` argument¶

The `--expand-versions` argument¶

`info` Detailed information about experiments¶

The `--name` argument¶

The `--version` argument¶