Monitoring¶

Oríon provides command line tools to help monitoring status of experiments and inspect available experiments in a database. The library API can also be used to query the database within python code.

Commands for terminal¶

`list` Overview of all experiments¶

Once you have launched a certain amount of experiments, you might start to lose track of some of them. You might forget their name and whether or not they are the children of some other experiment you have also forgotten. In any cases, the list command for Oríon will help you visualize the experiments inside your database by printing them in a easy-to-understand tree-like structure.

Configuration¶

As per usual with Oríon, if no configuration file is provided to the list command, the default configuration will be used. You can however provide a particular configuration file through the usual -c or --config argument. This configuration file needs only to contain a valid database configuration.

Basic Usage¶

The most basic usage of list is to use it without any arguments. This will simply print out every experiments inside the database in a tree-like fashion, so that the children of the experiments are easily identifiable. Here is a sample output to serve as an example:

       ┌child_2-v1
root-v1┤
       └child_1-v1┐
                  └grand_child_1-v1
other_root-v1

Here, you can see we have five experiments. Two of them are roots, which mean they do not have any parents. One of them, other_root, does not have any children and so, it does not have any branches coming out of it. On the other hand, the root experiment has multiple children, child_1 and child_2, which are printed on the same tree level, and one grand-child, grand_child_1 which branches from child_1.

The `--name` argument¶

The last example showed you how to print every experiments inside the database in a tree. However, if you wish to have an overview of the tree of a single experiment, you can add the --name argument to the call to list and only the experiment with the provided name and its children will be shown. Here’s two examples using the same set of experiments as above:

orion list --name root

Output

       ┌child_2-v1
root-v1┤
       └child_1-v1┐
                  └grand_child_1-v1

Here, the other_root experiment is not showned because it is not inside the root experiment tree.

orion list --name child_1

Output

child_1-v1┐
          └grand_child_1-v1

Here, the root and child_2 experiments are not present because they are not children of child_1.

`status` Overview of trials for experiments¶

When you reach a certain amount of trials, it becomes hard to keep track of them. This is where the status command comes into play. The status command outputs the status of the different trials inside every experiment or a specific EVC tree. It can either give you an overview of the different trials status, i.e., the number of currently completed trials and so on, or, it can give you a deeper view of the experiments by outlining every single trial, its status and its objective.

Basic Usage¶

The most basic of usages is to simply run status without any other arguments, except for a local configuration file if needed. This will then output a list of all the experiments inside your database with the count of every type of trials related to them. If an experiment has at least one completed trial associated with it, the objective value of the best one will be printed as well. Children experiments are printed below their parent and are indicated through a different tab alignment than their parent, mainly, one tab further. This continues on for grand-children, and so on and so forth. We provide an example output to illustrate this:

root-v1
=======
status       quantity    min example_objective
---------  ----------  -----------------------
completed           5                  4534.95


  child_1-v1
  ==========
  status       quantity    min example_objective
  ---------  ----------  -----------------------
  completed           5                  4547.28


other_root-v1
=============
status       quantity    min example_objective
---------  ----------  -----------------------
completed           5                  4543.73

The `--all` Argument¶

The basic status command combines statistics of all trials for each status. However, if you want to see every individual trial, with its id and its status, you can use the --all argument which will print out every single trial for each experiment with their full information. Here is a sample output using the same experiments and trials as before:

orion status --all

root-v1
=======
id                                status       min example_objective
--------------------------------  ---------  -----------------------
bc222aa1705b3fe3a266fd601598ac41  completed                  4555.13
59bf4c85305b7ba065fa770805e93cb1  completed                  4653.49
da5159d9d36ef44879e72cbe7955347c  completed                  4788.08
2c4c6409d3beeb179fce3c83b4f6d8f8  completed                  4752.45
6a82a8a55d2241d989978cf3a7ebbba0  completed                  4534.95


  child_1-v1
  ==========
  id                                status       min example_objective
  --------------------------------  ---------  -----------------------
  a3395b7192eee3ca586e93ccf4f12f59  completed                  4600.98
  e2e2e96e8e9febc33efb17b9de0920d1  completed                  4786.43
  7e0b4271f2972f539cf839fbd1b5430d  completed                  4602.58
  568acbcb2fa3e00c8607bdc2d2bda5e3  completed                  4748.09
  5f9743e88a29d0ee87b5c71246dbd2fb  completed                  4547.28


other_root-v1
=============
id                                status       min example_objective
--------------------------------  ---------  -----------------------
aaa16658770abd3516a027918eb91be5  completed                  4761.33
68233ce61ee5edfb6fb029ab7daf2db7  completed                  4543.73
cc0b0532c56c56fde63ad06fd73df63f  completed                  4753.5
b5335589cb897bbea2b58c5d4bd9c0c1  completed                  4751.15
a4a711389844567ac1b429eff96964e4  completed                  4790.87

The `--collapse` Argument¶

On the other hand, if you wish to only get an overview of the experiments and the amount of trials linked to them without looking through the whole EVC tree, you can use the --collapse option. As its name indicates, it will collapse every children into the root experiment and make a total count of the amount of trials in that EVC tree. As always, we provide an output to give you an example:

orion status --collapse

root-v1
=======
status       quantity    min example_objective
---------  ----------  -----------------------
completed          10                  4534.95


other_root-v1
=============
status       quantity    min example_objective
---------  ----------  -----------------------
completed           5                  4543.73

The `--name` Argument¶

If you wish to isolate a single EVC tree and look at their trials instead of listing every single experiments, you can use the --name argument by itself or combine it with the ones above to obtain the same results, but constrained. Once again, some examples for each type of scenrario is given:

orion status --name root

root-v1
=======
status       quantity    min example_objective
---------  ----------  -----------------------
completed          10                  4534.95


  child_1-v1
  ==========
  status       quantity    min example_objective
  ---------  ----------  -----------------------
  completed          10                  4547.28

orion status --name root --all

root-v1
=======
id                                status       min example_objective
--------------------------------  ---------  -----------------------
bc222aa1705b3fe3a266fd601598ac41  completed                  4555.13
59bf4c85305b7ba065fa770805e93cb1  completed                  4653.49
da5159d9d36ef44879e72cbe7955347c  completed                  4788.08
2c4c6409d3beeb179fce3c83b4f6d8f8  completed                  4752.45
6a82a8a55d2241d989978cf3a7ebbba0  completed                  4534.95


  child_1-v1
  ==========
  id                                status       min example_objective
  --------------------------------  ---------  -----------------------
  a3395b7192eee3ca586e93ccf4f12f59  completed                  4600.98
  e2e2e96e8e9febc33efb17b9de0920d1  completed                  4786.43
  7e0b4271f2972f539cf839fbd1b5430d  completed                  4602.58
  568acbcb2fa3e00c8607bdc2d2bda5e3  completed                  4748.09
  5f9743e88a29d0ee87b5c71246dbd2fb  completed                  4547.28

orion status --name root --collapse

root-v1
=======
status       quantity    min example_objective
---------  ----------  -----------------------
completed          10                  4534.95

`status` and the experiment tree¶

The status command handles the experiment tree in a particular fashion. Since most users will simply use the incrementing version mechanism instead of constantly renaming their experiments, the experiment tree can grow large in depth but not in breadth. This leads to a very hard to read output if the command was to print such a tree in the same way as presented above. Instead, if a root experiment does not have any children named differently, i.e. its tree only contains different version of itself, status will only print the latest version. However, if any of its children is named differently, than the whole tree will be printed just like above.

To illustrate the first case, suppose we have an experiment named test with three different versions: version 1, 2 and 3. Then running status as usual will only output version 3.

orion status --name test

test-v3
=======
empty

The `--version` argument¶

The –version argument allows you to specify a version to print instead of getting the latest one. Suppose we have the same setup as above with three experiments named test but with different versions. Then running the following command will output the second version instead of the latest.

orion status --name test --version 2

test-v2
=======
empty

It should be noted that using –version with any of –collapse or –expand-versions will lead to a RuntimeError.

The –expand-versions argument¶

As specified above, if there are no children of a root experiment with a different name then the experiment tree will not be printed in its entirety. The –expand-versions allows you to get the full output of the experiment tree, regardless if it only contains different versions. Once again, suppose we have the same setup with experiment test, then running the following command will print the experiment tree.

orion status --name test --expand-versions

test-v1
=======
empty


  test-v2
  =======
  empty


    test-v3
    =======
    empty

`info` Detailed informations about experiments¶

This command gives a detailed description of a given experiment. Here is an example of all the sections provided by the command.

orion info --name orion-tutorial

Identification
==============
name: orion-tutorial
version: 1
user: <username>

Commandline
===========
--lr~loguniform(1e-5, 1.0)


Config
======
pool size: 1
max trials: inf


Algorithm
=========
random:
    seed: None


Space
=====
/lr: reciprocal(1e-05, 1.0)


Meta-data
=========
user: <username>
datetime: 2019-07-18 18:57:57.840000
orion version: v0.1.5


Parent experiment
=================
root:
parent:
adapter:


Stats
=====
trials completed: 1
best trial:
  /lr: 0.03543491957849911
best evaluation: 0.9626
start time: 2019-07-18 18:57:57.840000
finish time: 2019-07-18 18:58:08.565000
duration: 0:00:10.725000

The last section contains information about the best trial so far, providing its hyperparameter values and the corresponding objective.

The `--name` argument¶

To get information on an experiment, you need to call info with the –name or -n argument like shown in the previous example. This will fetch the latest version of the experiment with that name inside the database and display its content.

The `--version` argument¶

To specify which version of an experiment you wish to observe, you can use the –version argument. If provided, this will fetch the experiment with a version number corresponding to that version instead of fetching the latest one. Note that the –version argument cannot be used alone and that an invalid version number, i.e. a version number greater than the latest version, will simply fetch the latest one instead.

For example, suppose we have two experiments named orion-tutorial inside the database, one with version 1 and one with version 2. Then running the following command would simply give us the latest version, so version 2.

orion info --name orion-tutorial

Whereas, running the next command will give us the first version instead:

orion info --name orion-tutorial --version 1

Library API¶

Results¶

You can fetch experiments and trials using python code. There is no need to understand the specific database backend used (such as MongoDB) since you can fetch results using the orion.core.worker.experiment.Experiment object. The class orion.core.io.experiment_builder.ExperimentBuilder provides simple methods to fetch experiments using their unique names. You do not need to explicitly open a connection to the database since it will automatically infer its configuration from the global configuration file as when calling Oríon in commandline. Otherwise you can pass other arguments to ExperimentBuilder().build_view_from() .

using the same dictionary structure as in the configuration file.

# Database automatically inferred
ExperimentBuilder().build_view_from(
    {"name": "orion-tutorial"})

# Database manually set
ExperimentBuilder().build_view_from(
    {"name": "orion-tutorial",
     "dataset": {
         "type": "mongodb",
         "name": "myother",
         "host": "localhost"}})

For a complete example, here is how you can fetch trials from a given experiment.

import datetime
import pprint

from orion.core.io.experiment_builder import ExperimentBuilder

some_datetime = datetime.datetime.now() - datetime.timedelta(minutes=5)

experiment = ExperimentBuilder().build_view_from({"name": "orion-tutorial"})

pprint.pprint(experiment.stats)

for trial in experiment.fetch_trials():
    print(trial.id)
    print(trial.status)
    print(trial.params)
    print(trial.results)
    print()
    pprint.pprint(trial.to_dict())

# Fetches only the completed trials
for trial in experiment.fetch_trials_by_status('completed'):
    print(trial.objective)

<orion.core.worker.experiment.Experiment> has many methods that allows you to query for different trials. You can find them in the code

Iterative Results with EVC¶

When using the experiment version control (described here), the experiments are connected in a tree structure which we call the EVC tree. You can retrieve results from different experiments with the EVC tree similarly as described in previous section. The only difference is we need to use EVCBuilder instead of ExperimentBuilder. The EVCBuilder will connect the experiment to the EVC tree, accessible through the node attribute. All trials of the tree can be fetched with option fetch_trials(with_evc_tree=True), with_evc_tree=False` will only fetch the trials of the specific experiment.

import pprint
from orion.core.io.evc_builder import EVCBuilder

experiment = EVCBuilder().build_view_from(
    {"name": "orion-tutorial-with-momentum"})

print(experiment.name)
pprint.pprint(experiment.stats)

parent_experiment = experiment.node.parent.item
print(parent_experiment.name)
pprint.pprint(parent_experiment.stats)

for child in experiment.node.children:
    child_experiment = child.item
    print(child_experiment.name)
    pprint.pprint(child_experiment.stats)

Monitoring¶

Commands for terminal¶

list Overview of all experiments¶

Configuration¶

Basic Usage¶

The --name argument¶

status Overview of trials for experiments¶

Basic Usage¶

The --all Argument¶

The --collapse Argument¶

The --name Argument¶

status and the experiment tree¶

The --version argument¶

The –expand-versions argument¶

info Detailed informations about experiments¶

The --name argument¶

The --version argument¶

Library API¶

Results¶

Iterative Results with EVC¶

`list` Overview of all experiments¶

The `--name` argument¶

`status` Overview of trials for experiments¶

The `--all` Argument¶

The `--collapse` Argument¶

The `--name` Argument¶

`status` and the experiment tree¶

The `--version` argument¶

`info` Detailed informations about experiments¶

The `--name` argument¶

The `--version` argument¶