Advanced Configuration

Oríon supports different levels of configuration to provide flexibility. The configuration levels are hierchical with higher levels having precedence over the lower ones. The levels are the following:

  1. Default configuration

  2. Global configuration

  3. Environment variables

  4. Experiment configuration from database*

  5. Local configuration

  6. Commandline arguments

Where larger numbers have precedence over the lower ones. We describe here further each type of configuration.

1. Default Configuration

Default values defined in core code of Oríon here.

2. Global Configuration

Defined in a yaml file in global configuration folders of Oríon. On Linux systems, this is typically at $HOME/.config/orion.core. You can get a list of these folders on your system with the following command.

$ python -c 'import orion.core; print("\n".join(orion.core.DEF_CONFIG_FILES_PATHS))'
/usr/share/awesome/orion.core/orion_config.yaml.example
/etc/xdg/xdg-awesome/orion.core/orion_config.yaml
/home/user/.config/orion.core/orion_config.yaml

This list is an example and will likely be differ on your end.

3. Environment Variables

Most options are configurable through environment variables as well. These are provided in the documented below at the Env var fields, whenever configurable otherwise the field is left empty.

4. Experiment Configuration from Database

*Experiment configuration from database is not configurable by the user per say, but it is represented here because whenever something differs between the experiment’s configuration and the levels below, the experiment’s configuration will have precedence. For instance if experiment.max_trials is set to 10 in global configuration and an experiment with max_trials set to 100 is resumed, then max_trials will be 100, not 10. However, if experiment.max_trials is set to 10 in local configuration file (the file passed with --config to the hunt command) or with commandline argument --exp-max-trials, then the experiments max_trials will be updated to 10. This make it possible to resume experiments without specifying the whole configuration, because experiment configuration from database is reused, but it also makes it possible to create a new experiment based on a previous one by simply specifying what to change.

5. Local Configuration

Defined in a yaml file that is passed to commandline.

orion [COMMAND] --config local.yaml

Not to be confused with a configuration file that may be passed to the user’s script.

orion hunt --config local.yaml ./myscript.sh --config script.yaml

Here script.yaml is the user’s script configuration (passed to ./myscript.sh), and local.yaml is the local configuration script (passed to hunt).

The configuration passed through the python API is also considered as local configuration.

6. Commandline Arguments

All arguments of orion hunt.

Full Example of Global Configuration

storage:
  database:
    host: orion_db.pkl
    type: pickleddb

experiment:
    algorithm:
        random:
            seed: None
    max_broken: 3
    max_trials: 1000000000
    worker_trials: 1000000000
    working_dir:

worker:
    n_workers: 1
    pool_size: 0
    executor: PoolExecutor
    executor_configuration: {}
    heartbeat: 120
    interrupt_signal_code: 130
    max_broken: 10
    idle_timeout: 60
    max_trials: 1000000000
    user_script_config: config

evc:
    algorithm_change: False
    auto_resolution: True
    cli_change_type: break
    code_change_type: break
    config_change_type: break
    orion_version_change: False
    ignore_code_changes: False
    manual_resolution: False
    non_monitored_arguments: []

Database

database:
    host: orion_db.pkl
    type: pickleddb

name

Type

str

Default

orion

Env var

ORION_DB_NAME

Description

Name of the database.

type

Type

str

Default

PickledDB

Env var

ORION_DB_TYPE

Description

Type of database. Builtin backends are mongodb, pickleddb and ephemeraldb.

host

Type

str

Default

“”

Env var

ORION_DB_ADDRESS

Description

URI for mongodb, or file path for pickleddb. An empty string will set the value depending on the database type (localhost or <user_data_dir>/orion/orion_db.pkl).

port

Type

int

Default

27017

Env var

ORION_DB_PORT

Description

Port address for mongodb.


Experiment

experiment:
    algorithm:
        random:
            seed: None
    max_broken: 3
    max_trials: 1000000000
    worker_trials: 1000000000
    working_dir:

name

Note

This option is only supported in local configuration.

Type

str

Default

Env var

Description

Name of the experiment.

version

Note

This option is only supported in local configuration.

Type

int

Default

None

Env var

Description

Version of the experiment. If not defined, latest experiment for the given name will be selected. Version is automatically incremented if there is any modification detected in the experiment’s configuration (search space, algorithm configuration, code version, …)

user

Note

This option is only supported in local configuration.

Type

str

Default

$USERNAME

Env var

Description

Name of the user to associate with the experiment.

max_trials

Type

int

Default

1000000000

Env var

ORION_EXP_MAX_TRIALS

Description

number of trials to be completed for the experiment. This value will be saved within the experiment configuration and reused across all workers to determine experiment’s completion.

worker_trials

Warning

DEPRECATED. This argument will be removed in v0.3. See worker: max_trials instead.

Type

int

Default

1000000000

Env var

Description

(DEPRECATED) This argument will be removed in v0.3. See worker: max_trials instead.

max_broken

Type

int

Default

3

Env var

ORION_EXP_MAX_BROKEN

Description

Maximum number of broken trials before experiment stops.

working_dir

Type

str

Default

Env var

ORION_WORKING_DIR

Description

Set working directory for running experiment.

algorithm

Type

dict

Default

random

Env var

Description

Algorithm configuration for the experiment.

strategy

Warning

DEPRECATED. This argument will be removed in v0.4. Set parallel strategy in algorithm configuration directly, if the algorithm supports it.

Type

dict

Default

MaxParallelStrategy

Env var

Description

(DEPRECATED) This argument will be removed in v0.4. Parallel strategies are now handled by algorithm directly and should be set in algorithm configuration when they support it.


Worker

worker:
    n_workers: 1
    pool_size: 0
    executor: PoolExecutor
    executor_configuration: {}
    heartbeat: 120
    interrupt_signal_code: 130
    max_broken: 10
    idle_timeout: 60
    max_trials: 1000000000
    user_script_config: config

n_workers

Type

int

Default

1

Env var

ORION_N_WORKERS

Description

Number of workers to run in parallel. It is possible to run many orion hunt in parallel, and each will spawn n_workers.

pool_size

Type

int

Default

0

Env var

Description

Number of trials to sample at a time. If 0, default to number of workers. Increase it to improve the sampling speed if workers spend too much time waiting for algorithms to sample points. An algorithm will try sampling pool_size trials but may return less.

executor

Type

str

Default

PoolExecutor

Env var

ORION_EXECUTOR

Description

The executor backend used to parallelize orion workers.

executor_configuration

Type

str

Default

{}

Description

The configuration of the executor. See orion.executor for documentation of executors configuration.

heartbeat

Type

int

Default

120

Env var

ORION_HEARTBEAT

Description

Frequency (seconds) at which the heartbeat of the trial is updated. If the heartbeat of a reserved trial is larger than twice the configured heartbeat, Oríon will reset the status of the trial to interrupted. This allows restoring lost trials (ex: due to killed worker).

max_trials

Type

int

Default

1000000000

Env var

ORION_WORKER_MAX_TRIALS

Description

Number of trials to be completed for this worker. If the experiment is completed, the worker will die even if it did not reach its maximum number of trials.

max_broken

Type

int

Default

3

Env var

ORION_WORKER_MAX_BROKEN

Description

Maximum number of broken trials before worker stops.

idle_timeout

Type

int

Default

60

Env var

ORION_IDLE_TIMEOUT

Description

Maximum time the experiment can spend trying to reserve a new suggestion. Such timeout are generally caused by slow database, large number of concurrent workers leading to many race conditions or small search spaces with integer/categorical dimensions that may be fully explored.

max_idle_time

Warning

DEPRECATED. This argument will be removed in v0.3. Use idle_timeout instead.

Type

int

Default

60

Env var

ORION_MAX_IDLE_TIME

Description

(DEPRECATED) This argument will be removed in v0.3. Use idle_timeout instead.

interrupt_signal_code

Type

int

Default

130

Env var

ORION_INTERRUPT_CODE

Description

Signal returned by user script to signal to Oríon that it was interrupted.

user_script_config

Type

str

Default

config

Env var

ORION_USER_SCRIPT_CONFIG

Description

Config argument name of user’s script (–config).


Experiment Version Control

evc:
    enable: False
    algorithm_change: False
    auto_resolution: True
    cli_change_type: break
    code_change_type: break
    config_change_type: break
    orion_version_change: False
    ignore_code_changes: False
    manual_resolution: False
    non_monitored_arguments: []

enable

Note

New in version v0.1.16. Previously the EVC was always enabled. It is now disable by default and can be enabled using this option.

Type

bool

Default

False

Env var

ORION_EVC_ENABLE

Description

Enable the Experiment Version Control. Defaults to False. When disabled, running an experiment different from an earlier one but sharing the same name will have the effect of overwriting the previous one in the database. Trials of the previous experiment will still point to the experiment but may be incoherent with the new search space.

auto_resolution

Warning

DEPRECATED. This argument will be removed in v0.3. See evc: manual_resolution to avoid auto-resolution.

Type

bool

Default

True

Env var

Description

(DEPRECATED) This argument will be removed in v0.3. Conflicts are now resolved automatically by default. See evc: manual_resolution to avoid auto-resolution.

manual_resolution

Type

bool

Default

False

Env var

ORION_EVC_MANUAL_RESOLUTION

Description

If True, enter experiment version control conflict resolver for manual resolution on branching events. Otherwise, auto-resolution is attempted.

non_monitored_arguments

Type

list

Default

[]

Env var

ORION_EVC_NON_MONITORED_ARGUMENTS

Description

Ignore these commandline arguments when looking for differences in user’s commandline call. Environment variable and commandline only supports one argument. Use global config or local config to pass a list of arguments to ignore. When defined with environment variables, use ‘:’ to delimit multiple arguments (ex: ‘three:different:arguments’).

ignore_code_changes

Type

bool

Default

False

Env var

ORION_EVC_IGNORE_CODE_CHANGES

Description

If True, ignore code changes when looking for differences.

algorithm_change

Type

bool

Default

False

Env var

ORION_EVC_ALGO_CHANGE

Description

If True, set algorithm change as resolved if a branching event occur. Child and parent experiment have access to all trials from each other when the only difference between them is the algorithm configuration.

code_change_type

Type

str

Default

break

Env var

ORION_EVC_CODE_CHANGE

Description

One of break, unsure or noeffect. Defines how trials should be filtered in Experiment Version Control tree if there is a change in the user’s code repository. If the effect of the change is unsure, the child experiment will access the trials of the parent but not the other way around. This is to ensure parent experiment does not get corrupted with possibly incompatible results. The child cannot access the trials from parent if code_change_type is break. The parent cannot access trials from child if code_change_type is unsure or break.

cli_change_type

Type

str

Default

break

Env var

ORION_EVC_CMDLINE_CHANGE

Description

One of break, unsure or noeffect. Defines how trials should be filtered in Experiment Version Control tree if there is a change in the user’s commandline call. If the effect of the change is unsure, the child experiment will access the trials of the parent but not the other way around. This is to ensure parent experiment does not get corrupted with possibly incompatible results. The child cannot access the trials from parent if cli_change_type is break. The parent cannot access trials from child if cli_change_type is unsure or break.

config_change_type

Type

str

Default

break

Env var

ORION_EVC_CONFIG_CHANGE

Description

One of break, unsure or noeffect. Defines how trials should be filtered in Experiment Version Control tree if there is a change in the user’s script. If the effect of the change is unsure, the child experiment will access the trials of the parent but not the other way around. This is to ensure parent experiment does not get corrupted with possibly incompatible results. The child cannot access the trials from parent if config_change_type is break. The parent cannot access trials from child if config_change_type is unsure or break.

orion_version_change

Type

bool

Default

False

Env var

ORION_EVC_ORION_VERSION_CHANGE

Description

If True, set orion version change as resolved if branching event occurred. Child and parent experiment have access to all trials from each other when the only difference between them is the orion version used during execution.