Multirun ignoring some overrides

[Entodi]

Here's an example of a script I run:

python main.py script.dataset.object.fold=${FOLD} --multirun

where FOLD is a bash variable. When it runs in a multi-run mode the sweep does not override the default value that was specified by the script.dataset.object.fold=${FOLD}. So, if my default fold was 0, but FOLD=4, it will just use the default value in the sweep.

[2023-06-07 00:38:48,645][HYDRA] 	#0 : wd=2.2e-06 lr=0.0019 epoch=5 bs=8

To make a small fix, I added a fold parameter in the config file and pointed it in script.dataset.object.fold=${fold}.

I'm not sure if this is the expected behavior to override only the first-order parameters. Because it does override script/dataset=dataset2, for example. Could you clarify this?

Thank you,
Alex

[Delaunay]

Can you give me your hydra configuration ?
Is Fold an hyperparameter ?

[Entodi]

The fold is not a hyperparameter, but I want to set it using the SLURM_ARRAY_TASK.

Here is the modified yaml:

defaults:
  - dataset: cifar10
  - override hydra/sweeper: orion

hydra:
  sweep:
    dir: multirun/${now:%Y-%m-%d}/${now:%H-%M-%S}
    subdir: ${hydra.sweeper.experiment.name}/${hydra.sweeper.experiment.uuid}/${hydra.job.id}

  sweeper:
    params:
      a: "uniform(0, 1)"
      b: "uniform(0, 1)"

    experiment:
      name: 'experiment'
      version: '1'

    algorithm:
      type: random
      config:
        seed: 1

    worker:
      n_workers: -1
      max_broken: 3
      max_trials: 100

    storage:
      type: legacy
      database:
        type: pickleddb
        host: 'database.pkl'

# Default values
a: 0
b: 0

The cifar10.yaml in the directory dataset:

object:
  fold: 0

If I run:

HYDRA_FULL_ERROR=1 python main.py dataset=cifar10 dataset.object.fold=3 --multirun

I get:

<my_dir>/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
[2023-06-07 13:59:37,540][HYDRA] Orion Optimizer {'type': 'random', 'config': {'seed': 1}}
[2023-06-07 13:59:37,541][HYDRA] with parametrization {'a': 'uniform(0, 1)', 'b': 'uniform(0, 1)'}
[2023-06-07 13:59:40,153][HYDRA] algorithms is deprecated and will be removed in v0.4.0. Use algorithm instead.
 <my_dir>/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning:In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
[2023-06-07 14:00:02,842][HYDRA] Launching 128 jobs locally
{'dataset': {'object': {'fold': 0}}, 'a': 0.2878, 'b': 0.8581}
[2023-06-07 14:06:53,268][HYDRA]        #1 : dataset=cifar10 b=0.72 a=0.4934
{'dataset': {'object': {'fold': 0}}, 'a': 0.4934, 'b': 0.72}
[2023-06-07 14:06:53,392][HYDRA]        #2 : dataset=cifar10 b=0.5606 a=0.9892
{'dataset': {'object': {'fold': 0}}, 'a': 0.9892, 'b': 0.5606}
[2023-06-07 14:06:53,610][HYDRA]        #3 : dataset=cifar10 b=0.9306 a=0.06306

If I run:

HYDRA_FULL_ERROR=1 python main.py dataset.object.fold=3 --multirun

I get the error:

<my_dir>/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
[2023-06-07 14:00:40,087][HYDRA] Orion Optimizer {'type': 'random', 'config': {'seed': 1}}
[2023-06-07 14:00:40,088][HYDRA] with parametrization {'a': 'uniform(0, 1)', 'b': 'uniform(0, 1)'}
[2023-06-07 14:00:42,677][HYDRA] algorithms is deprecated and will be removed in v0.4.0. Use algorithm instead.
Traceback (most recent call last):
  File "main.py", line 12, in <module>
    main()
  File "<my_dir>/lib/python3.8/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 465, in _run_app
    run_and_report(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/utils.py", line 466, in <lambda>
    lambda: hydra.multirun(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 162, in multirun
    ret = sweeper.sweep(arguments=task_overrides)
  File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/orion_sweeper.py", line 79, in sweep
    return self.sweeper.sweep(arguments)
  File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 500, in sweep
    raise e
  File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 497, in sweep
    self.optimize(self.client)
  File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 517, in optimize
    returns = self.execute_trials(trials)
  File "<my_dir>/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 557, in execute_trials
    self.validate_batch_is_legal(overrides)
  File "<my_dir>/lib/python3.8/site-packages/hydra/plugins/sweeper.py", line 63, in validate_batch_is_legal
    config_loader.load_sweep_config(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 319, in load_sweep_config
    sweep_config = self.load_configuration(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
    return self._load_configuration_impl(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 244, in _load_configuration_impl
    parsed_overrides, caching_repo = self._parse_overrides_and_create_caching_repo(
  File "<my_dir>/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 228, in _parse_overrides_and_create_caching_repo
    parsed_overrides = parser.parse_overrides(overrides=overrides)
  File "<my_dir>/lib/python3.8/site-packages/hydra/core/override_parser/overrides_parser.py", line 96, in parse_overrides
    raise OverrideParseException(
hydra.errors.OverrideParseException: no viable alternative at input '{'object''
See https://hydra.cc/docs/1.2/advanced/override_grammar/basic for details

My versions of hydra and orion packages:

(my_env) <my_dir>/orion_test$ pip show hydra-core
Name: hydra-core
Version: 1.3.2
Summary: A framework for elegantly configuring complex applications
Home-page: https://github.com/facebookresearch/hydra
Author: Omry Yadan
Author-email: omry@fb.com
License: MIT
Location: <my_dir>/anaconda3/envs/my_env/lib/python3.8/site-packages
Requires: antlr4-python3-runtime, importlib-resources, omegaconf, packaging
Required-by: hydra-orion-sweeper
(my_env) <my_dir>/orion_test$ pip show hydra-orion-sweeper
Name: hydra-orion-sweeper
Version: 1.6.2
Summary: Hydra Orion Sweeper plugin
Home-page: https://orion.readthedocs.io/
Author: Pierre Delaunay
Author-email: pierre.delaunay@mila.quebec
License: 
Location: <my_dir>/anaconda3/envs/my_env/lib/python3.8/site-packages
Requires: hydra-core, omegaconf, orion, typing-extensions
Required-by: 

[Delaunay]

Can you try with the latest version ?
The issue should be resolved, feel free to reopen this if this is not the case

[Entodi]

It worked for the command ✅:

HYDRA_FULL_ERROR=1 python main.py dataset=cifar10 dataset.object.fold=3 --multirun

But it does not work if ❌:

HYDRA_FULL_ERROR=1 python main.py dataset.object.fold=3 --multirun

The error:

(orion) <node>:<my_dir>/orion_test$ HYDRA_FULL_ERROR=1 python main.py dataset.object.fold=3 --multirun
<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
[2023-06-07 15:17:18,714][HYDRA] Orion Optimizer {'type': 'random', 'config': {'seed': 1}}
[2023-06-07 15:17:18,714][HYDRA] with parametrization {'a': 'uniform(0, 1)', 'b': 'uniform(0, 1)'}
[2023-06-07 15:17:21,178][HYDRA] algorithms is deprecated and will be removed in v0.4.0. Use algorithm instead.
Traceback (most recent call last):
  File "main.py", line 13, in <module>
    main()
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 465, in _run_app
    run_and_report(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/utils.py", line 466, in <lambda>
    lambda: hydra.multirun(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 162, in multirun
    ret = sweeper.sweep(arguments=task_overrides)
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/orion_sweeper.py", line 79, in sweep
    return self.sweeper.sweep(arguments)
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 510, in sweep
    raise e
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 507, in sweep
    self.optimize(self.client)
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 527, in optimize
    returns = self.execute_trials(trials)
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra_plugins/hydra_orion_sweeper/implementation.py", line 567, in execute_trials
    self.validate_batch_is_legal(overrides)
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/plugins/sweeper.py", line 63, in validate_batch_is_legal
    config_loader.load_sweep_config(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 319, in load_sweep_config
    sweep_config = self.load_configuration(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
    return self._load_configuration_impl(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 244, in _load_configuration_impl
    parsed_overrides, caching_repo = self._parse_overrides_and_create_caching_repo(
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 228, in _parse_overrides_and_create_caching_repo
    parsed_overrides = parser.parse_overrides(overrides=overrides)
  File "<my_dir>/anaconda3/envs/orion/lib/python3.8/site-packages/hydra/core/override_parser/overrides_parser.py", line 96, in parse_overrides
    raise OverrideParseException(
hydra.errors.OverrideParseException: no viable alternative at input '{'object''
See https://hydra.cc/docs/1.2/advanced/override_grammar/basic for details