Skip to content

Fix nested hyperparameters clobbering user config during optimization - #528

Closed
musaaero wants to merge 4 commits into
DLR-RM:masterfrom
musaaero:fix-deep-update-431
Closed

musaaero wants to merge 4 commits into
DLR-RM:masterfrom
musaaero:fix-deep-update-431

Conversation

@musaaero

@musaaero musaaero commented Oct 4, 2026

Copy link
Copy Markdown

Description

Replaces the shallow kwargs.update(sampled_hyperparams) in the Optuna objective()
with a recursive deep_update() that merges nested dicts key-by-key instead of
replacing them wholesale. Implemented as a local helper (the issue suggested pydantic's
deep_update; a local function avoids adding a dependency for one utility).
The helper deep-copies the base dict, which also fixes a latent leak: the previous
self._hyperparams.copy() was shallow, so nested dicts were shared across trials.

Motivation and Context

Fixes #431. The hyperparameter samplers return nested dicts (e.g.
{"policy_kwargs": {"net_arch": ..., "activation_fn": ...}}). The old dict.update()
replaced the entire user-specified policy_kwargs, silently dropping keys like
features_extractor_class / features_extractor_kwargs required for custom policies
(the reporter observed NaN actions as a result).

Note: read_hyperparameters() applies study hyperparameters from load_trial()
(--trial-id) via the same shallow update() pattern and has the same issue — happy
to send a follow-up PR for that path if this approach is approved.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • I've read the CONTRIBUTION guide
  • I have updated the changelog accordingly
  • I have updated the tests accordingly
  • I have checked the codestyle using make check-codestyle and make lint (CI)
  • I have ensured make pytest and make type both pass (CI)

AI assistance disclosure: I used AI assistance to draft parts of this change.
I have reviewed and understood every modified line and can explain the reasoning.

@musaaero musaaero changed the title Fix deep update 431 Fix nested hyperparameters clobbering user config during optimization Oct 4, 2026
@araffin araffin added the LLM generated We do not accept LLM generated issues/PR, please tell your human label Oct 4, 2026
@musaaero musaaero closed this Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

LLM generated We do not accept LLM generated issues/PR, please tell your human

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Custom Sub-Hyperparameters during train.py -> Optimize

2 participants