Question about perfomance and hyperparameters

[Rodyarad]

Hello, thanks for your work!

We have implemented a custom environment with discrete action spaces. We’ve observed that after reaching a certain level of performance (in terms of reward or success rate), the results begin to degrade during further training (we’re training with UniZero).

We have also encountered similar behavior with the original EfficientZero and EfficientZeroV2 repositories when running other custom environments.

Have you encountered performance degradation after a plateau? Are there any specific hyperparameters or strategies for solving it?

Thank you in advance for your response.

[puyuan1996]

Hello, thanks for your interest in LightZero and for your feedback!

We have indeed observed similar performance degradation issues with UniZero in the later stages of training (around 1M env steps). To address this, we have recently implemented several bug fixes and stability enhancements, which are detailed in #451.

Key improvements include:

1. KV Cache Fix: We fixed a critical bug that caused the kv_cache to be improperly overwritten.
2. Policy Stability: We introduced target-entropy control mechanisms and polished the stability of policy logits.

Currently, UniZero's performance on Qbert is much more stable than before (as shown in the attached figure). However, we still observe a drop around 2.6M steps, and we are actively working on further optimizations.

If you would like to test these latest fixes, you can run the configuration file located at zoo/atari/config/atari_unizero_segment_config.py on the dev-multitask-balance-clean-kvcachemanager branch associated with the PR.

Please Note: The code in this branch is primarily for developing and verifying features for the ScaleZero paper. It is currently somewhat messy and unoptimized, but we will clean up and standardize it as soon as possible.

Regarding the other algorithms you mentioned: The MuZero version in the LightZero repository currently shows stable improvement up to 8M steps.

Since our EfficientZero (EZ) implementation is based on MuZero, we expect its performance to be similarly stable. We have not tested the long-run performance of the original EfficientZero and EfficientZeroV2 repositories.

We hope this helps.