Config does not match paper for easy anti drift

[weedwind]

Thank you so much for open sourcing the code. In the stage 1 config, you only added noise to the history, no downsampling and exposure. Is it because that the other two are not useful?

Thank you very much

[SHYuanBest]

@weedwind Thanks for interest! We made a mistake in the Stage 1 code before, which led to the parameters related to "enable downsampling or exposure" not being passed through. Unfortunately, we didn't have enough time to retrain. However, in Stage 3, we fully enabled Easy Anti-Drift and conducted a detailed ablation study, which, based on our experiments, has shown to have a better effect on mitigating long video drifting. So, in theory, Helios-Base should perform better if Easy Anti-Drift is fully enabled, which would, in turn, reduce the training difficulty for Helios-Mid and Helios-Distilled.

[splinter21]

@weedwind Thanks for interest! We made a mistake in the Stage 1 code before, which led to the parameters related to "enable downsampling or exposure" not being passed through. Unfortunately, we didn't have enough time to retrain. However, in Stage 3, we fully enabled Easy Anti-Drift and conducted a detailed ablation study, which, based on our experiments, has shown to have a better effect on mitigating long video drifting. So, in theory, Helios-Base should perform better if Easy Anti-Drift is fully enabled, which would, in turn, reduce the training difficulty for Helios-Mid and Helios-Distilled.

@SHYuanBest If I want to train a Helios model on my own dataset, do I need to consider the p_abcd and abc_min/max settings mentioned in the paper, or can I simply use the default configs for each training stage provided in the GitHub repository?

[weedwind]

Thank you for your reply. Do you have an idea which type of distortion is the most important for anti drifting?

[SHYuanBest]

I think it’s a bit hard to measure directly. In our experiments, for example, although the training was done only on short videos with 109 frames, Helios-Base and Helios-Mid can already maintain non-drifting performance on long videos up to 1449 frames using only noise corruption. However, Helios-Distilled requires additional downsampling and saturation corruption to achieve a similar effect. This was also how we initially discovered that “enable downsampling or exposure” was not being passed through.

[weedwind]

Actually in the third stage, downsampling does not seem to be enabled.

[SHYuanBest]

The code and files underwent refactoring before release, so there may be some omissions. Please refer to the new config https://github.com/PKU-YuanGroup/Helios/blob/main/scripts/training/configs/stage_3_post_gan_version.yaml

[splinter21]

The code and files underwent refactoring before release, so there may be some omissions. Please refer to the new config https://github.com/PKU-YuanGroup/Helios/blob/main/scripts/training/configs/stage_3_post_gan_version.yaml

So when training the stage3-post, should I use the GAN config?

[SHYuanBest]

I think Gan Version maybe a litter better, but it need more CPU memory since we have to offload 3 * 14B model + multi-gan-heads to CPU memory.

[splinter21]

I think Gan Version maybe a litter better, but it need more CPU memory since we have to offload 3 * 14B model + multi-gan-heads to CPU memory.

If my GPU has enough VRAM, which parameters can I modify to disable these offloads?

[SHYuanBest]

I think Gan Version maybe a litter better, but it need more CPU memory since we have to offload 3 * 14B model + multi-gan-heads to CPU memory.

If my GPU has enough VRAM, which parameters can I modify to disable these offloads?

@splinter21 https://github.com/PKU-YuanGroup/Helios/blob/main/helios/utils/train_config.py#L269-L272