ODE Initialization?

[CzzzzH]

Thanks for the nice work!

Do you have an ETA for releasing the instructions for ODE initialization? (I am currently following the instructions from CausVid, but I am not sure if I performed it correctly due to differences in configs)

I am also curious about the recommended training iterations of ODE checkpoints. I tried the DMD training initialized from the pre-trained ODE initialization checkpoint and it converges very fast (several hours on 8 GPUs), but I am not sure how much time I need to train the initial ODE checkpoint for my own base model.

[hongruihuang]

Thanks for the nice work!

Do you have an ETA for releasing the instructions for ODE initialization? (I am currently following the instructions from CausVid, but I am not sure if I performed it correctly due to differences in configs)

I am also curious about the recommended training iterations of ODE checkpoints. I tried the DMD training initialized from the pre-trained ODE initialization checkpoint and it converges very fast (several hours on 8 GPUs), but I am not sure how much time I need to train the initial ODE checkpoint for my own base model.

Hi, I’m also following CausVid for this stage.
When I try to generate ODE pairs for the ODE initialization phase, I notice that with the following scheduler settings:

scheduler = FlowMatchScheduler(shift=5.0, sigma_min=0.0, extra_one_step=True)
scheduler.set_timesteps(num_inference_steps=50, denoising_strength=1.0)

the resulting scheduler.timesteps don‘t include some target timesteps used in self-forcing training—specifically, values like 937.5000 and 625.0000 (after warping in denoise_step_list) show some deviation.

Could you please share how you addressed this issue, or the exact scheduler configuration you used during ODE pair generation?

[CzzzzH]

@hongruihuang I am now using the Self-Forcing codebase to generate ODE pairs (see scripts/generate_ode_pairs.py), and also train the ODE initialization with this codebase

For the question you mentioned, I check the ODE config in CausVid and found that the warp_denoising_step is false, which means the timestep will be fixed to [1000, 750, 500, 250] when training the ODE initialization with the setting of Self-Forcing. But this still misaligns the timesteps used in scripts/generate_ode_pairs.py as you mentioned (I checked the results for CausVid and it is correct)

I am not sure if this will cause a huge problem because the ODE initialization I trained with this incorrect timestep setting is still useable (though my final results are suboptimal with flikerings between chunks). I think we can tune some parameters (the num_inference_steps and the selected index) to make the timesteps align better

Update:
I guess Self-Forcing use the warped timesteps beacuse if you use their scheduler setting with shift=5.0, the results will be exactly matched. It should be:

scheduler = FlowMatchScheduler(shift=5.0, sigma_min=0.0, extra_one_step=True)
scheduler.set_timesteps(num_inference_steps=48, denoising_strength=1.0)
index = [0, 12, 24, 36, -1]

[chunyu-li]

@CzzzzH If you use Self-Forcing codebase to train the ODE initialization, which config file do you use? I couldn't find the config file related to ODE initialization in the self-forcing codebase.

[CzzzzH]

@chunyu-li I wrote the config by myself. You can check the ODE trainer to see which parameters you need.

[ymingxie]

@CzzzzH Hi, thanks for sharing. Have you been able to reproduce the ODE initialization? If so, could you please share how many iterations/epochs you used for ODE training?
Thanks!

[CzzzzH]

@ymingxie For the T2V model, I successfully reproduced the ODE initialization only using 1k ODE pairs plus 3000 iterations of training. However, the I2V seems more difficult, and I cannot train a good model now...

[ymingxie]

@CzzzzH Thanks for your info!

[DaLawnChair]

@CzzzzH do you have plots for the training loss for the ODE and T2V training? I can't seem to find any anywhere online.

[TrinityNeo99]

@CzzzzH Thank you very much for your help, and sorry for the interruption. If it’s convenient for you, would you mind sharing your ODE configuration file?
Also, if I generate ODE pairs using CausVid, would it be appropriate/correct to use this data to train the ODE model in Self-Forcing?

[ymingxie]

For anyone who wants to generate ODE pairs and train the ODE model themselves, I recommend using the code in CausVid rather than the script https://github.com/guandeh17/Self-Forcing/blob/main/scripts/generate_ode_pairs.py.
Based on my experiments, that script contains a minor bug that can cause ODE training to fail (even though the code looks identical to CausVid's).

Details:
At https://github.com/guandeh17/Self-Forcing/blob/main/scripts/generate_ode_pairs.py#L61 , the variable prompt is a dictionary, not a text string. This happens because the dataloader returns a dict (see https://github.com/guandeh17/Self-Forcing/blob/main/utils/dataset.py#L28-L31).

[AdvancedHe]

Does ODE training need rolling kv cache? When I doing ODE initialization without rolling kv cache on Wan 2.2 TI2V 5B, its loss:

but when I use the trained checkpoint generate video, get this video:

ode_01.mp4

Does anyone know how to solve this problem? Thanks?

[codeysun]

@AdvancedHe I am also trying to run ODE training with Wan2.2 TI2V 5B. Do you mind sharing the config you used to train? I am currently hanging on FSDP wrapping text_encoder step with 2 H100s, despite 1.3B working.

Also, if you would like to verify 5B wrapper implementation, this is the results of causal distillation inference inference.py using the base 5B model, no training.

A.stylish.woman.strolls.down.a.bustling.Tokyo.street.the.warm.glow.of.neon.lights.and.animated.city-0.mp4

[HY-2837]

@codeysun Okay, I'll send it to you when I get my work computer.