how to load the videocrafter2 model ?

[laulampaul2]

In train.py:
if config.model.unet == 'videoCrafter2':
unet = UNet3DConditionModel.from_pretrained("/hpc2hdd/home/lwang592/ziyang/cache/videocrafterv2",subfolder='unet')

I wonder what is the list of the videocrafterv2 folder ? safetensors or ckpt model load in this setting ?

[ziyang1106]

Sorry for the late response.

You may download the model from VideoCrafter2(diffusers) and modify the corresponding path in code.

Feel free to contact me if any new problems arise.

[Fancy93]

Why do we still use the VideoCrafter2(diffusers) format? Can it support VideoCrafter2(ckpt) training?

[ziyang1106]

Why do we still use the VideoCrafter2(diffusers) format? Can it support VideoCrafter2(ckpt) training?

The diffuser format is used to quickly validate the effect of motion embedding on different pre-trained models within the existing framework. VideoCrafter2 diffusers format can be applied with a single line of code replacement. Currently, we do not have plans to develop the same algorithm in the ckpt format.

Thank you for your support.

[Fancy93]

Thank you very much, Can you provide other comparison codes? How do you conduct testing under a unified framework?