add_text2video_ort_pipeline

[naomili0924]

Relates to:
huggingface/diffusers#12846
huggingface/optimum#2389

This Pull Request is adding a uniform AutoText2VideoORTPipeline as requested from: huggingface/optimum#2168

import torch
from diffusers.utils import export_to_video

from optimum.onnxruntime.modeling_diffusion import ORTPipelineForText2Video

wan_list = [
    "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
    "ali-vilab/text-to-video-ms-1.7b",
]

providers = ["CUDAExecutionProvider", "CPUExecutionProvider"]

pipe = ORTPipelineForText2Video.from_pretrained(
    wan_list[1],
    provider=providers[0],  # Force GPU
    torch_dtype=torch.float16,
)
print("Loaded successfully on:", pipe.device)
prompt = "A cat walks on the grass, realistic"
negative_prompt = "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards"

output = pipe(prompt=prompt, negative_prompt=negative_prompt, height=256, width=256, num_frames=50).frames[0]
export_to_video(output, "output.mp4", fps=15)

Result:

output.mp4

[IlyasMoutawwakil]

I don't think an abstract video onnx config is needed as it doesn't really abstract much here

[IlyasMoutawwakil]

Hi ! thanks a lot for the contribution !
is the PR finished ? I believe there also needs to be a method that describes how the Wan pipeline is split and which components it needs to export / use. Also some testing with a tiny model on the exporters and onnxruntime side would be great.

[IlyasMoutawwakil]

@naomili0924 let's rather follow the same design we did with sana, i.e. having a specific function for splitting the wan pipelines.

[naomili0924]

Also some testing with a tiny model on the exporters and onnxruntime side would be great.
@IlyasMoutawwakil
It requires a tiny-Wan model and a tiny-text-to-video model to create a test case. Do you know how to create them?