Add SkyReels V2: Infinite-Length Film Generative Model

[tolgacangoz]

Thanks for the opportunity to fix #11374!

Original repo: https://github.com/SkyworkAI/SkyReels-V2

TODOs:
✅ FlowMatchUniPCMultistepScheduler: Just copy-pasted (for now).
⏳ SkyReelsV2DiffusionForcingPipeline
⏳ SkyReelsV2DiffusionForcingImageToVideoPipeline: Includes Start/End Frame Control.
⏳ SkyReelsV2DiffusionForcingVideoToVideoPipeline: Extends a given video.
⬜ SkyReelsV2Pipeline
⬜ SkyReelsV2ImageToVideoPipeline
⬜ Did you make sure to update the documentation with your changes?
⬜ Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

[ukaprch]

It's about time. Thanks.

[ukaprch]

FWIW, I have been successful in using the same T5 encoder for WAN 2.1 for this model just by fiddling with their pipeline:

        print('Quantize text_encoder qint8')
        class  QuantizedT5EncoderModelForCausalLM (QuantizedTransformersModel):
            auto_class = UMT5EncoderModel
            auto_class.from_config = auto_class._from_config
        text_encoder = QuantizedT5EncoderModelForCausalLM.from_pretrained(
            "./wan quantro T2V-14B-720P Diffusers/basemodel/t5encodermodel_qint8"
        ).to(dtype=dtype)
        
        pipe = Text2VideoPipeline(
            model_path=model_path,
            transformer=transformer,
            text_encoder=text_encoder,
            tokenizer=tokenizer,
            weight_dtype=dtype)

Then this: I incorporate my bitsandbytes nf4 transformer, their tokenizer and the WAN based T5 encoder:

def __init__(
    self, model_path, transformer, text_encoder, tokenizer, device: str = "cuda", weight_dtype=torch.bfloat16, use_usp=False, offload=False,
):
    self.transformer = transformer          #get_transformer(model_path, 'cpu', weight_dtype)
    vae_model_path = os.path.join(model_path, "Wan2.1_VAE.pth")
    self.vae = get_vae(vae_model_path, 'cpu', weight_dtype=torch.float32)
    if text_encoder is not None:
        self.text_encoder = text_encoder        #get_text_encoder(model_path, 'cuda', weight_dtype)
    if tokenizer is not None: 
        self.tokenizer = tokenizer

I need to add this function to the pipeline for the T5 encoder to work:

def encode(self, texts):
    ids, mask = self.tokenizer(texts, return_mask=True, add_special_tokens=True)
    ids = ids.to(self.device)
    mask = mask.to(self.device)
    context = self.text_encoder(ids, mask)
    #seq_lens = mask.gt(0).sum(dim=1).long()
    context = context.last_hidden_state * mask.unsqueeze(-1)
    return context

[tolgacangoz]

It seems appropriate to me. Only Diffusion Forcing pipelines are different for large models. How are the results with your setting?