Skip to content

Add SkyReels V2: Infinite-Length Film Generative Model #11518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 84 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
4dd739f
Add SkyReels-V2 pipelines for text-to-video, image-to-video, and diff…
tolgacangoz May 7, 2025
899f41c
Merge branch 'main' into skyreels-v2
tolgacangoz May 7, 2025
607b5ba
up
tolgacangoz May 7, 2025
3ccf201
second draft
tolgacangoz May 8, 2025
959ca1f
Merge branch 'main' into skyreels-v2
tolgacangoz May 8, 2025
37ca14f
up
tolgacangoz May 8, 2025
d80b505
Merge branch 'main' into skyreels-v2
tolgacangoz May 8, 2025
95d0621
3rd draft
tolgacangoz May 8, 2025
6f8a945
4th draft
tolgacangoz May 9, 2025
e781084
upup
tolgacangoz May 9, 2025
4806660
style
tolgacangoz May 9, 2025
0986e81
up
tolgacangoz May 9, 2025
6a300f5
up
tolgacangoz May 9, 2025
45e1680
fix fn name
tolgacangoz May 9, 2025
c8a0c14
update import structure for SkyReelsV2
tolgacangoz May 9, 2025
47306b6
add SkyreelsV2 pipeline classes with backend requirements
tolgacangoz May 9, 2025
c5b8da9
up
tolgacangoz May 10, 2025
5835eaa
up
tolgacangoz May 10, 2025
9d2880e
add draft transformer_skyreels_v2.py with a custom WanModel and atten…
tolgacangoz May 10, 2025
2c0586e
up
tolgacangoz May 10, 2025
52590ea
split i2v and t2v pipes for diffusion forcing
tolgacangoz May 10, 2025
c50fcad
Refactors SkyReelsV2 attention and normalizations
tolgacangoz May 10, 2025
d0c71fd
Add SkyReelsV2 image and time-text embeddings
tolgacangoz May 11, 2025
f318efa
up
tolgacangoz May 11, 2025
9688a82
Refactors the `SkyReelsV2Transformer3DModel` by removing unused metho…
tolgacangoz May 11, 2025
825c2c1
Refactors `SkyReelsV2TransformerBlock` to integrate its `forward()` m…
tolgacangoz May 12, 2025
d848500
Refactors `SkyReelsV2AttnProcessor2_0` to enhance the `forward()` met…
tolgacangoz May 12, 2025
2f5a4e2
Refactors `SkyReelsV2Transformer3DModel` to enhance the `forward()` m…
tolgacangoz May 12, 2025
e5870dd
Refactors `SkyReelsV2Transformer3DModel` to improve the `forward()` m…
tolgacangoz May 12, 2025
d54e3e1
Refactors `SkyReelsV2Transformer3DModel` forward pass
tolgacangoz May 13, 2025
10d7480
Add DF inference template.
tolgacangoz May 13, 2025
fc68bf3
style
tolgacangoz May 13, 2025
1cb6a9e
Refactor `SkyReelsV2DiffusionForcingPipeline` to remove image process…
tolgacangoz May 13, 2025
ded93bc
Enhance `SkyReelsV2DiffusionForcingImageToVideoPipeline` by refining …
tolgacangoz May 13, 2025
c9483b2
Remove unused dtype handling in `SkyReelsV2DiffusionForcingPipeline` …
tolgacangoz May 13, 2025
f7fed01
up
tolgacangoz May 13, 2025
0e7b21d
up
tolgacangoz May 13, 2025
b3698d7
Update references
tolgacangoz May 13, 2025
7e0f0f5
Add `generate_timestep_matrix` method to `SkyReelsV2DiffusionForcingP…
tolgacangoz May 14, 2025
47080c2
Merge branch 'main' into skyreels-v2
tolgacangoz May 14, 2025
8c23208
Remove training-related code
tolgacangoz May 14, 2025
1f8e268
Add gradient checkpointing support in `SkyReelsV2Transformer3DModel` …
tolgacangoz May 14, 2025
d853521
Refactor `SkyReelsV2TransformerBlock` and remove unused `Head` class.…
tolgacangoz May 14, 2025
2b79584
Remove unused parameter `y` and associated documentation from `SkyRee…
tolgacangoz May 14, 2025
600ced3
Update context length calculation in `SkyReelsV2AttnProcessor2_0` to …
tolgacangoz May 14, 2025
586fe56
Fix comparison logic in `SkyReelsV2AttnProcessor2_0` to correctly det…
tolgacangoz May 14, 2025
afcaf6e
Remove unused `flex_attention` variable from `transformer_skyreels_v2…
tolgacangoz May 14, 2025
465df8c
Updates SkyReelsV2 pipeline defaults and docs
tolgacangoz May 15, 2025
cad2d38
Remove `enable_teacache` functionality from `SkyReelsV2Transformer3DM…
tolgacangoz May 15, 2025
1fcdf98
Refactor `SkyReelsV2Transformer3DModel` to use configuration paramete…
tolgacangoz May 15, 2025
6d57725
Remove unused import of `numpy` and clean up whitespace in `transform…
tolgacangoz May 15, 2025
c4cec04
Refactor `SkyReelsV2DiffusionForcingPipeline` to improve error handli…
tolgacangoz May 15, 2025
6a85ba1
Refactor `SkyReelsV2DiffusionForcingPipeline` to enhance sample sched…
tolgacangoz May 16, 2025
76af29b
update template for df_i2v
tolgacangoz May 16, 2025
81206ce
style
tolgacangoz May 16, 2025
906b6f5
Refactor `SkyReelsV2DiffusionForcingPipeline` to improve the handling…
tolgacangoz May 16, 2025
e2391b6
Add newly released `SkyReelsV2DiffusionForcingVideoToVideoPipeline` t…
tolgacangoz May 16, 2025
245534f
up df_i2v
tolgacangoz May 16, 2025
aaa8a8b
Refactor `SkyReelsV2DiffusionForcingPipeline` to improve the handling…
tolgacangoz May 16, 2025
ca3f7bd
Integrate video decoding in pipeline
tolgacangoz May 18, 2025
b4e26fd
up
tolgacangoz May 18, 2025
c3bcd1d
Fix variable name typo in `SkyReelsV2DiffusionForcingPipeline` from `…
tolgacangoz May 18, 2025
c9bea14
Fix variable name from `casual_block_size` to `causal_block_size` for…
tolgacangoz May 18, 2025
00fdeb0
Update `_no_split_modules` in `SkyReelsV2Transformer3DModel` and adju…
tolgacangoz May 18, 2025
cf91fb4
Refactor type hint for `device` parameter in `_prepare_blockwise_caus…
tolgacangoz May 18, 2025
256fa6d
Refactor `SkyReelsV2DiffusionForcingPipeline` to streamline the setti…
tolgacangoz May 19, 2025
a74252c
Add `flag_df` parameter to `SkyReelsV2Transformer3DModel` for improve…
tolgacangoz May 19, 2025
771fb05
Refactor `SkyReelsV2DiffusionForcingPipeline` to enhance clarity and …
tolgacangoz May 19, 2025
8e61893
Merge branch 'main' into skyreels-v2
tolgacangoz May 19, 2025
bccad55
Add script for converting SkyReelsV2 models to Diffusers format
tolgacangoz May 20, 2025
59c1e88
down
tolgacangoz May 20, 2025
02f038d
Update documentation in `SkyReelsV2DiffusionForcingPipeline` to clari…
tolgacangoz May 20, 2025
32ca01a
up
tolgacangoz May 20, 2025
02ffe0c
Refactor model directory path handling in `convert_transformer` funct…
tolgacangoz May 20, 2025
a215677
fix "inject_sample_info": true,
tolgacangoz May 20, 2025
1e4c501
temp fix
tolgacangoz May 20, 2025
322ce0c
up
tolgacangoz May 20, 2025
b7d54d6
fix `qk_norm`
tolgacangoz May 20, 2025
be77ad8
Refactor `convert_skyreelsv2_to_diffusers.py` to use `SkyreelsV2Image…
tolgacangoz May 20, 2025
6f8ffb2
for vae
tolgacangoz May 20, 2025
4576f6e
for t5
tolgacangoz May 20, 2025
10174ca
up
tolgacangoz May 20, 2025
9223f2d
temp fix
tolgacangoz May 20, 2025
a1aadd3
up
tolgacangoz May 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
467 changes: 467 additions & 0 deletions scripts/convert_skyreelsv2_to_diffusers.py

Large diffs are not rendered by default.

12 changes: 12 additions & 0 deletions src/diffusers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@
"SD3ControlNetModel",
"SD3MultiControlNetModel",
"SD3Transformer2DModel",
"SkyReelsV2Transformer3DModel",
"SparseControlNetModel",
"StableAudioDiTModel",
"StableCascadeUNet",
Expand Down Expand Up @@ -272,6 +273,7 @@
"FlowMatchEulerDiscreteScheduler",
"FlowMatchHeunDiscreteScheduler",
"FlowMatchLCMScheduler",
"FlowMatchUniPCMultistepScheduler",
"HeunDiscreteScheduler",
"IPNDMScheduler",
"KarrasVeScheduler",
Expand Down Expand Up @@ -445,6 +447,10 @@
"SemanticStableDiffusionPipeline",
"ShapEImg2ImgPipeline",
"ShapEPipeline",
"SkyreelsV2DiffusionForcingImageToVideoPipeline",
"SkyreelsV2DiffusionForcingPipeline",
"SkyreelsV2ImageToVideoPipeline",
"SkyreelsV2Pipeline",
"StableAudioPipeline",
"StableAudioProjectionModel",
"StableCascadeCombinedPipeline",
Expand Down Expand Up @@ -803,6 +809,7 @@
SD3ControlNetModel,
SD3MultiControlNetModel,
SD3Transformer2DModel,
SkyReelsV2Transformer3DModel,
SparseControlNetModel,
StableAudioDiTModel,
T2IAdapter,
Expand Down Expand Up @@ -875,6 +882,7 @@
FlowMatchEulerDiscreteScheduler,
FlowMatchHeunDiscreteScheduler,
FlowMatchLCMScheduler,
FlowMatchUniPCMultistepScheduler,
HeunDiscreteScheduler,
IPNDMScheduler,
KarrasVeScheduler,
Expand Down Expand Up @@ -1029,6 +1037,10 @@
SemanticStableDiffusionPipeline,
ShapEImg2ImgPipeline,
ShapEPipeline,
SkyreelsV2DiffusionForcingImageToVideoPipeline,
SkyreelsV2DiffusionForcingPipeline,
SkyreelsV2ImageToVideoPipeline,
SkyreelsV2Pipeline,
StableAudioPipeline,
StableAudioProjectionModel,
StableCascadeCombinedPipeline,
Expand Down
2 changes: 2 additions & 0 deletions src/diffusers/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@
_import_structure["transformers.transformer_mochi"] = ["MochiTransformer3DModel"]
_import_structure["transformers.transformer_omnigen"] = ["OmniGenTransformer2DModel"]
_import_structure["transformers.transformer_sd3"] = ["SD3Transformer2DModel"]
_import_structure["transformers.transformer_skyreels_v2"] = ["SkyReelsV2Transformer3DModel"]
_import_structure["transformers.transformer_temporal"] = ["TransformerTemporalModel"]
_import_structure["transformers.transformer_wan"] = ["WanTransformer3DModel"]
_import_structure["unets.unet_1d"] = ["UNet1DModel"]
Expand Down Expand Up @@ -173,6 +174,7 @@
PriorTransformer,
SanaTransformer2DModel,
SD3Transformer2DModel,
SkyReelsV2Transformer3DModel,
StableAudioDiTModel,
T5FilmDecoder,
Transformer2DModel,
Expand Down
1 change: 1 addition & 0 deletions src/diffusers/models/transformers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,6 @@
from .transformer_mochi import MochiTransformer3DModel
from .transformer_omnigen import OmniGenTransformer2DModel
from .transformer_sd3 import SD3Transformer2DModel
from .transformer_skyreels_v2 import SkyReelsV2Transformer3DModel
from .transformer_temporal import TransformerTemporalModel
from .transformer_wan import WanTransformer3DModel
Loading