Welcome to the magical world of Diffusion models! DiffSynth-Studio is an open-source Diffusion model engine developed and maintained by the ModelScope Community. We aim to build a universal Diffusion model framework that fosters technological innovation through framework construction, aggregates the power of the open-source community, and explores the boundaries of generative model technology!
Documentation Reading Guide
graph LR;
I_want_to_use_models_for_inference_and_training-->sec1[Section 1: Getting Started];
I_want_to_use_models_for_inference_and_training-->sec2[Section 2: Model Details];
I_want_to_use_models_for_inference_and_training-->sec3[Section 3: Training Framework];
I_want_to_develop_based_on_this_framework-->sec3[Section 3: Training Framework];
I_want_to_develop_based_on_this_framework-->sec4[Section 4: Model Integration];
I_want_to_develop_based_on_this_framework-->sec5[Section 5: API Reference];
I_want_to_explore_new_technologies_based_on_this_project-->sec4[Section 4: Model Integration];
I_want_to_explore_new_technologies_based_on_this_project-->sec5[Section 5: API Reference];
I_want_to_explore_new_technologies_based_on_this_project-->sec6[Section 6: Academic Guide];
I_encountered_a_problem-->sec7[Section 7: Frequently Asked Questions];
This section introduces the basic usage of DiffSynth-Studio, including how to enable VRAM management for inference on GPUs with extremely low VRAM, and how to train various base models, LoRAs, ControlNets, and other models.
- Installation Dependencies
- Model Inference
- VRAM Management
- Model Training
- Environment Variables
- GPU/NPU Support
This section introduces the Diffusion models supported by DiffSynth-Studio. Some model pipelines feature special functionalities such as controllable generation and parallel acceleration.
This section introduces the design philosophy of the training framework in DiffSynth-Studio, helping developers understand the principles of Diffusion model training algorithms.
- Basic Principles of Diffusion Models
- Standard Supervised Training
- Enabling FP8 Precision in Training
- End-to-End Distillation Accelerated Training
- Two-Stage Split Training
- Differential LoRA Training
This section introduces how to integrate models into DiffSynth-Studio to utilize the framework's basic functions, helping developers provide support for new models in this project or perform inference and training of private models.
- Integrating Model Architecture
- Building a Pipeline
- Enabling Fine-Grained VRAM Management
- Model Training Integration
This section introduces the independent core module diffsynth.core in DiffSynth-Studio, explaining how internal functions are designed and operate. Developers can use these functional modules in other codebase developments if needed.
diffsynth.core.attention: Attention mechanism implementationdiffsynth.core.data: Data processing operators and general datasetsdiffsynth.core.gradient: Gradient checkpointingdiffsynth.core.loader: Model download and loadingdiffsynth.core.vram: VRAM management
This section introduces how to use DiffSynth-Studio to train new models, helping researchers explore new model technologies.
- Training models from scratch
- Inference improvement techniques 【coming soon】
- Designing controllable generation models 【coming soon】
- Creating new training paradigms 【coming soon】
This section summarizes common developer questions. If you encounter issues during usage or development, please refer to this section. If you still cannot resolve the problem, please submit an issue on GitHub.