Skip to content

Latest commit

 

History

History
151 lines (125 loc) · 13.4 KB

FLUX2.md

File metadata and controls

151 lines (125 loc) · 13.4 KB

FLUX.2

FLUX.2 is an image generation model trained and open-sourced by Black Forest Labs.

Model Lineage

graph LR;
    FLUX.2-Series-->black-forest-labs/FLUX.2-dev;
    FLUX.2-Series-->black-forest-labs/FLUX.2-klein-4B;
    FLUX.2-Series-->black-forest-labs/FLUX.2-klein-9B;
Loading

Installation

Before using this project for model inference and training, please install DiffSynth-Studio first.

git clone https://github.com/modelscope/DiffSynth-Studio.git
cd DiffSynth-Studio
pip install -e .

For more information about installation, please refer to Install Dependencies.

Quick Start

Run the following code to quickly load the black-forest-labs/FLUX.2-dev model and perform inference. VRAM management is enabled, and the framework will automatically control model parameter loading based on remaining VRAM. Minimum 10GB VRAM is required to run.

from diffsynth.pipelines.flux2_image import Flux2ImagePipeline, ModelConfig
import torch

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": torch.float8_e4m3fn,
    "onload_device": "cpu",
    "preparing_dtype": torch.float8_e4m3fn,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = Flux2ImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="black-forest-labs/FLUX.2-dev", origin_file_pattern="text_encoder/*.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.2-dev", origin_file_pattern="transformer/*.safetensors", **vram_config),
        ModelConfig(model_id="black-forest-labs/FLUX.2-dev", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"),
    ],
    tokenizer_config=ModelConfig(model_id="black-forest-labs/FLUX.2-dev", origin_file_pattern="tokenizer/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5,
)
prompt = "High resolution. A dreamy underwater portrait of a serene young woman in a flowing blue dress. Her hair floats softly around her face, strands delicately suspended in the water. Clear, shimmering light filters through, casting gentle highlights, while tiny bubbles rise around her. Her expression is calm, her features finely detailed—creating a tranquil, ethereal scene."
image = pipe(prompt, seed=42, rand_device="cuda", num_inference_steps=50)
image.save("image.jpg")

Model Overview

Model ID Inference Low VRAM Inference Full Training Validation After Full Training LoRA Training Validation After LoRA Training
black-forest-labs/FLUX.2-dev code code - - code code
black-forest-labs/FLUX.2-klein-4B code code code code code code
black-forest-labs/FLUX.2-klein-9B code code code code code code
black-forest-labs/FLUX.2-klein-base-4B code code code code code code
black-forest-labs/FLUX.2-klein-base-9B code code code code code code

Special Training Scripts:

  • Differential LoRA Training: doc
  • FP8 Precision Training: doc
  • Two-stage Split Training: doc
  • End-to-end Direct Distillation: doc

Model Inference

Models are loaded via Flux2ImagePipeline.from_pretrained, see Loading Models.

Input parameters for Flux2ImagePipeline inference include:

  • prompt: Prompt describing the content appearing in the image.
  • negative_prompt: Negative prompt describing content that should not appear in the image, default value is "".
  • cfg_scale: Classifier-free guidance parameter, default value is 1. When set to a value greater than 1, CFG is enabled.
  • height: Image height, must be a multiple of 16.
  • width: Image width, must be a multiple of 16.
  • seed: Random seed. Default is None, meaning completely random.
  • rand_device: Computing device for generating random Gaussian noise matrix, default is "cpu". When set to cuda, different GPUs will produce different generation results.
  • num_inference_steps: Number of inference steps, default value is 30.
  • embedded_guidance: Embedded guidance parameter, default value is 3.5.
  • t5_sequence_length: Sequence length of the T5 text encoder, default is 512.
  • tiled: Whether to enable VAE tiling inference, default is False. Setting to True can significantly reduce VRAM usage during VAE encoding/decoding stages, producing slight errors and slightly longer inference time.
  • tile_size: Tile size during VAE encoding/decoding stages, default is 128, only effective when tiled=True.
  • tile_stride: Tile stride during VAE encoding/decoding stages, default is 64, only effective when tiled=True, must be less than or equal to tile_size.
  • progress_bar_cmd: Progress bar, default is tqdm.tqdm. Can be disabled by setting to lambda x:x.

If VRAM is insufficient, please enable VRAM Management. We provide recommended low VRAM configurations for each model in the example code, see the table in the "Model Overview" section above.

Model Training

FLUX.2 series models are uniformly trained through examples/flux2/model_training/train.py, and the script parameters include:

  • General Training Parameters
    • Dataset Basic Configuration
      • --dataset_base_path: Root directory of the dataset.
      • --dataset_metadata_path: Metadata file path of the dataset.
      • --dataset_repeat: Number of times the dataset is repeated in each epoch.
      • --dataset_num_workers: Number of processes for each DataLoader.
      • --data_file_keys: Field names to be loaded from metadata, usually image or video file paths, separated by ,.
    • Model Loading Configuration
      • --model_paths: Paths of models to be loaded. JSON format.
      • --model_id_with_origin_paths: Model IDs with original paths, e.g., "black-forest-labs/FLUX.2-dev:text_encoder/*.safetensors". Separated by commas.
      • --extra_inputs: Extra input parameters required by the model Pipeline, e.g., controlnet_inputs when training ControlNet models, separated by ,.
      • --fp8_models: Models loaded in FP8 format, consistent with --model_paths or --model_id_with_origin_paths format. Currently only supports models whose parameters are not updated by gradients (no gradient backpropagation, or gradients only update their LoRA).
    • Training Basic Configuration
      • --learning_rate: Learning rate.
      • --num_epochs: Number of epochs.
      • --trainable_models: Trainable models, e.g., dit, vae, text_encoder.
      • --find_unused_parameters: Whether there are unused parameters in DDP training. Some models contain redundant parameters that do not participate in gradient calculation, and this setting needs to be enabled to avoid errors in multi-GPU training.
      • --weight_decay: Weight decay size, see torch.optim.AdamW.
      • --task: Training task, default is sft. Some models support more training modes, please refer to the documentation of each specific model.
    • Output Configuration
      • --output_path: Model saving path.
      • --remove_prefix_in_ckpt: Remove prefix in the state dict of the model file.
      • --save_steps: Interval of training steps to save the model. If this parameter is left blank, the model is saved once per epoch.
    • LoRA Configuration
      • --lora_base_model: Which model to add LoRA to.
      • --lora_target_modules: Which layers to add LoRA to.
      • --lora_rank: Rank of LoRA.
      • --lora_checkpoint: Path of the LoRA checkpoint. If this path is provided, LoRA will be loaded from this checkpoint.
      • --preset_lora_path: Preset LoRA checkpoint path. If this path is provided, this LoRA will be loaded in the form of being merged into the base model. This parameter is used for LoRA differential training.
      • --preset_lora_model: Model that the preset LoRA is merged into, e.g., dit.
    • Gradient Configuration
      • --use_gradient_checkpointing: Whether to enable gradient checkpointing.
      • --use_gradient_checkpointing_offload: Whether to offload gradient checkpointing to memory.
      • --gradient_accumulation_steps: Number of gradient accumulation steps.
    • Image Width/Height Configuration (Applicable to Image Generation and Video Generation Models)
      • --height: Height of image or video. Leave height and width blank to enable dynamic resolution.
      • --width: Width of image or video. Leave height and width blank to enable dynamic resolution.
      • --max_pixels: Maximum pixel area of image or video frames. When dynamic resolution is enabled, images with resolution larger than this value will be downscaled, and images with resolution smaller than this value will remain unchanged.
  • FLUX.2 Specific Parameters
    • --tokenizer_path: Path of the tokenizer, applicable to text-to-image models, leave blank to automatically download from remote.

We have built a sample image dataset for your testing. You can download this dataset with the following command:

modelscope download --dataset DiffSynth-Studio/example_image_dataset --local_dir ./data/example_image_dataset

We have written recommended training scripts for each model, please refer to the table in the "Model Overview" section above. For how to write model training scripts, please refer to Model Training; for more advanced training algorithms, please refer to Training Framework Detailed Explanation.