This is the official reproduction of RelaCtrl, which represents an efficient controlnet-like architecture designed for DiTs.
RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers
Ke Cao*, Jing Wang*, Ao Ma*, Jiasong Feng, Zhanjie Zhang, Xuanhua He, Shanyuan Liu, Bo Cheng, Dawei Leng‡, Yuhui Yin, Jie Zhang‡(*Equal Contribution, ‡Corresponding Authors)
- [2025.04.07] We released the inference pipeline and some weights of RelaCtrl-PixArt.
- [2025.02.21] We have released our paper RelaCtrl and created a dedicated project homepage.
We are seeking academic interns in the AIGC field. If interested, please send your resume to [email protected].
conda create -n relactrl python=3.10
conda activate relactrl
pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu118
git clone https://github.com/360CVGroup/RelaCtrl.git
cd RelaCtrl
pip install -r requirements.txtDownload the necessary model weights for PixArt from the links below:
| Model | Parameters | Download Link |
|---|---|---|
| T5 | 4.3B | T5 |
| VAE | 80M | VAE |
| PixArt-α-1024 | 0.6B | PixArt-XL-2-1024-MS.pth or Diffusers Version |
Download the required conditional weights for RelaCtrl:
| Model | Parameters | Download Link |
|---|---|---|
| RelaCtrl_PixArt_Canny | 45M | Canny |
python pipeline/test_relactrl_pixart_1024.py diffusion/configs/config_relactrl_pixart_1024.pyThe PixArt model weights are derived from the open-source project PixArt-alpha.
Please refer to the original repository for detailed license information.
@misc{cao2025relactrl,
title={RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers},
author={Ke Cao and Jing Wang and Ao Ma and Jiasong Feng and Zhanjie Zhang and Xuanhua He and Shanyuan Liu and Bo Cheng and Dawei Leng and Yuhui Yin and Jie Zhang},
year={2025},
eprint={2502.14377},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2502.14377},
}