MindSpore ONE

This repository contains SoTA algorithms, models, and interesting projects in the area of multimodal understanding and content generation.

ONE is short for "ONE for all"

News

• [2025.04.10] We release v0.3.0. More than 15 SoTA generative models are added, including Flux, CogView4, OpenSora2.0, Movie Gen 30B , CogVideoX 5B~30B. Have fun!
• [2025.02.21] We support DeepSeek Janus-Pro, a SoTA multimodal understanding and generation model. See here
• [2024.11.06] v0.2.0 is released

Quick tour

To install v0.3.0, please install MindSpore 2.5.0 and run pip install mindone

Alternatively, to install the latest version from the master branch, please run.

git clone https://github.com/mindspore-lab/mindone.git
cd mindone
pip install -e .

We support state-of-the-art diffusion models for generating images, audio, and video. Let's get started using Stable Diffusion 3 as an example.

Hello MindSpore from Stable Diffusion 3!

import mindspore
from mindone.diffusers import StableDiffusion3Pipeline

pipe = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium-diffusers",
    mindspore_dtype=mindspore.float16,
)
prompt = "A cat holding a sign that says 'Hello MindSpore'"
image = pipe(prompt)[0][0]
image.save("sd3.png")

run hf diffusers on mindspore

• mindone diffusers is under active development, most tasks were tested with mindspore 2.5.0 on Ascend Atlas 800T A2 machines.
• compatibale with hf diffusers 0.32.2

component	features
pipeline	support text-to-image,text-to-video,text-to-audio tasks 160+
models	support audoencoder & transformers base models same as hf diffusers 50+
schedulers	support diffusion schedulers (e.g., ddpm and dpm solver) same as hf diffusers 35+

supported models under mindone/examples

task	model	inference	finetune	pretrain	institute
Image-to-Video	hunyuanvideo-i2v 🔥🔥	✅	✖️	✖️	Tencent
Text/Image-to-Video	wan2.1 🔥🔥🔥	✅	✖️	✖️	Alibaba
Text-to-Image	cogview4 🔥🔥🔥	✅	✖️	✖️	Zhipuai
Text-to-Video	step_video_t2v 🔥🔥	✅	✖️	✖️	StepFun
Image-Text-to-Text	qwen2_vl 🔥🔥🔥	✅	✖️	✖️	Alibaba
Any-to-Any	janus 🔥🔥🔥	✅	✅	✅	DeepSeek
Any-to-Any	emu3 🔥🔥	✅	✅	✅	BAAI
Class-to-Image	var🔥🔥	✅	✅	✅	ByteDance
Text/Image-to-Video	hpcai open sora 1.2/2.0 🔥🔥	✅	✅	✅	HPC-AI Tech
Text/Image-to-Video	cogvideox 1.5 5B~30B 🔥🔥	✅	✅	✅	Zhipu
Text-to-Video	open sora plan 1.3 🔥🔥	✅	✅	✅	PKU
Text-to-Video	hunyuanvideo 🔥🔥	✅	✅	✅	Tencent
Text-to-Video	movie gen 30B 🔥🔥	✅	✅	✅	Meta
Video-Encode-Decode	magvit	✅	✅	✅	Google
Text-to-Image	story_diffusion	✅	✖️	✖️	ByteDance
Image-to-Video	dynamicrafter	✅	✖️	✖️	Tencent
Video-to-Video	venhancer	✅	✖️	✖️	Shanghai AI Lab
Text-to-Video	t2v_turbo	✅	✅	✅	Google
Image-to-Video	svd	✅	✅	✅	Stability AI
Text-to-Video	animate diff	✅	✅	✅	CUHK
Text/Image-to-Video	video composer	✅	✅	✅	Alibaba
Text-to-Image	flux 🔥	✅	✅	✖️	Black Forest Lab
Text-to-Image	stable diffusion 3 🔥	✅	✅	✖️	Stability AI
Text-to-Image	kohya_sd_scripts	✅	✅	✖️	kohya
Text-to-Image	stable diffusion xl	✅	✅	✅	Stability AI
Text-to-Image	stable diffusion	✅	✅	✅	Stability AI
Text-to-Image	hunyuan_dit	✅	✅	✅	Tencent
Text-to-Image	pixart_sigma	✅	✅	✅	Huawei
Text-to-Image	fit	✅	✅	✅	Shanghai AI Lab
Class-to-Video	latte	✅	✅	✅	Shanghai AI Lab
Class-to-Image	dit	✅	✅	✅	Meta
Text-to-Image	t2i-adapter	✅	✅	✅	Shanghai AI Lab
Text-to-Image	ip adapter	✅	✅	✅	Tencent
Text-to-3D	mvdream	✅	✅	✅	ByteDance
Image-to-3D	instantmesh	✅	✅	✅	Tencent
Image-to-3D	sv3d	✅	✅	✅	Stability AI
Text/Image-to-3D	hunyuan3d-1.0	✅	✅	✅	Tencent

supported captioner

task	model	inference	finetune	pretrain	features
Image-Text-to-Text	pllava 🔥	✅	✖️	✖️	support video and image captioning