[2025-10] Beta Release of LMMs-Engine

[Luodian]

Models Scripts

We should test the following models scripts to see if they can run without bugs, we can also prepare corresponding CICD script for these models.

• Qwen LLM series
• Qwen VL series
• Qwen Omni Series

Optimizations

• Test BAGEL MFU on large-scale training.

Documentations

• main README to include highlights and scopes of lmms-engine.
• clean current documentations for features or models, re-org them to correct places.

[mwxely]

Some Comments on Stream Packing vs. Naive Map-Style DataLoader

Memory Efficiency

Stream Packing adopts an on-the-fly loading strategy that fundamentally decouples memory usage from dataset size:

• Constant Memory Footprint: Memory consumption is O(batch_size) rather than O(dataset_size)
• Real Impact: For a 500GB dataset, Stream Packing requires only ~10GB memory vs. 600GB+ for Naive approach
• Scale Without Limits: Theoretically supports TB-scale datasets on commodity hardware

Zero-Preprocessing Startup

No preprocessing overhead means you can begin training immediately:

• Naive Approach: Requires 30-60 minutes preprocessing for 100GB datasets (loading → token estimation → global packing optimization)
• Stream Packing: Starts in seconds, packing happens during iteration

Trade-offs

Since packing happens on-the-fly, the exact number of steps per epoch is not known upfront. This means:

• Training must be configured via max_steps instead of num_epochs
• Framework automatically logs perf/global_seq_len_avg metrics to help estimate progress
• This is rarely an issue for production training, and step count stabilizes after the first epoch

[BIGKnight]

Summary

1. I refactored the Muon and DLLM code paths. We now support both single-device Muon and multi-device Muon.

2. I also drafted a README for pretraining DLLM with the Muon optimizer.

Observation

In a quick test, the DLLM MFU is only ~0.06. I suspect two main causes:

1. The current dllm_trainer is still built on HF Trainer; we should add compatibility with Kaichen’s FSDP trainer.

2. The streaming data pipeline introduces I/O overhead and slows training.

TODO

1. Clean up the Wan components.

2. Since current DLLM training relies on the dllm_trainer class, add compatibility with the FSDP trainer.

[Luodian]

Efficiency Report

1. Qwen LLM series
2. Qwen VLM series
3. Qwen Omni series
4. BAGEL

[kcz358]

Parallelism Support (TODO)

1. Parallelism register plan, custom parallelism for different model type
2. Tensor parallelism plan on qwen3 vl
3. Expert parallelism plan potentially on qwen3_vl_moe