[Feature] 请求支持qwen3.5系列模型

[BUJIDAOVS]

Motivation

请求支持qwen3.5系列模型

Related resources

No response

Additional context

No response

[maxin9966]

lmdeploy是不是不再针对turbomind模式，进行主流模型的四位量化适配了？已经很久没有支持主流模型的量化版本了，都是全量模式的适配。

[hfassold]

+1

[lvhan028]

lmdeploy是不是不再针对turbomind模式，进行主流模型的四位量化适配了？已经很久没有支持主流模型的量化版本了，都是全量模式的适配。

在进行。只是，像 qwen3-next，qwen3.5 这样结构的模型，对于 turbomind 引擎是巨大的挑战。要花比较长的时间。

[sacrrie]

现在部署的主力还是turbomind下的awq Qwen3 30B A3B 又快又好，加油

[ga-it]

感谢 @lvhan028 和 @lzhangzz 的回复。我完全理解，为 qwen3-next 和 qwen3.5 这样复杂的架构适配 TurboMind 引擎是一个巨大的挑战，并且需要花费较长的时间。非常感谢团队持续的努力！我将我的 Feature Request 附在下方，主要是想提供一些额外的背景信息，说明为什么大家如此迫切地期待 TurboMind 能支持 Qwen3.5，特别是对于依赖 Volta V100 等老一代企业级硬件的用户而言。

Support for Qwen3.5 Architecture (qwen3_5_moe / Gated Delta Networks) in TurboMind

Description

Motivation
Currently, attempting to serve the new Qwen3.5 models (e.g., Qwen/Qwen3.5-35B-A3B) using LMDeploy results in an automatic fallback to the slower PyTorch engine because the qwen3_5_moe architecture is not yet recognized by the C++ TurboMind engine:

WARNING - archs.py:45 - Fallback to pytorch engine because /root/.cache/huggingface/hub/models--Qwen--Qwen3.5-35B-A3B/... not supported by turbomind engine.

While other frameworks like vLLM and SGLang have added day-zero support for Qwen3.5, their implementations rely heavily on modern Triton and FlashInfer kernels. These kernels either fail to compile or suffer severe performance regressions on older architectures like Volta (sm70) and Turing (sm75).

Because TurboMind maintains its own highly optimized, bespoke CUDA kernels with excellent legacy hardware support, it remains the absolute best (and sometimes only) viable engine for running state-of-the-art models on enterprise hardware like 8x V100 32GB servers. Adding TurboMind support for Qwen3.5 would be a massive win for users operating on constrained or older GPU architectures who still want to leverage advanced reasoning and massive context windows.

Related resources

Example Model: Qwen/Qwen3.5-35B-A3B

Hugging Face Architecture: qwen3_5_moe (Requires transformers >= 5.0.0.dev0)

Previous related discussions: Issue #3961 (Qwen3-Next architecture preview)

Additional context
The Qwen3.5 architecture introduces several new mechanics that will likely require custom TurboMind layer objects:

Hybrid Attention Layout: An interleaved mix of Gated Delta Networks (Linear Attention) and Gated Attention (Standard Full Attention), typically in a 3:1 ratio.

Mismatched Head Dimensions: The linear attention layers use a head_dim of 128, while the standard attention layers use a head_dim of 256. (Note: This may also impact TurboMind's current KV cache quantization logic, which usually expects a uniform head dimension).

High-Sparsity MoE: e.g., 256 total experts with 8 routed experts + 1 shared expert activated per token.

Thank you to the LMDeploy team for your incredible work maintaining support for a wide range of hardware!