You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
bugSomething isn't workingpendingThis problem is yet to be addressednpuThis problem is related to NPU devices
1 participant
Converted from issue
This discussion was converted from issue #9015 on August 25, 2025 07:41.
Heading
Bold
Italic
Quote
Code
Link
Numbered list
Unordered list
Task list
Attach files
Mention
Reference
Menu
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Reminder
System Info
在NPU2节点*8卡上跑qwen3-32B时,微调过程调整序列长度是2K的情况下能跑下来,但是超过2K,例如4K往上就跑不下来了,模型加载过程就会报OOM;这时候增加卡数,也不好使。在开启deepspeed的stage3的cpu卸载时,模型加载过程不会报OOM但是在训练时候反向传播过程会报OOM。请问有办法解决吗?另外训练2K序列长度的qwen3-32B模型在GPU上部署的时候加载模型之后也会报OOM。
Reproduction
Others
No response
All reactions