v1.4.0

中文版

新特性

1. 新增 model_type 支持：bailing_moe、qwen3_asr。
2. 支持 Qwen3-Next 以 Mcore-GDN 方式运行（默认），从而支持序列 packing、FP8 及 CP。
3. 对 transformer_block / transformer_layer 进行重构，通过可继承的方式便于新模型的接入。
4. 兼容 Python 3.13。
5. 支持 transformers 中以 grouped 方式组织专家的 MoE 模型的 LoRA 权重存储与读取。（注意：该 LoRA 权重不支持通过 transformers 直接加载，但可通过 Megatron 加载以用于后续继续训练。）
6. 新增 padding_mask 支持，修复了在 padding_free=False 时，moe_aux_loss 对 padding token 错误计算 routing loss 的问题。

English Version

New Features

1. Added model_type support for bailing_moe and qwen3_asr.
2. Support running Qwen3-Next with Mcore-GDN (default), enabling sequence packing, FP8, and CP.
3. Refactored transformer_block / transformer_layer with an inheritable design to simplify the integration of new models.
4. Added compatibility with Python 3.13.
5. Support LoRA weight saving and loading for MoE models whose experts are organized in grouped mode in transformers. (Note: these LoRA weights cannot be loaded directly via transformers, but can be loaded via Megatron for continued training.)
6. Added padding_mask support, fixing an issue where moe_aux_loss incorrectly computed routing loss on padding tokens when padding_free=False.

What's Changed

• [bugfix] fix MTP & mcore 0.15 (NPU) by @Jintao-Huang in #67
• compat python 3.13 by @Jintao-Huang in #68
• compat lint py313 by @Jintao-Huang in #69
• compat lint py3.13 by @Jintao-Huang in #70
• [model] support bailing by @Jintao-Huang in #55
• update gpt_model by @Jintao-Huang in #71
• refactor transformer_block by @Jintao-Huang in #72
• [bugfix] fix tie_word_embeddings by @Jintao-Huang in #74
• [bugfix] fix qwen3_vl by @Jintao-Huang in #73
• remove hf_grouped lora error by @Jintao-Huang in #75
• [model] support qwen3_next gdn by @Jintao-Huang in #76
• compat megatron.core 0.18 by @Jintao-Huang in #77
• [model] support qwen3_asr by @Jintao-Huang in #78
• Support padding mask by @Jintao-Huang in #79
• compat peft 0.19 by @Jintao-Huang in #80
• [readme] Update readme by @Jintao-Huang in #81
• [docs] update readme by @Jintao-Huang in #82
• [bugfix] fix minimax qk_norm sp by @Jintao-Huang in #83

Full Changelog: v1.3.0...v1.4.0

Patch release v1.3.2

Full Changelog: v1.3.1...v1.3.2

Patch release v1.3.1

Full Changelog: v1.3.0...v1.3.1

v1.3.0

中文版

新特性

1. 新增 model_type 支持：kimi_k25、hy_v3、llava_onevision。
2. mlp_padding_free 兼容 Sequence Parallelism。
3. 移除对 megatron-core 0.12 - 0.14 版本的依赖支持。

English Version

New Features

1. Added model_type support: kimi_k25, hy_v3, llava_onevision.
2. mlp_padding_free is now compatible with Sequence Parallelism.
3. Removed dependency support for megatron-core versions 0.12 - 0.14.

What's Changed

• [docs] update readme by @Jintao-Huang in #49
• update requirements by @Jintao-Huang in #51
• npu qwen3.5 megatron padding_free fix by @addsubmuldiv in #50
• [model] support kimi_k25 by @Jintao-Huang in #52
• [model] support hy_v3 by @Jintao-Huang in #53
• Add support for LLaVA-OneVision-1.5 model by @randydl in #54
• [bugfix] fix torch_dtype by @Jintao-Huang in #57
• fix qwen3_next by @Jintao-Huang in #58
• remove mcore0.12-mcore0.14 by @Jintao-Huang in #59
• fix kwargs by @Jintao-Huang in #61
• [megatron] support mlp_padding_free & sp; refactor TransformerLayer by @Jintao-Huang in #62
• [bugfix] fix gather_from_sp by @Jintao-Huang in #63
• update transformers by @Jintao-Huang in #65
• update requirements by @Jintao-Huang in #66

New Contributors

• @randydl made their first contribution in #54

Full Changelog: v1.2.0...v1.3.0

Patch release v1.2.3

Full Changelog: v1.2.2...v1.2.3

Patch release v1.2.2

Full Changelog: v1.2.1...v1.2.2

Patch release v1.2.1

Full Changelog: v1.2.0...v1.2.1

v1.2.0

中文版

新特性

1. 支持 GLM-5 共享参数 MTP ，可通过mtp_shared_weights参数启用。
2. 支持 Qwen3.5 FP8 训练和权重导入导出。
3. 支持控制 MTP 分支中 decoder_input 是否停止梯度，即 MTP loss 能否直接通过 decoder_input 回传梯度到 Embedding/ViT，使用mtp_decoder_input_detach参数。
4. 昇腾 NPU 训练兼容 megatron-core 0.15.3。

English Version

New Features

1. Added support for GLM-5 shared-weight MTP, which can be enabled via the mtp_shared_weights argument.
2. Added support for Qwen3.5 FP8 training and FP8 weight import/export.
3. Added support for controlling whether gradients are stopped at decoder_input in the MTP branch, i.e., whether the MTP loss can be back-propagated through decoder_input to Embedding/ViT. This can be configured via the mtp_decoder_input_detach argument.
4. Added compatibility with Megatron-Core 0.15.3 for training on Huawei Ascend NPU.

What's Changed

• [docs] update readme by @Jintao-Huang in #17
• [qwen3.5] compat transformers 5.4.0 by @Jintao-Huang in #18
• [bugfix] fix gptq_bridge by @Jintao-Huang in #19
• Revert qwen3.5 save weight by @Jintao-Huang in #20
• [bugfix] fix multimodal mtp by @Jintao-Huang in #21
• update get_parameter_local_cp by @Jintao-Huang in #22
• [bugfix] Fix the multi-LoRA issue in Twinkle by @Jintao-Huang in #24
• Adapt Mindspeed/Megatron 0.15.3 by @addsubmuldiv in #25
• [bugfix] fix qwen3.5 gpt_bridge lora by @Jintao-Huang in #28
• [bugfix] fix gdn sharded_state_dict lora by @Jintao-Huang in #23
• support Qwen3.5 FP8 by @Jintao-Huang in #30
• [bugfix] fix fp8 by @Jintao-Huang in #32
• [bugfix] fix set_module lora by @Jintao-Huang in #33
• [compat] gdn compat mcore main by @Jintao-Huang in #34
• [bugfix] Fix mtp fp8 by @Jintao-Huang in #35
• support mtp_decoder_input_detach by @Jintao-Huang in #37
• [bugfix] fix gate_up_proj by @Jintao-Huang in #39
• fix mtp_num_layer >= 2 multimodal by @Jintao-Huang in #40
• support mtp_shared_weights by @Jintao-Huang in #41
• compat peft 0.19 by @Jintao-Huang in #42
• [bugfix] fix peft_format qwen3_5_moe by @Jintao-Huang in #43
• fix: Add is_mtp parameter to _set_moe_state avoid type error by @0hujun in #45
• [bugfix] fix grpo qwen3_5_moe full by @Jintao-Huang in #46
• [bugfix] fix safe_ddp_context hang by @Jintao-Huang in #47

New Contributors

• @addsubmuldiv made their first contribution in #25
• @0hujun made their first contribution in #45

Full Changelog: v1.1.0...v1.2.0

Patch release v1.1.2

Full Changelog: v1.1.1...v1.1.2

Patch release v1.1.1

Full Changelog: v1.1.0...v1.1.1