fix#1559
Draft
gbyu-amd wants to merge 24 commits into
Draft
Conversation
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
GLM-5.2 (glm_moe_dsa) was hard-excluded from the fused indexer path added for DeepSeek-V3.2 in #788, forcing the indexer's rope + fp8-quant + kv-cache write to run as 5-6 separate ops per layer per token. GLM's indexer is structurally identical to V3.2 (index_head_dim=128, qk_rope_head_dim=64, per_1x128 fp8 quant, always-neox indexer rope), so the fused indexer_qk_rope_quant_and_cache kernel is math-equivalent to the per-op path. Notably the recent GLM acc fix (da360a2) fixed a bug that only exists in the per-op path (dropping q_nope on recombine); the fused kernel never had it. - Allow glm_moe_dsa through _supports_fused_indexer_kernel_config, gated by a new env ATOM_ENABLE_GLM_FUSED_INDEXER (default on) for easy A/B + rollback. - Keep the wk+weights_proj GEMM merge OFF for GLM (decoupled in _can_fuse_indexer_wk_weights_proj): that fusion requires merging two checkpoint tensors and GLM names weights_proj "indexers_proj" without a pre-mergeable layout. The dominant win (rope+quant+cache) needs no merge. DeepSeek-V3.2 behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the guard that kept the indexer wk+weights_proj GEMM-merge off for glm_moe_dsa. The guard was overly conservative: it assumed GLM checkpoints ship the projection under a non-standard "indexers_proj" name that couldn't feed the packed_modules_mapping merge. In fact "indexers_proj" only appears in the HF quant config's modules_to_not_convert (remapped via quant_exclude_name_mapping); the actual checkpoint tensors use the standard indexer.wk (fp8 block-scale) + indexer.weights_proj (bf16) paths — exactly the layout IndexerWkWeightsProjLinear's fp8-wk load and the merge expect. GLM now takes the same path as DeepSeek-V3.2: wk + weights_proj collapse into a single BF16 GEMM (IndexerWkWeightsProjLinear), on top of the already-enabled qk-rope+quant+cache fusion. Still gated by ATOM_ENABLE_GLM_FUSED_INDEXER for rollback. DeepSeek-V3.2 behavior unchanged. Verify on GPU: weights load without missing indexer.wk / indexer.weights_proj, accuracy unchanged, and the two small indexer GEMMs collapse to one in the decode breakdown. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
* enable ar+norm+quant fusion * fix fusion with online quant
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Restore the "Offline Inference with DP Attention + Expert Parallel" example and the fp8-kv-cache caveat that were dropped when the GLM-5 recipe was reworked into the server-style configs. Requested by maintainer review. Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
- recipes/GLM-5.md: the GLM-5.2 FP8 server recipes loaded zai-org/GLM-5-FP8 (GLM-5) despite the GLM-5.2 titles; point them at zai-org/GLM-5.2-FP8. - quantization/quark/utils.py: stop passing a bare fp8_e8m0 literal for the MXFP8 online-quant scale. E8M0 is the block-scale dtype mandated by the MX (microscaling) format for any per_1x32 scheme (FP4 or FP8), so derive it from the scheme via a single-source _mx_block_scale_dtype() helper and gate on the MX scheme rather than special-casing fp8. The MXFP4 path reuses the same helper so there is one source of truth. Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
quant draft model to fp8 Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Merged
1 task
zejunchen-zejun
added a commit
that referenced
this pull request
Jul 11, 2026
…legal memory access (#1560) * [plugin][fix] DSA indexer: single-source decode_lens to fix random illegal memory access _build_indexer's multi-token-decode branch feeds torch.repeat_interleave with repeats=decode_lens and output_size=actual_expanded. With output_size given, repeat_interleave skips the device sync and trusts output_size == repeats.sum(); a violation makes its internal gather index run past the source rows, so the underlying index_select over-reads and triggers an intermittent illegal memory access (faults only when the over-read lands on an unmapped page -> random, data/timing dependent; surfaced late by async_copy_ready_event.synchronize() under async scheduling). The two operands came from different sources: decode_lens was diffed from the GPU query_start_loc, while actual_expanded came from the CPU query_start_loc_cpu. Under async scheduling the GPU/CPU copies can momentarily disagree, so repeats and output_size could mismatch. Derive decode_lens from the same CPU query_start_loc_cpu (copied into the persistent GPU buffer) so decode_lens.sum() == actual_expanded by construction. Same single-source-of-truth principle as the native-path fix in #1559. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [fix] ReplicatedEmbedding: mask out-of-range ids to fix random illegal memory access Under async scheduling + MTP spec-decode, input_ids can transiently carry the optimistic placeholder token -1 (an unresolved "assumed-accepted" draft/bonus slot: produced in gpu_model_runner as output_token_ids.extend([-1]*...), read back via the proposer's prepare_next_token_ids_padded backup before the deferred correction lands, during batch churn). It flows into BOTH the target input_ids and the shared draft input_ids (prev_sampled_token_ids = next_token_ids). GLM-5.2 uses ReplicatedEmbedding whose forward did a raw F.embedding, so -1 read the row before the [vocab, hidden] table -> random illegal memory access (BFloat16 indexSelectSmallIndex; faults only when the -1 row lands on an unmapped page -> rare, ~52 min). vLLM-native VocabParallelEmbedding masks the same -1 to 0 (get_masked_input_and_mask), which is why native never crashes; ReplicatedEmbedding dropped that safety net. Route replicated_embedding through the existing masked kernel with the full-table range [0, num_rows): out-of-range ids get a zero vector via a masked load (never a raw gather). Bit-identical to F.embedding for every valid token, and matches native's zero for -1; the unverified -1 slots are discarded/corrected by async spec-decode, so no accuracy change. Stays a torch_compile_guard custom op (opaque to inductor). Fixes both the target and the shared draft embedding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Technical Details
Test Plan
Test Result
Submission Checklist