Skip to content

feat(megatron): add Target and DSpark Draft backends - #49

Closed
zhouaihui wants to merge 2 commits into
inclusionAI:mainfrom
zhouaihui:feat/megatron-backends
Closed

zhouaihui wants to merge 2 commits into
inclusionAI:mainfrom
zhouaihui:feat/megatron-backends

Conversation

@zhouaihui

@zhouaihui zhouaihui commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Outcome

Add independent upstream Megatron Core backends for frozen Qwen3 Target prefill and dense DSpark Draft training. This is the first integration PR for #42; THD/CP, Target PP and shared Target support remain follow-ups.

  • Target uses public GPT model/layer specs, strict HF weight mapping, Qwen3 BF16 norm order and direct feature capture into bounded Mooncake storage. Target TP1/TP2 does not materialize vocabulary logits or a persistent KV pool.
  • Draft uses public Core TP projections and FP32 DDP buffers while retaining ASpec's DSpark graph, complete-window supervision-weighted normalization, clipping/AdamW, work ledger, checkpointing and native assets. Vocabulary/confidence computations remain replicated. Input broadcasts run between microbatches; only source GET runs in the bounded background prefetcher.
  • Target/Draft selection is independent. Planning rejects unsupported combinations before worker startup. The current lane is BF16, one node, MBS1, CP1/PP1, dense full GQA, Target TP1/TP2 and Draft TP1/TP2 with DP.
  • Add the dependency-only ling-aspec-megatron profile alongside the existing Draft/SGLang profiles and shared runtime. All ASpec distributions use the same release version; each role owns its interpreter and native libraries. Pin Core 0.20.0 (685fb8bd970185e9ac3283fe32a5d6667bf4b268), Torch 2.13.0 and Transformer Engine 2.20.2. No Core changes or internal Megatron imports are required.

Verification

Linux Python 3.12, CUDA 13.0, SM90, locally supplied Qwen3-8B checkpoint:

  • Full offline suite: 3098 passed, 206 skipped, 51 subtests passed. Source constraints: 7 passed; additional SGLang-environment regressions: 39 passed. Ruff and maintained documentation links passed. GitHub quality and cpu-contract-tests also passed. Skips do not count as GPU acceptance.

  • Frozen Target TP1/TP2 versus HF at 128/4096 tokens: all five residual captures plus final norm passed; worst relative L2 0.8787%, minimum cosine 0.999961; parameters remained frozen.

  • Native Draft short-context loss/gradient/update and future-replay checks passed for TP1, TP2 and TP2 x DP2. The matched TP reference uses ordinary PyTorch BF16 projections without Core operators or distributed communication. Optional TP2 backbone recomputation also passed.

  • Online 4K/512-anchor checks use accumulation two, evaluation, four optimizer updates, a fresh process restart after update two, state/RNG/work audits, native export and strict loading:

    Target Draft Result
    SGLang TP1 Megatron TP1 x DP1 Passed
    Megatron TP2 Megatron TP2 x DP2 Passed; 16 unique training work items committed
    Megatron TP1 Existing FSDP Draft, installed wheels outside checkout Passed
  • Successful runs drained leases with zero resident feature outputs. Failed/canceled runs verified reader-process fencing before source teardown. Q/K CuTeDSL, tiled FP32 L1, block compaction and fused clipping were observed. Multi-chunk head cache/recomputation activation belongs to the THD/packing follow-up; the retained one-chunk workload does not establish it.

  • Built and audited four wheels and four sdists. Installed role profiles in three independent environments; CLI/doctor and module-origin checks passed outside the checkout. The baseline Draft environment has no Megatron installed. All 616 Python files in the official Core wheel matched the pinned public snapshot. Final source and wheel Python bytes match the tested commit.

Open acceptance gates — keep this PR in Draft

  • 4K TP2 update relative L2 = 3.01748%, above the unchanged 3% gate. Full 4K future bitwise replay also failed. Checkpoint state loading was exact; an in-memory restore also showed differing subsequent arithmetic. These are separate results, and the latter does not prove that every numerical difference has the same cause.
  • The unpartitioned BF16 TP2 reference update comparison also failed (approximately 9.7% on the first short-context update). Passing a matched TP arithmetic reference does not erase it. The same-gradient independent clipping/Adam oracle agreed below 3.1e-7 relative update L2.
  • No held-out quality or matched resident/continuous-online performance acceptance is claimed. The functional corpus repeats text to exercise bounded execution and recovery.

Compatibility and provenance

Exact checkpoint restoration requires the same Draft backend, TP topology and world size; cross-backend initialization is weights-only. Logical TP shards include FP32 masters/moments and Core RNG state; export gathers the existing strict native Draft format. Existing FSDP defaults and legacy checkpoint interpretation are retained.

The integration guide records setup/build prerequisites, runnable commands, tested topology, retained optimization boundaries and failed numerical gates. Source headers/references identify public Core and Megatron Bridge interfaces and retained ASpec/TorchSpec ancestry. This PR builds release artifacts without publishing them to PyPI.

@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@zhouaihui zhouaihui closed this Nov 18, 2026
@zhouaihui
zhouaihui deleted the feat/megatron-backends branch November 18, 2026 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant