Repository navigation
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Outcome
Add independent upstream Megatron Core backends for frozen Qwen3 Target prefill and dense DSpark Draft training. This is the first integration PR for #42; THD/CP, Target PP and shared Target support remain follow-ups.
ling-aspec-megatronprofile alongside the existing Draft/SGLang profiles and shared runtime. All ASpec distributions use the same release version; each role owns its interpreter and native libraries. Pin Core 0.20.0 (685fb8bd970185e9ac3283fe32a5d6667bf4b268), Torch 2.13.0 and Transformer Engine 2.20.2. No Core changes or internal Megatron imports are required.Verification
Linux Python 3.12, CUDA 13.0, SM90, locally supplied Qwen3-8B checkpoint:
Full offline suite: 3098 passed, 206 skipped, 51 subtests passed. Source constraints: 7 passed; additional SGLang-environment regressions: 39 passed. Ruff and maintained documentation links passed. GitHub
qualityandcpu-contract-testsalso passed. Skips do not count as GPU acceptance.Frozen Target TP1/TP2 versus HF at 128/4096 tokens: all five residual captures plus final norm passed; worst relative L2 0.8787%, minimum cosine 0.999961; parameters remained frozen.
Native Draft short-context loss/gradient/update and future-replay checks passed for TP1, TP2 and TP2 x DP2. The matched TP reference uses ordinary PyTorch BF16 projections without Core operators or distributed communication. Optional TP2 backbone recomputation also passed.
Online 4K/512-anchor checks use accumulation two, evaluation, four optimizer updates, a fresh process restart after update two, state/RNG/work audits, native export and strict loading:
Successful runs drained leases with zero resident feature outputs. Failed/canceled runs verified reader-process fencing before source teardown. Q/K CuTeDSL, tiled FP32 L1, block compaction and fused clipping were observed. Multi-chunk head cache/recomputation activation belongs to the THD/packing follow-up; the retained one-chunk workload does not establish it.
Built and audited four wheels and four sdists. Installed role profiles in three independent environments; CLI/doctor and module-origin checks passed outside the checkout. The baseline Draft environment has no Megatron installed. All 616 Python files in the official Core wheel matched the pinned public snapshot. Final source and wheel Python bytes match the tested commit.
Open acceptance gates — keep this PR in Draft
Compatibility and provenance
Exact checkpoint restoration requires the same Draft backend, TP topology and world size; cross-backend initialization is weights-only. Logical TP shards include FP32 masters/moments and Core RNG state; export gathers the existing strict native Draft format. Existing FSDP defaults and legacy checkpoint interpretation are retained.
The integration guide records setup/build prerequisites, runnable commands, tested topology, retained optimization boundaries and failed numerical gates. Source headers/references identify public Core and Megatron Bridge interfaces and retained ASpec/TorchSpec ancestry. This PR builds release artifacts without publishing them to PyPI.