Skip to content

feat(medusa): register heads-only training and wire Plan/trainer - #41

Draft
sysuTobo wants to merge 4 commits into
inclusionAI:mainfrom
sysuTobo:feat/medusa-integration
Draft

sysuTobo wants to merge 4 commits into
inclusionAI:mainfrom
sysuTobo:feat/medusa-integration

Conversation

@sysuTobo

@sysuTobo sysuTobo commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Second increment for #17: register the heads-only Medusa family and connect it to Plan, the feature contract, the producer and the trainer, after the model/loss core landed in #32.

  • Registration: MedusaDraftModel resolves to the medusa family; heads-only attention/structure policies (no invented head geometry); checkpoint semantics include head count and depth.
  • Plan: fail-closed scope — SDPA only, CP1, MBS1, non-packing, no duplicate teacher slot, no compile/anchors/frozen head, and no user-supplied per-head weights. The objective records parallel future-token CE, prediction_start=2 and the full vocabulary.
  • Features: the single-slot post-norm final-state contract now covers {mtp, medusa}; the producer publishes the final state for medusa.
  • Trainer: MedusaTrainer reuses the existing optimizer/checkpoint/ownership path; the topology pins head count and uniform per-head weights, and exact resume rejects a head-count change.
  • Assets/docs: shipped Qwen3-8B K=5 draft config, a Store/TCP 4K example, docs/user-guide/medusa.md, and support/algorithms entries.
  • Tests: tests/cpu/test_medusa_integration.py covers registration, heads-only policies, Plan compilation plus one illegal case per gate, the producer hook, and checkpoint topology/resume.

Outcome: a config can now select the medusa family and compile a Plan, and training runs through the existing path. Export, a maintained recipe, per-head held-out observations and candidate-tree acceptance remain PR-3.

Checks actually run

Environment: isolated venv, Python 3.12, torch 2.13.0+cpu, transformers 5.12.1 (CI CPU lane), merged main e113c13.

  • ruff check aspec tests tools setup.py — pass
  • python -m tools.check_docs — pass
  • python -m compileall -q aspec tests tools — pass
  • pytest -q tests/source — 7 passed
  • CUDA_VISIBLE_DEVICES="" HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 pytest -q tests/cpu — 1047 passed (incl. 63 Medusa tests)
  • Merge with main (e113c13) kept the DFlash objective fields intact; the full offline suite passes on the merged tree.
  • Plan compile of configs/examples/medusa_qwen3_8b_4k_tcp.yaml against a local Qwen3-8B config shape: objective num_heads=5, single post-norm hidden_states, ttt_length=5, ploss_weights=[1.0]*5

GPU validation status (pending)

The 2x L20 smoke could not be scheduled: multi-GPU L20 requests stayed Pending while the user's own two-GPU jobs queued for 114+ minutes on the shared cluster, and the H200 pool is not available to this team. This PR claims no GPU training or resume result. The example uses SDPA and Store/TCP (no NVLink); the intended validation is --dry-run plan, an 8-step run and an exact resume.

Capacity note: K=5 on Qwen3-8B is about 3.2B trainable parameters (~51 GB with bf16 weights/gradients and Adam state in REPLICATE). A single 48 GB device is expected to OOM, so the smoke plan falls back to 1T+2D with fsdp_strategy: SHARD when needed.

Unverified scope

  • No GPU execution, exact resume or per-head training metrics on hardware yet.
  • Warm start (initial_draft_model_path), FULL_SHARD configurations beyond the documented fallback, packing/CP/FA4, multi-node.
  • Export, strict reloading, recipe catalog, held-out and acceptance evidence (PR-3).

Part of #17.

Register MedusaDraftModel as the medusa family and connect the heads-only
objective end to end:
- heads-only attention/structure policies and validation backends
- Plan gates: SDPA only, CP1, MBS1, non-packing, no teacher/logits slot,
  no compile/anchors/frozen head; objective records per-head CE
- single-slot post-norm feature contract shared with native MTP, and the
  producer publishes the final state for medusa
- MedusaTrainer over the existing optimizer/checkpoint path; topology pins
  head count and uniform per-head loss weights
- shipped Qwen3-8B K=5 draft config, Store/TCP 4K example, docs page
Adds CPU integration contracts for every gate and the producer hook.
@coderabbitai

coderabbitai Bot commented Nov 17, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@sysuTobo sysuTobo closed this Nov 17, 2026
@sysuTobo sysuTobo reopened this Nov 17, 2026
@sysuTobo

Copy link
Copy Markdown
Contributor Author

Branch updated to merge the latest main (e113c13) and CI is green.

  • The merge surfaced a real regression in my earlier commit: the DFlash objective had lost its family/block_size fields when the medusa branch was inserted. Fixed in 2f71f3f; the new test_target_mtp_features test from main now passes.
  • One CI run hit a 1-ULP mismatch (3.893681049 vs 3.893681526, 4.8e-7) in the pre-existing DMTP roundtrip test, which compares with rtol=0, atol=0 across two separately constructed modules. The same commit passed on retrigger; locally the full suite passes on Python 3.11 and 3.12 (1064 tests) and the test passes 5/5 isolated runs. I did not change that upstream gate.
  • GPU validation is still pending: 2x L20 cannot be scheduled on the shared cluster and the H200 pool is unavailable to this team.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant