Lossless speculative decoding for Apple Silicon (MLX): 2× faster Qwen3 generation on a Mac with a trained hidden-state drafter. MIT.
-
Updated
Oct 22, 2026 - Python
Lossless speculative decoding for Apple Silicon (MLX): 2× faster Qwen3 generation on a Mac with a trained hidden-state drafter. MIT.
On-policy distillation for speculative decoding: train the draft model on the target's own rollouts, no offline feature cache. A fork of deepseek-ai/DeepSpec with DeepSeek-V4-Flash (DSpark) support.
Speculative decoding runtime with rejection sampling, adaptive gamma strategy, and provable correctness guarantees. Achieves 1.41x speedup on CPU with Qwen2-0.5B/1.5B pair. Draft model generates candidates, target model verifies in single forward pass. 31/31 tests passing.
Make your own LLM generate up to 2.8x faster with one command: train a DSpark speculative-decoding drafter for any local model and serve it with vLLM. Faster inference, same performance.
Multi-Future Drafting (MFD): mixture block drafters and an exact mixture tree for lossless speculative decoding. Inference code, SGLang patch and released drafters.
MiniCPM5-2B on a GTX 1080 (Pascal, sm_61): from 33 to 83 tokens/s with llama.cpp kernels, KV f16 and an FR-Spec draft, bit-identical where it matters
Draft-target model speculative decoding verification engine with rejection sampling, acceptance rate telemetry, and dynamic speedup ratio estimation.
Nanbeige4.3-33B on one M5 Ultra with TensorFold native Metal: the bf16 release as published, with its bf16 Nanbeige4.1-3B drafter (replies identical, 1.3-7x)
Tree-based speculative decoding benchmarked against linear under equal verification budget — branch factor, depth, and draft quality sweep with fair node-level comparison.
CLI for building and testing DFlash-style speculative decoding draft models.
Calibrated simulation benchmark for real-time LLM request routing, comparing complexity signals, output-length awareness, cost savings, and quality-risk trade-offs.
Analytical benchmark for speculative decoding at batch sizes 1-64. Finds the gamma crossover where speculative decoding becomes slower than greedy, and shows that batch size does not degrade performance when continuous refill is active.
Retrained EAGLE-3 draft head for Spark-X2.6-4B: draft acceptance on Chinese prompts. Rebuilt 48K draft vocabulary, trained on the deployed Q6_K weights.
Simulates speculative decoding to find the optimal speculation length K across 576 configurations (3 draft models x 8 K values x 6 acceptance rates x 4 cost ratios). Key findings: 6.06x max speedup, breakeven at cost_ratio=0.25, optimal K grows from 1-3 at 50% acceptance to 7-15 at 95% acceptance.
Speculative-draft training pipeline (DFLASH/DSPARK/DFLASH2), orchestrating the vllm-project/speculators engine end to end, with preset configs for Qwen-family models
Find the draft model (or n-gram lookup) that speeds up your local LLM: speculative-decoding sweeps for llama.cpp and vLLM with tok/s, acceptance rate, tokens per joule and peak VRAM, plus a pick per VRAM budget.
Check whether a draft model can speculate for a target model from GGUF headers or HF tokenizers before loading, scan folders for usable drafts, and bench real speedup on llama-server.
Measures real speculative decoding speedup using the official HuggingFace assistant_model API across 4 model pairs and output lengths up to 512 tokens. Best result: distilgpt2->gpt2-medium achieves 1.747x speedup at 512 output tokens. Validates that cost_ratio and output_length are the key parameters.
MTP drafter for lossless speculative decoding of Qwen3.8-27B-AEON on MLX. Weights on Hugging Face.
multiple tokens, and a verifier filters them using the main model’s confidence. Focuses on speed–accuracy tradeoffs, visualization, and modular design for easy benchmarking and research.
To associate your repository with the draft-model topic, visit your repo's landing page and select "manage topics."