Skip to content

About

Find the draft model (or n-gram lookup) that speeds up your local LLM: speculative-decoding sweeps for llama.cpp and vLLM with tok/s, acceptance rate, tokens per joule and peak VRAM, plus a pick per VRAM budget.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

amagi-draftbench

Find out which draft model, if any, makes your local LLM faster on your own GPU. amagi-draftbench starts llama-server (or vLLM, or any server you give it) once per target and draft pair, then measures decode tok/s, speedup, draft acceptance rate, tokens per joule and peak VRAM. It also tests n-gram drafting, which needs no draft model at all, and recommends a draft for each VRAM budget.

amagi-draftbench showing RTX 3090 results and a per-VRAM recommendation

Install

pipx install amagi-draftbench

Not on PyPI yet. Until it is, install from a clone with pipx install . or pip install .

Python 3.9 or newer, no other dependencies. You also need a llama.cpp build with llama-server (or vLLM) and GGUF models.

Usage

Write a sweep config, run it, then read the results:

amagi-draftbench init-config --target ~/models/Qwen2.5-32B-Instruct-Q4_K_M.gguf \
    --draft ~/models/Qwen2.5-0.5B-Instruct-Q8_0.gguf --draft ~/models/Qwen2.5-1.5B-Instruct-Q4_K_M.gguf \
    --ngram --hardware rtx3090 -o configs/q25-32b.json
amagi-draftbench sweep configs/q25-32b.json
amagi-draftbench report results/ --by-prompt
amagi-draftbench recommend results/

sweep runs the target alone first, then once with each draft. Every run gets a fresh server, a warm-up request and the same prompts at temperature 0. Results land in results/<hardware>_<backend>_<name>.json. If a sweep stops partway, run it again and it carries on from where it stopped.

To benchmark one server you started yourself (upstream issue #2):

llama-server -m target.gguf -md draft.gguf -ngl 99 -ngld 99 -np 1 --port 8081
amagi-draftbench bench --url http://127.0.0.1:8081
amagi-draftbench bench --url http://127.0.0.1:8081 --compare-url http://127.0.0.1:8080   # 8080 runs without a draft

find-server shows which llama-server it will use and which draft features that build has:

$ amagi-draftbench find-server ~/llama.cpp/build/bin/llama-server
~/llama.cpp/build/bin/llama-server
  --spec-type (n-gram drafting): yes
  n-gram types: ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache
  draft size flag: --spec-draft-n-max
  -np (slots): yes

Results on an RTX 3090

One RTX 3090 (24 GB, power limit 300 W), llama.cpp build 11753 with flash attention, temperature 0, five prompts sent twice each. Replies are 512 tokens on the 7B and the Gemma MoE, and 256 tokens on the 27B and 32B (see the note on heat under How it measures). Qwen3.8 ran with thinking off. The raw files are in results/rtx3090/.

qwen25-32b (rtx3090, llamacpp)

Target Draft tok/s Speedup Accepted tok/J Watts VRAM GiB Same output
Qwen2.5-32B Q4_K_M none (baseline) 37.0 1.00x - 0.14 267 19.6 -
Qwen2.5-32B Q4_K_M Qwen2.5-0.5B Q8_0 62.8 1.70x 76% 0.24 252 20.2 5/10
Qwen2.5-32B Q4_K_M Qwen2.5-1.5B Q4_K_M 62.4 1.69x 85% 0.24 249 20.7 5/10
Qwen2.5-32B Q4_K_M Qwen2.5-1.5B Q8_0 60.9 1.65x 87% 0.23 255 21.3 5/10
Qwen2.5-32B Q4_K_M Qwen2.5-3B Q4_K_M 54.6 1.48x 84% 0.22 242 21.6 5/10
Qwen2.5-32B Q4_K_M ngram-mod 43.7 1.18x 70% 0.16 272 19.6 10/10

qwen25-7b (rtx3090, llamacpp)

Target Draft tok/s Speedup Accepted tok/J Watts VRAM GiB Same output
Qwen2.5-7B Q4_K_M none (baseline) 140.8 1.00x - 0.51 273 4.7 -
Qwen2.5-7B Q4_K_M Qwen2.5-0.5B Q8_0 170.2 1.21x 80% 0.59 282 5.3 10/10
Qwen2.5-7B Q4_K_M Qwen2.5-1.5B Q4_K_M 153.2 1.09x 90% 0.57 265 5.8 10/10
Qwen2.5-7B Q4_K_M ngram-mod 156.2 1.11x 35% 0.65 239 4.7 10/10

qwen38-27b (rtx3090, llamacpp)

Target Draft tok/s Speedup Accepted tok/J Watts VRAM GiB Same output
Qwen3.8-27B Q4_K_M none (baseline) 42.1 1.00x - 0.15 263 15.6 -
Qwen3.8-27B Q4_K_M Qwen3.5-0.8B Q8_0 44.6 1.06x 88% 0.18 239 16.5 6/10
Qwen3.8-27B Q4_K_M Qwen3.5-2B Q4_K_M 44.2 1.05x 89% 0.17 241 16.9 8/10
Qwen3.8-27B Q4_K_M ngram-mod 49.6 1.18x 63% 0.18 266 15.6 10/10

gemma4-26b-a4b (rtx3090, llamacpp)

Target Draft tok/s Speedup Accepted tok/J Watts VRAM GiB Same output
Gemma-4-26B-A4B Q4_K_M none (baseline) 122.3 1.00x - 0.42 282 16.6 -
Gemma-4-26B-A4B Q4_K_M Gemma-4-E2B Q4_K_M 92.2 0.75x 71% 0.34 262 18.2 1/10
Gemma-4-26B-A4B Q4_K_M ngram-mod 122.9 1.00x 35% 0.45 266 16.6 8/10

A draft model gave a 50-80% speedup on this 3090 only for the dense 32B.

  • Qwen2.5-32B Q4_K_M: a Qwen2.5-0.5B Q8_0 draft took it from 37.0 to 62.8 tok/s (1.70x), and 1.5B Q4_K_M did 1.69x. The 3B draft got a similar share of tokens accepted but costs more to run, so it landed at 1.48x. Energy per token improved along with speed, from 0.14 to 0.24 tok/J.
  • Qwen2.5-7B Q4_K_M: the same 0.5B draft gave 1.21x. The 7B already runs at 141 tok/s on its own, and the draft's own cost is a bigger share of each step.
  • Qwen3.8-27B Q4_K_M with Qwen3.5 drafts: 1.05x to 1.06x, even though 88-89% of draft tokens were accepted. A separate single request showed the same thing: the server reported 85% acceptance and 3.5 tokens per draft step, and decode stayed at 42.4 tok/s against 42.2 without the draft. I have not found the cause.
  • Gemma-4-26B-A4B (MoE): the Gemma-4-E2B draft made it slower, 122.3 down to 92.2 tok/s (0.75x). This MoE only reads about 4B parameters per token, so it is already fast, and running the draft costs more than it saves.
  • n-gram drafting (ngram-mod, no draft model, no extra VRAM) gave 1.11x to 1.18x on the Qwen targets and 1.00x on Gemma. All of the gain came from the two edit prompts, where the reply copies most of the prompt: 1.5x to 2.0x there, and 1.0x on fresh code, JSON and lists.

The per-prompt view for the 32B shows which kinds of reply each draft speeds up:

qwen25-32b: tok/s per prompt

Target Draft code json list edit_code edit_text
Qwen2.5-32B Q4_K_M none (baseline) 37 37 37 37 37
Qwen2.5-32B Q4_K_M Qwen2.5-0.5B Q8_0 66 (1.8x) 71 (1.9x) 45 (1.2x) 71 (1.9x) 74 (2.0x)
Qwen2.5-32B Q4_K_M Qwen2.5-1.5B Q4_K_M 63 (1.7x) 65 (1.8x) 53 (1.4x) 64 (1.7x) 70 (1.9x)
Qwen2.5-32B Q4_K_M Qwen2.5-1.5B Q8_0 64 (1.7x) 63 (1.7x) 51 (1.4x) 62 (1.7x) 67 (1.8x)
Qwen2.5-32B Q4_K_M Qwen2.5-3B Q4_K_M 55 (1.5x) 56 (1.5x) 48 (1.3x) 57 (1.6x) 60 (1.6x)
Qwen2.5-32B Q4_K_M ngram-mod 37 (1.0x) 37 (1.0x) 37 (1.0x) 58 (1.6x) 73 (2.0x)

recommend turns the same files into a pick per card size (24 GB shown):

$ amagi-draftbench recommend results/rtx3090 --budgets 24
24 GB card:
  Qwen2.5-32B Q4_K_M: draft Qwen2.5-0.5B Q8_0 -> 62.8 tok/s (1.70x vs 37.0 plain, 76% accepted), 20.2 GiB
  Qwen2.5-7B Q4_K_M: draft Qwen2.5-0.5B Q8_0 -> 170.2 tok/s (1.21x vs 140.8 plain, 80% accepted), 5.3 GiB
  Qwen3.8-27B Q4_K_M: draft ngram-mod -> 49.6 tok/s (1.18x vs 42.1 plain, 63% accepted), 15.6 GiB
      best draft model: Qwen3.5-0.8B Q8_0 -> 44.6 tok/s (1.06x vs 42.1 plain, 88% accepted), 16.5 GiB
  Gemma-4-26B-A4B Q4_K_M: skip drafts (122.3 tok/s, 16.6 GiB) -- no draft that fits is at least 5% faster; run it plain

n-gram drafting only pays off when the reply repeats text from the prompt (edits, refactors). Check 'report --by-prompt' before relying on it.

FAQ: the six open draftbench issues

#1, GGUF vs MLX. amagi-draftbench has a command backend that runs any server from a shell line, so mlx_lm.server --draft-model works through it (examples/mlx_command.json). I have no Apple Silicon results, so there is no GGUF vs MLX comparison here yet.

#2, how do I run a single benchmark? amagi-draftbench bench --url ... against a server you started. See Usage.

#3, TPS shows 0.0. Thinking models (Qwen3, DeepSeek-R1, gpt-oss) stream their reasoning in delta.reasoning_content, not delta.content. draftbench only counted content, so a reply that spent its whole token budget thinking never produced a "first token" and scored 0. Both fields count now. To turn thinking off, add "extra_body": {"chat_template_kwargs": {"enable_thinking": false}} to a config, or pass --no-think to bench.

#4, llama-server autodetection. Searches PATH (the approach from PR #4 by alxarch), then the usual build folders, including Windows build/bin/Release/llama-server.exe. Set llama_bin in the config to skip the search.

#5, 400 Bad Request. draftbench threw the server's error message away, so all you saw was "400". amagi-draftbench prints the message the server sent. It also starts llama-server with one slot (-np 1) and a context sized to fit the longest prompt plus max_tokens. Recent builds default to several slots that split the context, and a request that does not fit gets a 400. I could not reproduce the 400 with draftbench's own flags on llama.cpp build 11753: all four slots got the full 4096 tokens (unified KV cache) and every request returned 200. Older builds may still behave as the issue describes.

#6, vLLM. Current vLLM dropped --speculative-model in favor of --speculative-config. The vllm backend uses the new flag for both draft models and n-gram drafting, and reads acceptance from vLLM's /metrics counters (examples/vllm_qwen3.json). I have no vLLM results yet; the commands above come from sweep --dry-run.

How it measures

  • tok/s is decode speed. On llama.cpp it is the server's own timings.predicted_per_second, summed over all requests as total tokens / total decode time. Elsewhere it is measured on the client from the stream.
  • Accepted is draft tokens the target kept divided by draft tokens proposed (draft_n_accepted / draft_n), or vLLM's spec-decode counters.
  • tok/J is generated tokens divided by GPU energy, a trapezoid integral of nvidia-smi power readings taken every 200 ms during each request. Time spent cooling down between requests does not count.
  • VRAM GiB is peak memory.used during the run minus the idle reading before the server started.
  • Same output counts responses whose text hash matches the run without a draft. At temperature 0 a draft should not change the text in theory. In practice the target sees a batch of tokens during verification instead of one at a time, and the floating-point math changes enough to flip a token now and then. Once one token differs, the rest of the reply goes its own way.
  • N-gram runs start a fresh server for every scored request. llama-server keeps one n-gram table for the life of the process, shared by every request. In the first sweep each prompt ran twice on one server, and the second pass found its own first answer in the table: the 7B ran edit_text at 1442 tok/s on the second pass against 277 on the first. Those files are in results/rtx3090-ngram-warm-table/. In a long chat or agent session the table does carry over, so repeated content gets that boost for real. A benchmark of one request should not count it, so the sweep doesn't.

Hot cards: --temp-cmd reads memory temperature (for example a GDDR6X junction reader), --abort-mem/--abort-core stop the sweep and discard the run in progress, and --cool-at 92 --cool-to 82 makes it wait before a request until the card cools down. A dense 32B pushed this RTX 3090's memory to 108 C within one 512-token request, which is why the 27B and 32B configs use 256 tokens.

Config

{
  "name": "qwen25-32b",
  "hardware": "rtx3090",
  "backend": "llamacpp",
  "targets": [{"label": "Qwen2.5-32B Q4_K_M", "path": "/models/Qwen2.5-32B-Instruct-Q4_K_M.gguf"}],
  "drafts": [
    {"label": "Qwen2.5-0.5B Q8_0", "path": "/models/Qwen2.5-0.5B-Instruct-Q8_0.gguf"},
    {"label": "ngram-mod", "type": "ngram"}
  ],
  "settings": {"runs": 2, "max_tokens": 256, "prompts": "all", "extra_args": ["-fa", "on"]}
}

prompts is default (draftbench's code, JSON and list prompts), edit (two prompts that hand back given text with a small change, where n-gram drafting does best) or all. Other settings: llama_bin, port, ctx_size, temperature, extra_body, n_max per draft, ngram_cold. More in examples/.

Claude Code skill

.claude/skills/amagi-draftbench/SKILL.md teaches Claude Code when to run a sweep, how to read the numbers and what not to claim from them.

Limits

  • Results come from one RTX 3090, one llama.cpp build (11753) and Q4_K_M targets. Other cards, quants and builds will differ.
  • The prompts are short, and each reply is 256 or 512 tokens long. Long-context work is not covered.
  • Temperature 0 only. Sampling at higher temperatures lowers acceptance.
  • Power and VRAM come from nvidia-smi, so NVIDIA only.

Star

If this saved you a guessing session, a star helps other people find it.

More from AmagiLabs

amagi-tokbench measures how fast a local LLM server runs: generation and prompt tok/s, time to first token and peak VRAM. github.com/amagilabs has the other small open source dev tools from the same lab.

Credit

Based on draftbench by Alex Ziskind (MIT), github.com/alexziskind1/draftbench

The sweep loop, server launcher and config format started from draftbench. The reasoning-token fix, n-gram drafting, power and VRAM sampling, the per-prompt report, the recommendations and the tests are new. The results/upstream/ files are draftbench's own published results, kept so report can read the old format.

License

MIT, see LICENSE. Copyright (c) 2026 Alex, and Copyright (c) 2026 Amagisolutionsgroup, Inc. (AmagiLabs).

About

Find the draft model (or n-gram lookup) that speeds up your local LLM: speculative-decoding sweeps for llama.cpp and vLLM with tok/s, acceptance rate, tokens per joule and peak VRAM, plus a pick per VRAM budget.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages