Skip to content

Real-model serving: llama.cpp backend and measured GPU throughput (P10) - #3

Merged
Abhayrkhot merged 1 commit into
mainfrom
p10-real-model
Nov 21, 2026
Merged

Abhayrkhot merged 1 commit into
mainfrom
p10-real-model

Conversation

@Abhayrkhot

Copy link
Copy Markdown
Owner

Stacked on #2 (P9). Until #2 merges, this diff is only P10.

What changed

Until now an Engine never ran a model, so every number was scheduler behaviour under an assumed cost. This PR puts a real model behind the same scheduler.

  • Executor seam. StepExecutor sits between ContinuousEngine's decisions (admission, batching, preemption, paged KV, radix prefix cache) and their execution. CostModelExecutor is the default and reproduces the P9 results exactly: 188.4 vs 298.5 QPS, 69.7 vs 39.3 µs per token.
  • llama.cpp executor. LlamaExecutor (llama.cpp pinned at b12188, optional build INFERSCHED_WITH_LLAMA) runs a GGUF model on exactly the scheduled batches:
    • one fused prefill+decode llama_decode per iteration;
    • a unified KV pool governed by PagedKvCache;
    • prefix hits share real KV cells through llama_memory_seq_cp, and cached blocks are parked in holder sequences until the radix cache evicts them;
    • greedy sampling on the device.
  • Benchmarks and tooling.
    • infersched_real_serving_bench: open-loop wall-clock benchmark reporting time to first token, time per token, tokens/s, NVML busy time and power.
    • scripts/exp_real_serving.py: experiment driver (Docker on the box, Apptainer under Slurm).
    • scripts/bench_openai_like.py: replays identical token-id workloads against llama-server and vLLM.
    • scripts/fit_cost_model.py: fits the cost model to measured steps.
    • scripts/explorer/*.sbatch: Explorer job scripts.
  • Engine flags. infersched_engine --backend llama --model … [--backend-sampling on] [--prefix-cache off].

Results (docs/p10-real-model-serving.md)

Measured on one NVIDIA H200 with Qwen3-8B FP16. All three systems served identical requests.

Comparison Result
Continuous vs request-level vs serial, 256-request burst 1,683 vs 1,251 vs 179 output tok/s; 0.27 vs 0.38 vs 3.56 J per output token
17.4 QPS offered Continuous 14.6 req/s at 60 ms median time to first token; request-level saturates at 11.0 req/s with 8.0 s median
llama-server, same kernels Saturates near 7.3 req/s with the GPU 42–48% busy; InferSched serves 1.96×
vLLM 0.34 Throughput within 1–4% up to 15.1 QPS; vLLM's time per token is 1.6–3.4× lower
Prefix cache on vs off Prompt compute −85%, time to first token p50 −59%, busy time only −5%
Cost model fitted to 4,298 steps 6.5 ms + 0.28 ms/sequence + 55 µs/prompt token; predicts measured capacity within 3–11%. The real batching gain is 1.35×; P9's modeled gain was ~2.1×

A second H200 trial agrees within 1.4% on throughput and 4% on medians.

Findings that cut against the claims:

  • NVML "GPU busy" is 90–98% under every policy, so it doesn't measure useful work.
  • Continuous batching raises time per token (p99 48.9 vs 15.9 ms) because new prompts ride in the same steps.
  • Prefix caching mostly saves latency; its compute saving is small.

Caveats:

  • InferSched is measured in-process, while llama-server and vLLM go through HTTP.
  • CUDA graphs are off for every llama.cpp run. At b12188 they crash on the H200 ("illegal instruction" in ggml_cuda_graph_launch_chunk) for steps carrying many prompts, which I reproduced in isolation. Where graphs are safe, turning them on adds 3.3%.
  • The full HTTP → Kafka → Router → Engine → Postgres path with a real model was checked only on CPU: 40/40 requests completed, and real text is stored.

Testing

  • Box: 76/76 default tests (55 service-free + 21 live Kafka/Postgres/Redis); 55/55 under ASan + LeakSanitizer; 62/62 in the CPU llama build.
  • The 7 real-model tests (260K-parameter model) also passed on the H200 node. They check:
    • every scheduled token comes out of the model;
    • a reused prefix gives the same greedy output as recomputing it;
    • device sampling picks the same tokens as the host argmax;
    • preemption, cancellation and eviction leave zero KV cells in both the scheduler and llama.cpp.
  • New CI job real-model-llama-cpu: CPU llama.cpp build, the real-model tests, and a benchmark smoke run.

Add a StepExecutor seam between ContinuousEngine's decisions and their
execution. The cost model stays the default (P1-P9 numbers are unchanged);
a llama.cpp executor (pinned release b12188) runs a GGUF model on exactly
the scheduled batches: one fused prefill+decode llama_decode per
iteration, KV cells governed by PagedKvCache, prefix-cache hits shared
through llama_memory_seq_cp with holder sequences that park cached blocks,
and greedy sampling on the device.

Measured on an H200 with Qwen3-8B (docs/p10-real-model-serving.md):
continuous batching 1,683 vs request-level 1,251 vs serial 179 output
tok/s; within 1-4% of vLLM 0.34 throughput on identical requests (vLLM's
time per token is 1.6-3.4x lower); 1.96x llama-server on the same
kernels; prefix caching cuts prompt compute 85%. The cost model fitted
to the measured steps predicts capacity within 3-11%.

Also adds the wall-clock serving benchmark, the experiment driver (Docker
or Apptainer), a replay client for llama-server and vLLM, the cost-model
fit, Explorer job scripts, a CPU real-model CI job and an executor
contract test. CUDA graphs are off for every llama.cpp run: at b12188
they crash on the H200 for steps that carry many prompts.
@Abhayrkhot
Abhayrkhot changed the base branch from p9-compute-efficiency to main November 21, 2026 17:58
@Abhayrkhot
Abhayrkhot merged commit 3a0a0c6 into main Nov 21, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant