Repository navigation
Real-model serving: llama.cpp backend and measured GPU throughput (P10) - #3
Merged
Merged
Conversation
Add a StepExecutor seam between ContinuousEngine's decisions and their execution. The cost model stays the default (P1-P9 numbers are unchanged); a llama.cpp executor (pinned release b12188) runs a GGUF model on exactly the scheduled batches: one fused prefill+decode llama_decode per iteration, KV cells governed by PagedKvCache, prefix-cache hits shared through llama_memory_seq_cp with holder sequences that park cached blocks, and greedy sampling on the device. Measured on an H200 with Qwen3-8B (docs/p10-real-model-serving.md): continuous batching 1,683 vs request-level 1,251 vs serial 179 output tok/s; within 1-4% of vLLM 0.34 throughput on identical requests (vLLM's time per token is 1.6-3.4x lower); 1.96x llama-server on the same kernels; prefix caching cuts prompt compute 85%. The cost model fitted to the measured steps predicts capacity within 3-11%. Also adds the wall-clock serving benchmark, the experiment driver (Docker or Apptainer), a replay client for llama-server and vLLM, the cost-model fit, Explorer job scripts, a CPU real-model CI job and an executor contract test. CUDA graphs are off for every llama.cpp run: at b12188 they crash on the H200 for steps that carry many prompts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2 (P9). Until #2 merges, this diff is only P10.
What changed
Until now an Engine never ran a model, so every number was scheduler behaviour under an assumed cost. This PR puts a real model behind the same scheduler.
StepExecutorsits betweenContinuousEngine's decisions (admission, batching, preemption, paged KV, radix prefix cache) and their execution.CostModelExecutoris the default and reproduces the P9 results exactly: 188.4 vs 298.5 QPS, 69.7 vs 39.3 µs per token.LlamaExecutor(llama.cpp pinned at b12188, optional buildINFERSCHED_WITH_LLAMA) runs a GGUF model on exactly the scheduled batches:llama_decodeper iteration;PagedKvCache;llama_memory_seq_cp, and cached blocks are parked in holder sequences until the radix cache evicts them;infersched_real_serving_bench: open-loop wall-clock benchmark reporting time to first token, time per token, tokens/s, NVML busy time and power.scripts/exp_real_serving.py: experiment driver (Docker on the box, Apptainer under Slurm).scripts/bench_openai_like.py: replays identical token-id workloads against llama-server and vLLM.scripts/fit_cost_model.py: fits the cost model to measured steps.scripts/explorer/*.sbatch: Explorer job scripts.infersched_engine --backend llama --model … [--backend-sampling on] [--prefix-cache off].Results (
docs/p10-real-model-serving.md)Measured on one NVIDIA H200 with Qwen3-8B FP16. All three systems served identical requests.
A second H200 trial agrees within 1.4% on throughput and 4% on medians.
Findings that cut against the claims:
Caveats:
ggml_cuda_graph_launch_chunk) for steps carrying many prompts, which I reproduced in isolation. Where graphs are safe, turning them on adds 3.3%.Testing
real-model-llama-cpu: CPU llama.cpp build, the real-model tests, and a benchmark smoke run.