Repository navigation
Measure compute efficiency, not just throughput - #2
Merged
Merged
Conversation
Throughput can rise by spending more compute, so the Engine now accounts modeled accelerator time per delivered token, and four experiments each change one scheduling decision. Engine accounting (ContinuousEngineStats, EngineService::UtilizationJson) - Decode tokens computed per step (mean batch occupancy), useful output tokens, tokens wasted by preemption/cancellation/abandoned attempts, prefill tokens computed vs reused, modeled prefill/decode/busy time. - Engines print the snapshot as a JSON line on SIGTERM. Experiment switches - AdmissionPolicy::kStatic (request-level batching) inside the same ContinuousEngine, so only the policy differs. - Router --routing least-loaded|round-robin|first; --redis none. - Engine --cancellation off, --policy, and --*-us cost-model flags. Results (docs/p9-compute-efficiency.md) - Batching, deterministic: on mixed output lengths with a memory-bound model, continuous sustains 400 vs ~188 QPS, with 44% less decode compute per token at 300 QPS. CTest freezes it. - Prefix affinity: prefill computed -84%, fleet hardware time -29..-33%. - Cancellation: hardware time -34..-42% for the same completions. - Least-loaded vs round-robin on a 3x-slower Engine: -10..-19% hardware. - Reported against: continuous admission raises p99 on prefill-heavy load (no chunked prefill), and consolidating onto one Engine uses ~60% less compute than spreading. Test fix: the multi-Router chaos test now starts its fresh consumer group at the end of the topic instead of replaying all history from earliest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Answers "does the scheduling improve compute utilization, not just throughput?" Write-up:
docs/p9-compute-efficiency.md.What was missing
The only utilization figure was a simulated busy percentage with no baseline. Busy time can't tell productive work from waste, and the docs said no static-vs-continuous comparison existed. The distributed benchmarks ran Engines at a fixed per-iteration delay, so batch size couldn't affect cost there.
What this adds
AdmissionPolicy::kStaticinside the sameContinuousEngine--routing least-loaded|round-robin|first--redis none--cancellation offscripts/exp_batching_policy.sh(deterministic) andscripts/exp_fleet_efficiency.py(real processes, 3 runs).Results
Reported against:
Testing
MultiRouterKillAndRebalance. With the unique consumer groups from Route across an Engine fleet with concurrent, fenced dispatch #1, it replayed all ofinference.requestsfromearlieston a long-lived topic. The fresh group now starts at the end of the topic.