Skip to content

Measure compute efficiency, not just throughput - #2

Merged
Abhayrkhot merged 1 commit into
mainfrom
p9-compute-efficiency
Nov 21, 2026
Merged

Abhayrkhot merged 1 commit into
mainfrom
p9-compute-efficiency

Conversation

@Abhayrkhot

Copy link
Copy Markdown
Owner

Answers "does the scheduling improve compute utilization, not just throughput?" Write-up: docs/p9-compute-efficiency.md.

What was missing

The only utilization figure was a simulated busy percentage with no baseline. Busy time can't tell productive work from waste, and the docs said no static-vs-continuous comparison existed. The distributed benchmarks ran Engines at a fixed per-iteration delay, so batch size couldn't affect cost there.

What this adds

  • Engine compute accounting: batch occupancy, useful vs wasted tokens (preemption, cancellation, abandoned attempts), prefill computed vs reused, and modeled busy time, printed as JSON on Engine shutdown.
  • One-variable switches:
    • AdmissionPolicy::kStatic inside the same ContinuousEngine
    • --routing least-loaded|round-robin|first
    • --redis none
    • --cancellation off
    • cost-model flags
  • Experiments: scripts/exp_batching_policy.sh (deterministic) and scripts/exp_fleet_efficiency.py (real processes, 3 runs).

Results

Decision Effect on compute
Continuous vs request-level batching (mixed lengths, memory-bound model) 400 vs ~188 QPS sustained; −44% decode compute per token at 300 QPS
Redis prefix affinity (64 shared 512-token prefixes, 4 Engines) prefill −84%, hardware time −29% to −33%
Cancellation (10% of requests miss a 60 ms deadline) hardware time −34% to −42%, same completions
Least-loaded vs round-robin (one Engine 3× slower) hardware time −10% to −19%

Reported against:

  • Continuous admission raises p99 on prefill-heavy load, because chunked prefill isn't implemented.
  • Putting all traffic on one Engine uses ~60% less compute than spreading, so spreading is a latency choice, not a saving.
  • "Compute" here is the cost model's modeled accelerator time, not GPU kernels.

Testing

  • Box, Ubuntu 24.04 container: 74/74 tests (53 hermetic + 21 live), and ASan + LeakSanitizer 53/53 hermetic.
  • New tests: 5 unit tests plus a CTest that freezes the batching result.
  • Fixed MultiRouterKillAndRebalance. With the unique consumer groups from Route across an Engine fleet with concurrent, fenced dispatch #1, it replayed all of inference.requests from earliest on a long-lived topic. The fresh group now starts at the end of the topic.

Throughput can rise by spending more compute, so the Engine now accounts
modeled accelerator time per delivered token, and four experiments each
change one scheduling decision.

Engine accounting (ContinuousEngineStats, EngineService::UtilizationJson)
- Decode tokens computed per step (mean batch occupancy), useful output
  tokens, tokens wasted by preemption/cancellation/abandoned attempts,
  prefill tokens computed vs reused, modeled prefill/decode/busy time.
- Engines print the snapshot as a JSON line on SIGTERM.

Experiment switches
- AdmissionPolicy::kStatic (request-level batching) inside the same
  ContinuousEngine, so only the policy differs.
- Router --routing least-loaded|round-robin|first; --redis none.
- Engine --cancellation off, --policy, and --*-us cost-model flags.

Results (docs/p9-compute-efficiency.md)
- Batching, deterministic: on mixed output lengths with a memory-bound
  model, continuous sustains 400 vs ~188 QPS, with 44% less decode compute
  per token at 300 QPS. CTest freezes it.
- Prefix affinity: prefill computed -84%, fleet hardware time -29..-33%.
- Cancellation: hardware time -34..-42% for the same completions.
- Least-loaded vs round-robin on a 3x-slower Engine: -10..-19% hardware.
- Reported against: continuous admission raises p99 on prefill-heavy
  load (no chunked prefill), and consolidating onto one Engine uses ~60%
  less compute than spreading.

Test fix: the multi-Router chaos test now starts its fresh consumer group
at the end of the topic instead of replaying all history from earliest.
@Abhayrkhot
Abhayrkhot merged commit c0aa0e4 into main Nov 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant