prefix-sharing-disaggregated-bench

Simulation benchmark for prefix KV sharing in disaggregated LLM serving.

Measures when a decode node should transfer, recompute, or persistently replicate a prefix whose cache hit happened on a separate prefill node.

Why This Exists

Three prior projects created the pieces without connecting them:

• disaggregated-prefill-decode-sim showed that prefill and decode can run on separate nodes.
• system-prompt-cache-bench showed that shared prefixes can eliminate most prefill work.
• multi-turn-kv-reuse-bench showed that routing misses destroy accumulated locality.

That leaves an important systems question:

If the prefix KV cache lives on the prefill node, what is the best way to make it useful to the decode node?

This project compares four answers:

1. transfer_per_request — move the prefix KV on every request
2. recompute_on_decode — rebuild the prefix on the decode node
3. replicate_hotset — keep known hot prefixes resident on the decode node
4. replicate_all_lru — opportunistically persist every reused prefix under LRU eviction

Key Results

1) Persistent replication is clearly worth it when memory headroom exists

In the relaxed regime (budget_factor >= 0.75), decode-side replication dominates:

Policy	Mean TTFT	Throughput	Reject rate	Shared request frac	Replica hit rate
transfer_per_request	198.1 ms	7.65 req/s	0.7%	0.000	0.000
recompute_on_decode	205.3 ms	7.60 req/s	1.2%	0.000	0.000
replicate_hotset	150.5 ms	7.67 req/s	0.6%	0.602	0.395
replicate_all_lru	112.8 ms	7.69 req/s	0.4%	0.775	0.474

Relative to per-request transfer:

• replicate_all_lru cuts TTFT by 43.0%
• replicate_hotset cuts TTFT by 24.0%

The improvement comes from turning decode-side reuse into a persistent memory asset instead of paying transfer cost every time.

2) LRU replication is the strongest pure latency policy

Best relaxed-regime case:

• Model: qwen2_0.5b
• Workload: enterprise_hotset
• Budget: 1.00
• transfer_per_request: 165.4 ms
• replicate_all_lru: 51.2 ms
• TTFT reduction: 69.0%

This policy wins by caching aggressively on the decode node whenever enough memory headroom exists.

3) Prewarmed hotset replication is the best operational compromise

After adding:

• admission-time trimming of inactive replicas before rejecting requests
• prewarming of known hot prefixes for replicate_hotset

the hotset policy became much stronger:

• Relaxed-regime TTFT: 150.5 ms
• TTFT improvement vs transfer: 24.0%
• Relaxed shared request fraction: 0.602
• Relaxed replica hit rate: 0.395
• Relaxed average replica KV: 59.6 MB

Compared with full LRU replication:

• replicate_hotset average replica KV: 59.6 MB
• replicate_all_lru average replica KV: 85.4 MB

So hotset replication captures much of the latency gain with a simpler and more predictable operational model.

4) Tight budgets expose the memory admission cliff

In the tight regime (budget_factor <= 0.50), differences compress sharply:

Policy	Mean TTFT	Throughput	Reject rate	Shared request frac	Replica hit rate
transfer_per_request	582.0 ms	6.64 req/s	12.5%	0.000	0.000
recompute_on_decode	548.0 ms	6.68 req/s	12.3%	0.000	0.000
replicate_hotset	563.7 ms	6.65 req/s	12.4%	0.330	0.166
replicate_all_lru	546.4 ms	6.68 req/s	12.1%	0.476	0.204

Relative to transfer in tight budgets:

• replicate_all_lru: 6.1% TTFT improvement
• replicate_hotset: 3.2% TTFT improvement

Average gains still exist, but they are much smaller because resident replicas now compete directly with active decode KV.

5) Near the cliff, raw headroom matters more than sharing policy

Representative failure case: Qwen2-1.5B / enterprise_hotset / budget_factor = 0.35

Policy	Mean TTFT	Reject rate	Throughput	Shared request frac
recompute_on_decode	1056.7 ms	66.4%	3.56 req/s	0.000
replicate_all_lru	1024.0 ms	62.7%	3.96 req/s	0.022
replicate_hotset	1024.0 ms	62.7%	3.96 req/s	0.000
transfer_per_request	1024.0 ms	62.7%	3.96 req/s	0.000

At this point, no strategy can fully escape the admission cliff.

6) Transfer vs recompute is model-dependent, not universal

Across all workloads and budgets:

• Qwen2-0.5B: recompute beats transfer everywhere in this latency model (transfer - recompute ranges from 13.1 ms to 361.2 ms)
• Qwen2-1.5B: transfer beats recompute everywhere in this latency model (transfer - recompute ranges from -198.9 ms to -5.4 ms)

So the transfer-vs-recompute decision depends on the model's compute slope relative to KV transfer slope.

Hotset Policy Details

replicate_hotset uses a prewarmed decode-side replica set for globally popular prefixes.

Average results by workload after prewarm:

Workload	Mean TTFT	Reject rate	Shared request frac	Replica hit rate	Avg replica KV
enterprise_hotset	523.8 ms	16.4%	0.276	0.175	27.7 MB
long_context_support	728.2 ms	8.9%	0.377	0.163	37.7 MB
medium_chat	27.5 ms	0.0%	0.738	0.521	52.0 MB
tool_call	148.8 ms	0.6%	0.472	0.264	39.0 MB

Policies

Policy	Description
transfer_per_request	Transfer prefix KV from prefill node to decode node on each reusable request
recompute_on_decode	Ignore prefill-side reuse and rebuild prefix KV locally on decode
replicate_hotset	Keep a prewarmed resident set of globally hot prefixes on the decode node
replicate_all_lru	Opportunistically persist every reused prefix on decode until LRU eviction

Experimental Scope

• Models: 2
• Workloads: 4
• Budget factors: 4
• Policies: 4
• Seeds: 3
• Raw simulation runs: 384
• Aggregated scenario rows: 128

Balanced Recommendation Counts

• recompute_on_decode: 7
• replicate_all_lru: 24
• transfer_per_request: 1

Quick Start

git clone https://github.com/JohnScheuer/prefix-sharing-disaggregated-bench
cd prefix-sharing-disaggregated-bench

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

python run.py

Runtime: approximately 1-2 minutes.
Hardware: CPU-only simulation.

Output Files

results/
  summary.csv
  summary_agg.csv
  events.csv
  prefixes.csv
  recommendations.csv
  crossover.csv

plots/
  01_mean_ttft_by_policy.png
  02_throughput_vs_reject.png
  03_replica_memory_tradeoff.png
  04_transfer_recompute_crossover.png
  05_recommendation_matrix.png

Project Structure

prefix-sharing-disaggregated-bench/
├── src/
│   ├── config.py
│   ├── workload.py
│   ├── policies.py
│   ├── simulator.py
│   ├── bench.py
│   └── analysis.py
├── results/
├── plots/
├── run.py
├── SUMMARY.txt
├── DESIGN.md
├── LICENSE
└── requirements.txt

Documentation

• DESIGN.md — simulator design, assumptions, admission logic, replica cache behavior
• SUMMARY.txt — plain-text findings with the final benchmark numbers
• LICENSE — MIT License

Related Projects

• disaggregated-prefill-decode-sim
• system-prompt-cache-bench
• multi-turn-kv-reuse-bench
• kv-cache-tiering-bench

What This Benchmark Found

The main lesson is that prefix cache reuse changes meaning once serving is disaggregated.

In a coupled server, a prefix hit is just a local KV hit.
In a disaggregated server, a prefix hit becomes a new systems decision:

• transfer the KV
• recompute it
• or keep a persistent replica on the decode side

For this calibrated setup:

• relaxed headroom + high reuse → replicate_all_lru wins
• known hot prefixes + simpler operations → replicate_hotset is the best compromise
• tight headroom / admission cliff → no policy escapes memory pressure
• transfer vs recompute depends strongly on model size

License

This project is released under the MIT License.
See LICENSE for details.

Author

João Felipe De Souza