| id | TD-025 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| title | KV-segment composition ("KV Lego") — research spike, engine-blocked, not a feature | ||||||||
| status | proposed | ||||||||
| date | 2026-06-22 | ||||||||
| supersedes | |||||||||
| superseded_by | |||||||||
| tags |
|
TD-023 reuses the KV of a fixed prefix. The tempting generalization: pre-compute the KV of independent segments —
KV_SYSTEM KV_AUTH KV_BILLING KV_DOCKER
— and assemble only the ones a task needs (merge(KV_SYSTEM, KV_AUTH, KV_TESTS)) instead of
prefilling the whole context. For a coding agent that keeps re-reading the same files, this is the
real prize: the variable body (selected files) is where prefill cost lives, and prefix-only caching
can't touch it.
KV is not position-independent. Keys/values depend on absolute position and RoPE, and on the
attention each token paid to everything before it. So KV_A ++ KV_B (naive concat) is
mathematically wrong — wrong positions, missing cross-segment attention. This is the part ChatGPT
rated 20–40%, and on the "just concatenate" reading it's ~0%.
| Approach | Idea | Cost |
|---|---|---|
| Prompt Cache (Gim et al., MLSys 2024) | "Prompt modules" with pre-assigned position ranges so a segment's KV is reusable wherever it's slotted | layout discipline; quality dip on cross-segment deps |
| CacheBlend (EuroSys 2025) | Reuse non-prefix KV chunks, then selectively recompute cross-attention for a small fraction of tokens to repair the dependencies | partial recompute (small, tunable) + a quality knob |
| APE / Block-Attention | Encode chunks independently, combine at inference (RAG-shaped) | calibration; accuracy trade |
So the honest verdict is not "impossible" — it's "a real technique with a real recompute cost and a real quality hit", which is a very different bet than "free KV Lego".
Keep it a research spike, not a roadmap feature — for now. Carbon does not adopt KV-segment composition until a concrete, measured path exists, because:
- Neither engine implements it. llama.cpp (IPEX + SYCL) and OpenVINO GenAI (TD-011) do prefix reuse, not arbitrary-segment splicing. Adopting this means patching an engine or moving to a runtime that supports it — a large, off-thesis commitment vs Carbon's "thin orchestrator" stance (TD-016).
- The win may not survive this hardware. Decode here is bandwidth-bound (TD-004); composition only saves prefill. CacheBlend's selective recompute partly re-pays that, so the net TTFT gain on the Arc 140T is unproven and must be measured before any engine surgery.
- Quantify the prize: measure what fraction of agentic TTFT is non-prefix prefill (files) vs the already-cached system prefix (TD-023). If small, stop — TD-024 + RAG already capture most of it.
- Paper-prototype CacheBlend offline (Python, outside the served engines) on 2–3 cached file segments; measure quality delta vs full prefill and the recompute fraction needed.
- Only if (1) is large and (2) holds quality at a small recompute cost → write a follow-up TD for an engine integration.
Highest-ceiling, highest-risk idea in the persistent-context set. Park as a documented spike; let TD-024 (no engine changes) and TD-020 (RAG) deliver the near-term wins first.
- Gim et al., Prompt Cache: Modular Attention Reuse for Low-Latency Inference, MLSys 2024.
- CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion, EuroSys 2025.