Who this is for: your app sends every request to one expensive model, but most requests are easy. You want to automatically send the easy ones to a cheap model and keep only the hard ones on the expensive one.
The plain idea. A router looks at each incoming query and decides — before making the call — whether a cheap model can handle it or it needs the frontier model. The catch: the router sits in front of every request, so it has to be nearly instant (a ~10ms budget). That's why it's a tiny classifier on text embeddings, not another LLM.
Every number below is from a real run on the author's own coding-agent traffic (June 2026, M1 Pro, $0 API spend).
- Main goal: route as much traffic as possible to the cheap model without hurting quality. Report the whole trade-off curve, not one point — how aggressive to be is a dial you set later.
- The guardrail that matters most: don't send a hard query to the cheap model. Getting that wrong fails a user; the reverse (sending an easy query to the expensive model) just wastes a little money. Bar: accuracy ≥ 0.90 on never-seen request types, and zero hard-queries-routed-cheap at the shipped setting.
- Router latency < 10ms.
The training data was ~92,000 logged prompts from real coding sessions. Real logs aren't a clean sample — you have to structure them first:
- Remove near-duplicates (5,294 → 1,747 unique). Agent traffic is heavily templated; the de-duplication warning fires exactly as designed. Keep template variants where the actual payload differs.
- Group into families by similarity (→ 149 families; the top 10 cover 63% of all traffic). Label at the family level, then spread the label to variants — and verify that spreading was safe.
- Split by family, never by individual record. If variants of the same template land in both training and test data, you get a fake "99% accuracy" that means nothing. This is called leakage, and it's the #1 way evaluations lie.
Labels were generated for free by the Claude Code session itself (no API key needed), under a frozen rule: cheap-OK = bounded tasks (classify, summarize given text, reformat); needs-frontier = planning, open-ended synthesis; when in doubt → frontier.
Round 1 scored 0.988 accuracy — but one hard query out of 23 got routed cheap. Bar failed. The autopsy is the most instructive part of this recipe:
The mistake was a "template twin." Two prompts shared ~90% of their text; one asked only for selections from a fixed menu (cheap-OK), the other also demanded open-ended designed questions (needs-frontier). The labels were correct — the router was fooled, because an embedding-based router blurs two prompts whose only real difference is a short instruction buried inside a long, near-identical prompt. This failure is invisible to record-level splits and to headline accuracy. Only the "zero hard-queries-routed-cheap" guardrail caught it.
The fix was more data for the weak spot, not a weaker bar: 19 boundary examples mined from unseen sessions, the deciding rule made explicit, retrain, and test on a fresh set (the round-1 test set was "spent" the moment it triggered a fix and can't be reused).
Fresh test = 60 never-before-seen families, 1,328 records:
| Metric | Result | Bar |
|---|---|---|
| Accuracy | 0.999 | ≥ 0.90 ✅ |
| Hard-queries-routed-cheap | 0 / 274 | 0 ✅ |
| Traffic handled by cheap model | 79.3% | (the dial) |
| Routed to cheap model with zero misses | 39.0% of events |
| Cost, all-frontier | $72.06 |
| Cost, hybrid | $53.04 — 26.4% saved |
| Router latency | ~0.6ms per query — 15× under budget |
| API spend to build it | $0.00 |
Cost model stated plainly: input tokens ≈ characters/4 plus 300 output tokens per event; Opus-tier $5/$25 vs Haiku-tier $1/$5 per million tokens (verified June 2026). Your traffic will differ — the reason routing pays is that a few templates dominate the volume, which is also why savings must be measured over the raw event stream, not unique queries.
- "Zero misses" is bounded by a 274-record sample and the quality of session-generated labels; both improve with scale. Re-checking fresh traffic monthly is the maintenance plan.
- No logs yet? Start with a simple category→model table and let the learned router replace it as logs accumulate.
- The ~21% kept on the frontier includes the borderline cases on purpose — those are the most valuable data for the next round.
Full run log, both rounds: dogfood/router/EXPERIMENT-LOG.md.