Date: 2026-05-12 Sources: 150+ web searches, academic papers, production case studies, framework docs
- Karpathy's Context Engineering Framework
- The 100x Agentic Engineer
- Karpathy's 4 CLAUDE.md Rules (The "41% to 11%" Claim)
- The Mnilax 8 Additional Rules
- CLAUDE.md Best Practices
- Harness Engineering > Prompt Engineering
- Prompt Caching vs Semantic Caching
- KV Cache Management at Scale
- Speculative Decoding vs Quantization
- Structured Output Failures & Fallback Chains
- Evals: LLM-as-Judge + Human Evals
- Cost Attribution Per Feature
- Agent Guardrails & Loop Budgets
- LLM Observability
- Model Routing & Graceful Fallback Logic
- Fine-Tuning vs In-Context Learning
- DeepSeek V4 Pro — The 75% Discount Opportunity
- The Agentic Stack: Frameworks & Memory Systems
- ROI Maximization Playbook
- Synthesis: The Karpathy Playbook
"+1 for 'context engineering' over 'prompt engineering'. People associate prompts with short task descriptions you'd give an LLM in your day-to-day use. When in every industrial-strength LLM app, context engineering is the delicate art and science of filling the context window with just the right information for the next step." — Andrej Karpathy, X post
This was a quote-reply to Shopify CEO Tobi Lutke who originally pushed for the term.
Karpathy laid out three eras:
| Era | What Humans Do | Unit of Work |
|---|---|---|
| Software 1.0 | Write explicit code | Functions |
| Software 2.0 | Curate datasets for neural networks | Training examples |
| Software 3.0 | Write prompts; context window IS the program | Paragraphs |
"What's in the context window is your lever over the interpreter, and the interpreter is the LLM." — Karpathy, Sequoia Ascent 2026
- Prompt engineering = writing short task descriptions for day-to-day LLM use
- Context engineering = the full discipline of filling the context window with exactly the right information — system prompts, tool definitions, retrieved documents, conversation history, memory blocks, examples, guardrails
In production LLM apps, the "prompt" is often thousands of tokens of carefully orchestrated context, not a one-liner.
"Traditional software automates what you can specify. LLMs and reinforcement learning automate what you can verify."
This explains "jagged intelligence" — AI peaks where training data has clear reward signals (code, math, chess) and stagnates where verification is ambiguous.
"If you're in the circuits that were part of the RL, you fly."
Sources:
- Karpathy's Sequoia Ascent 2026 Blog
- Karpathy's Software 3.0 Playbook: 12 Lessons
- Karpathy Context Engineering Tweet
From Karpathy's fireside chat at Sequoia's AI Ascent 2026, amplified by Avid on X:
"People who are very good at this can peak much higher than that [10x]... People who master agentic workflows may outperform others by far more than 10x."
"Vibe coding raises the floor. Agentic engineering is about extrapolating the ceiling."
The agentic engineer does NOT blindly accept generated code. They:
- Design specs
- Supervise plans
- Inspect diffs
- Write tests
- Create evaluation loops
- Manage permissions
- Isolate worktrees
- Preserve quality
Key mindset quotes:
"You are not allowed to introduce vulnerabilities because of vibe coding. You are still responsible for your software."
"You can outsource your thinking, but you can't outsource your understanding."
"You still have to understand the fundamentals... storage, views, memory copies, invariants, identity, security boundaries."
Karpathy revealed that from November to December 2025, he went from 80% manual coding to only 20% manual. His candid admission: "The ability to write code manually is gradually atrophying" — described as "'a bit hard on the ego' but too useful to abandon."
Sources:
On January 26, 2026, Karpathy posted field notes from intensive Claude Code usage (tweet):
"I really am mostly programming in English now." "Easily the biggest change to my basic coding workflow in 2 decades."
Developer Forrest Chang (GitHub) distilled these into a CLAUDE.md file. The repo hit 126,000+ stars and 12,800+ forks — briefly the second most-starred repo on all of GitHub.
Rule 1 — Think Before Coding
"Don't assume. Don't hide confusion. Surface tradeoffs."
- State assumptions explicitly. If uncertain, ask.
- If multiple interpretations exist, present them — don't pick silently.
- If a simpler approach exists, say so. Push back when warranted.
- If something is unclear, stop. Name what's confusing. Ask.
Rule 2 — Simplicity First
"Minimum code that solves the problem. Nothing speculative."
- No features beyond what was asked.
- No abstractions for single-use code.
- No "flexibility" or "configurability" that wasn't requested.
- No error handling for impossible scenarios.
- If you write 200 lines and it could be 50, rewrite it.
- Test: "Would a senior engineer say this is overcomplicated?" If yes, simplify.
Rule 3 — Surgical Changes
"Touch only what you must. Clean up only your own mess."
- Don't "improve" adjacent code, comments, or formatting.
- Don't refactor things that aren't broken.
- Match existing style, even if you'd do it differently.
- If you notice unrelated dead code, mention it — don't delete it.
- Every changed line should trace directly to the user's request.
Rule 4 — Goal-Driven Execution
"Define success criteria. Loop until verified."
- Transform tasks into verifiable goals (e.g., "Add validation" → "Write tests for invalid inputs, then make them pass")
- For multi-step tasks, state a brief plan with verification at each step.
- Strong success criteria let you loop independently. Weak criteria require constant clarification.
UNVERIFIED. After exhaustive searching:
- Karpathy himself never cited these numbers
- The forrestchang repo contains no benchmarks or measurement data
- No independent researcher has published a controlled study
- The claim appears to originate from content creator @PrajwalTomar_ for engagement
- The one empirical paper (arXiv:2509.14744) analyzed 253 CLAUDE.md files for structure but did not measure error rates
Community validation is directionally positive — the "Surgical Changes" and "Simplicity First" rules get the strongest endorsements.
Source: Raw CLAUDE.md on GitHub
The tweet (Mnilax/status/2053116311132155938) links to an X Article locked behind JavaScript rendering. 22,440 bookmarks, 5,927 likes. The article is not indexed by search engines.
The claim trajectory: 41% baseline → 11% with Karpathy's 4 rules → 3% with all 12 rules across 30 codebases.
A publicly documented extension (GitHub Gist) adds 6 runtime rules to Karpathy's 4 edit-time rules:
Rule 5 — Deterministic First Reserve AI for judgment tasks (classification, drafting, summarization). Use deterministic code for routing, filtering, persisting, dispatching.
Rule 6 — Declare Budgets, Halt On Breach Every AI step runs under a token budget: per-step, per-pipeline, per-day. Exceeding any halts immediately.
Rule 7 — Human-In-The-Loop Is A First-Class Step Type Destructive actions (emails, CRM updates, messages) require explicit approval steps before execution.
Rule 8 — Validate AI Output Against A Schema Every AI step declares an output schema. Runtime rejects anything that doesn't match.
Rule 9 — Sanitize Operator Input Before It Reaches A Prompt Strip role markers and enforce length limits to prevent prompt injection.
Rule 10 — Log Rejections Silently Never echo rejection reasons back to users; prevents attackers from learning which patterns to attempt.
Include:
- Bash commands Claude can't guess
- Code style rules that differ from defaults
- Testing instructions and preferred test runners
- Repository etiquette (branch naming, PR conventions)
- Architectural decisions specific to your project
- Common gotchas or non-obvious behaviors
Exclude:
- Anything Claude can figure out by reading code
- Standard language conventions
- Detailed API docs (link instead)
- Information that changes frequently
- Self-evident practices like "write clean code"
Critical guidance:
"CLAUDE.md is advisory, meaning Claude follows it about 80% of the time." "If Claude keeps doing something despite a rule, the file is probably too long and the rule is getting lost." "Keep it concise. Bloated CLAUDE.md files cause Claude to ignore your actual instructions!"
- Files over 200 lines consume excess context and reduce adherence
- 150 is the ceiling for effective rules — beyond that, counterproductive
- Pragmatic target: 80% compliance via CLAUDE.md + hooks for remaining 20%
- Hard stops belong in
settings.json, linters, or pre-commit hooks — not CLAUDE.md - Grow incrementally: Add a line each time Claude makes a mistake a rule would prevent
Sources:
- HackerNoon: CLAUDE.md Done Right
- DEV: 200 Lines of Rules That Were Ignored
- Developers Digest: Use Rules as Menu, Not Template
| Level | Scope | What It Controls |
|---|---|---|
| Prompt Engineering | Message-level | Instructions to model |
| Context Engineering | Context-window-level | What model sees |
| Harness Engineering | System-level | Tools, memory, constraints, feedback loops, phase gates, orchestration |
Agent = Model + Harness. The wrapper around a fixed model can change performance by 6x on the same benchmark.
| Company/Experiment | Result | What Changed |
|---|---|---|
| Stripe "Minions" | 1,300 AI-generated PRs/week | Heavily modified Goose agent fork, isolated sandboxes |
| Hashline Edit Format | Grok Code Fast 1: 6.7% → 68.3% (10x) | Only changed the edit format (harness only) |
| LangChain Terminal Bench | gpt-5.2-codex: 52.8% → 66.5% (+13.7pts) | Only improved the harness |
| OpenAI Codex Internal | ~1,500 PRs merged, ~1M lines, 10x speed | Zero human-written code |
Key insight: Most agent failures are configuration problems, not model limitations. LLM compliance is probabilistic; harness constraints must be combined with deterministic outer-harness constraints (linters, CI gates).
Sources:
- MindStudio: What Is Harness Engineering
- Harness Engineering Deep Dive
- Martin Fowler: Harness Engineering
- Epsilla: Harness Engineering Evolution
| Model | Base Input/M | Cache Write 5min | Cache Write 1hr | Cache Read/M | Savings |
|---|---|---|---|---|---|
| Opus 4.7 | $5.00 | $6.25 (1.25x) | $10.00 (2x) | $0.50 (0.1x) | 90% |
| Sonnet 4.6 | $3.00 | $3.75 | $6.00 | $0.30 | 90% |
| Haiku 4.5 | $1.00 | $1.25 | $2.00 | $0.10 | 90% |
Hit rates in practice:
- Naive implementation: 7% hit rate
- After moving dynamic content out of cacheable prefix: 74%
- With explicit breakpoint placement: up to 84%
- Multi-turn agent loop with 10K-token system prompt: 5-10x cost reduction
Break-even: Cache pays for itself after 1 read (5min TTL) or 2 reads (1hr TTL).
Real-world: ProjectDiscovery cut LLM costs by 59% with prompt caching alone. Another team cut RCA costs by 90%.
Stores LLM response outputs keyed by semantically similar inputs. On cache hit, returns stored response without any LLM invocation — 100% savings on that call.
- Up to 86% cost reductions and 88% response time improvements
- 20ms lookup overhead vs 850ms LLM call = net 65% latency improvement at 67% hit rate
- Remote vector search needs 15-20% hit rates to offset 30ms lookup cost
- In-memory systems profitable at 3-5% hit rates
Risk: Threshold tuning is critical. Low thresholds risk returning incorrect cached responses.
Request → Semantic Cache (100% savings on hit)
→ Prefix Cache (50-90% savings on hit)
→ Full Inference
~31% of queries show semantic similarity, representing billions in wasted inference industry-wide.
| Model | Cache Hit Input/M | Cache Miss Input/M | Output/M |
|---|---|---|---|
| V4 Flash | $0.0028 | $0.14 | $0.28 |
| V4 Pro (promo) | $0.003625 | $0.435 | $0.87 |
Cache-hit input on V4 Flash is 50x cheaper than cache-miss.
Sources:
- Anthropic Prompt Caching Docs
- Anthropic Pricing
- ProjectDiscovery: 59% Cost Cut
- TrueFoundry: Semantic Caching
- Introl: Prompt Caching Infrastructure
During transformer inference, the model computes Key and Value tensors for each token. KV cache stores these so they don't need recomputation. Memory scales linearly with context length.
The problem: LLM inference systems waste 60-80% of allocated KV cache memory through fragmentation.
| Provider | Approach | Savings | Config |
|---|---|---|---|
| Anthropic | Explicit cache_control params |
90% on cache reads | Manual |
| OpenAI | Automatic caching | 50% | Automatic |
| DeepSeek V4 | Hybrid Attention (CSA 4x + HCA 128x) | KV cache at 10% of V3.2 | Automatic |
| Technique | Result |
|---|---|
| INT4 KV quantization | 4x size reduction |
| Low-rank + quantized caches | Up to 10x reduction, <5% perplexity increase |
| PagedAttention (vLLM) | Eliminates fragmentation |
| Cache offloading (GPU→CPU) | 14x faster TTFT for large inputs (NVIDIA) |
| KV cache-aware routing | 87% cache hit rate, 88% faster TTFT |
| LMCache + vLLM | Token read time 11s → 1.5s at 128K context |
| Entropy-guided eviction | Higher-entropy layers get larger budgets |
Sources:
- Why KV Cache Drives AI Economics
- KV Cache Optimization Guide
- vLLM DeepSeek V4 Analysis
- llm-d: KV Cache Wins
Small "draft" model generates candidate tokens, large "target" model verifies in parallel.
| Method | Speedup | Conditions |
|---|---|---|
| EAGLE-3 (paper) | 4.1-6.5x | Temperature 0, academic benchmarks |
| EAGLE-3 (production) | 2-3x | Off-the-shelf draft models, general queries |
| 70B-class models | 4-6x | Low batch sizes, matched training |
| DeepSeek V3 MTP | 1.8x | >80% acceptance rates |
| At 32+ concurrent batches | 1.05x | GPU memory-bound, speedup collapses |
Key constraint: 2-3x at low concurrency (1-10 requests), diminishing returns above that.
| Method | Speedup | Quality Impact |
|---|---|---|
| FP8 | ~2x memory reduction | Nearly indistinguishable from BF16 (2026 sweet spot) |
| AWQ INT4 | ~3x throughput vs BF16 | Stable accuracy even at 4-bit |
| GPTQ INT4 | ~3x throughput vs BF16 | Significant accuracy loss on small models |
| Marlin-AWQ kernels | 741 tok/s (10.9x vs standard AWQ) | — |
| Q5_K_M / Q8_0 | Substantial | 95-99% of original performance |
AWQ > GPTQ consistently. AWQ identifies ~1% salient weight channels and scales them before quantization.
Effects are multiplicative. NVIDIA H200: 3.6x throughput combining FP8 + speculative decoding on Llama 3.1-405B.
When to use each:
- Quantization: Always. FP8 is virtually free quality-wise. Start here.
- Speculative decoding: Low-concurrency, latency-sensitive (1-10 concurrent). Not worth it at high batch.
- Both: Maximum single-request speed with memory savings.
Sources:
- Speculative Decoding 2-3x Faster (2026)
- E2E: EAGLE-3 Guide
- vLLM Quantization Guide
- VRLA Tech: Quantization Explained
| Method | Success Rate | Notes |
|---|---|---|
| Prompt-only JSON | 80-95% | Fails 5-20% in production |
| Function Calling / Tool Use | 95-99% | Schema is a hint, not constraint |
| Constrained Decoding (XGrammar/Outlines) | ~100% | Grammar-enforced guarantee |
| Engine | Success Rate | Hallucination Rate | Speed |
|---|---|---|---|
| Outlines (one-shot) | ~93% | 1.8% | Good |
| XGrammar (one-shot) | 60-78% | 10.7% | <40 microseconds/token |
XGrammar is the default in vLLM, SGLang, TensorRT-LLM as of March 2026. Up to 100x speedup over traditional grammar methods.
1. Try constrained decoding (XGrammar/Outlines)
2. If unsupported schema → fall back to JSON mode
3. If parse failure → retry with error context + compact schema example
4. 3 retries sufficient (most models self-correct on second attempt)
5. If all fail → return raw text + error flag for human review
Sources:
- GPT-4-class judges: 80% agreement with human preference scores
- This matches human-to-human inter-annotator agreement (~80%)
- GPT-5-class judges: 80-90% agreement on most quality dimensions
- Fine-tuned BERT models outperform LLM judges on in-domain sentence-level tasks (ACL 2025)
| Method | Cost per 10K evals | Notes |
|---|---|---|
| Human review | $50,000 - $100,000 | Expert annotators |
| LLM-as-judge | $100 - $200 | 500x-5000x cheaper |
| Fine-tuned specialist | Initial $2K-5K training | ROI in 3-6 months at volume |
| Framework | Free Tier | Best For |
|---|---|---|
| Braintrust | 1M trace spans + 10K scores | CI/CD eval integration, quality gates |
| LangSmith | Limited | LangChain teams |
| DeepEval | Open-source | pytest integration, agent-aware |
| Deepchecks | — | Ongoing reliability measurement |
Best Practices (from Anthropic Engineering)
- Start with 20-50 tasks from real production failures
- Use pass@k (optimistic) and pass^k (consistency) metrics
- Graduate capability evals into regression suites as they saturate
- Validate LLM scorer against human judgments on a calibration set
- Use chain-of-thought in scorer prompts
- "Like Swiss Cheese — no single eval layer catches every issue"
Sources:
Tag every API request with metadata: user_id, feature_name, team. Map spend to operation names.
| Tool | Type | Free Tier | Key Feature |
|---|---|---|---|
| LiteLLM | Open-source proxy | Yes | Per-key/user/team spend, tag budgets |
| Langfuse | Open-source (MIT) | 50K events/mo | Trace-level cost per step |
| Braintrust | SaaS | 1M trace spans | Custom tag grouping |
| Helicone | Proxy-based | 10K req/mo | 2-minute setup, built-in caching |
| Bifrost | Open-source gateway | Yes | Multi-step agent cost breakdown |
# LiteLLM example
response = litellm.completion(
model="claude-sonnet-4-6",
messages=[...],
extra_body={"metadata": {"feature": "search", "team": "core"}}
)Sources:
- 57% of organizations have agents in production
- 90% of agent projects fail within 30 days — runaway costs are #1 pain point
- Claude Code recursion loop: 1.67 billion tokens in 5 hours (~$16K-50K)
- LangChain agent: $47,000 in 11 days
| Layer | Purpose | Example |
|---|---|---|
| Per-request ceiling | Hard stop per call | MAX_TOOL_CALLS=12 |
| Token-cost circuit breaker | Kill if 2x budget | Stops runaway inference |
| Progress detection | Detect loops | Same tool + same args twice = stop |
| Session budget | Total conversation cap | max_total_cost_usd |
| Spend rate monitor | Rolling window | tokens_per_task_limit: 30000 |
"Budget enforcement must live OUTSIDE the agent code. If a gateway enforces the budget before forwarding, the agent literally cannot make a violating call."
maxConsecutiveNoProgress: 5
maxConsecutiveFailures: 3
tokenVelocityMultiplier: 3.0 # trip if >3x rolling average
max_tool_calls: 12
token_budget_multiplier: 2.0- TokenFence: Wrap OpenAI/Anthropic clients in 2 lines. Budget caps, auto model downgrades, instant loop stoppage.
- Google's guidance: "The most reliable guardrails are implemented in code rather than prompts."
Sources:
- AI Security Gateway: Token Budget Strategies
- CodirsHub: Prevent Agent Loop Costs
- TokenFence
- Google: 5 Lessons from Production Agents
Minimum viable: Cost, latency, error rates. Add quality metrics once baseline exists.
| Tool | Type | Latency Overhead | Free Tier | Strength |
|---|---|---|---|---|
| Langfuse | SDK (MIT) | Zero | 50K events/mo | Full tracing + prompt mgmt + evals |
| Helicone | Proxy (Apache 2.0) | 50-80ms | 10K req/mo | Fastest setup, caching saves 20-40% |
| Braintrust | SDK | Minimal | 1M spans | CI/CD eval integration |
| LangSmith | SDK | Minimal | Limited | Deep LangChain integration |
| Phoenix (Arize) | SDK (OSS) | Minimal | Self-host | Notebook-friendly, trace viz |
Per request: prompt sent, model response, token usage (input/output/cache), latency (TTFT, total), tool calls, cost tags.
Across requests: cost per feature, error rates by model, latency percentiles (p50/p95/p99), cache hit rates, quality scores.
Start with Langfuse (self-host free, full features) or Helicone (proxy, 2-minute setup).
Sources:
| Framework | Cost Savings | Quality Retention | Notes |
|---|---|---|---|
| RouteLLM (UC Berkeley, ICLR 2025) | 85% | 95% of GPT-4 | 26% calls routed to expensive model |
| Martian ($1.3B valuation) | 20-97% | Often beats GPT-4 | Predicts model behavior without running it |
| OpenRouter | Variable | Variable | 300+ models, auto provider fallback |
| Morph Router | Significant | — | Prompt difficulty classification |
| LiteLLM | Variable | — | Open-source proxy with routing |
- Matrix factorization router: 95% GPT-4 performance with only 26% GPT-4 calls (48% cheaper)
- With augmented training: only 14% expensive calls (75% cheaper)
- 40% cheaper than commercial routers (Martian, Unify AI) at comparable quality
- Generalizes to model pairs not in training data
Product search → Gemini Flash (speed)
Customer complaints → Claude Sonnet (tone/empathy)
Fraud analysis → GPT-4o (multi-step reasoning)
Sources:
| Factor | Use Few-Shot / ICL | Use Fine-Tuning |
|---|---|---|
| Data volume | <100 examples | >500 examples (ideally >2K) |
| Domain | General | Highly specialized / proprietary |
| Output format | Flexible | Strict, consistent format |
| Request volume | Low | High (amortize training cost) |
| Iteration speed | Fast (change prompts) | Slow (retrain) |
| Reversibility | Fully reversible | Not reversible |
ICL Hidden Cost
Processing k training examples increases compute by approximately (k+1)x vs the query alone. With 10 few-shot examples, you're paying 11x the compute of a zero-shot call.
- Barnett et al. (2024): Fine-tuning reduced accuracy in retrieval-augmented tasks
- Can overfit to training distribution
- Locks you into retraining when base model shifts
Try prompt engineering first. If few-shot with 5-10 examples doesn't hit your bar after iterations, and you have >500 labeled examples, fine-tune a smaller model (GPT-4.1 Mini, Llama 70B via QLoRA).
Sources:
- 1.6 trillion total parameters, 49 billion active (MoE)
- 1 million token context window
- Up to 384K tokens max output per request
- Pre-trained on 32+ trillion tokens
- KV cache at 10% of V3.2; inference FLOPs at 27%
| Benchmark | V4 Pro | Claude Opus 4.6 | GPT-5.4 |
|---|---|---|---|
| SWE-bench Verified | 80.6% | 80.8% (+0.2) | — |
| Codeforces Rating | 3,206 (highest) | — | 3,168 |
| LiveCodeBench | 93.5% | — | — |
| Terminal-Bench 2.0 | 67.9% | 65.4% | — |
| Token Type | Promo Price/M | List Price/M | Savings |
|---|---|---|---|
| Cache Hit Input | $0.003625 | $0.0145 | 75% |
| Cache Miss Input | $0.435 | $1.74 | 75% |
| Output | $0.87 | $3.48 | 75% |
vs Competitors during promo:
- ~7x cheaper than GPT-5.5 on input
- ~6x cheaper than Claude Opus 4.7 on input/output
- ~11.5x cheaper on output than frontier models
| Model | Input/M | Output/M | SWE-bench |
|---|---|---|---|
| V4 Pro (promo) | $0.435 | $0.87 | 80.6% |
| V4 Flash | $0.14 | $0.28 | ~79.0% |
Flash trails Pro by only 1.6 SWE-bench points but costs 25x less per output token. At $0.14/M input, Flash is roughly 35-100x cheaper than GPT-5.5 or Claude Opus 4.7.
Strategy 1: Batch cache-heavy workloads NOW. Cache-hit input at $0.003625/M is essentially free. Any workload with repetitive system prompts should be front-loaded.
Strategy 2: Build and benchmark V4 Pro into your pipeline. Use discount to run eval suite. If it passes at 80%+ frontier quality, you've identified a permanent cost-reduction tier even at list price.
Strategy 3: Two-tier DeepSeek pipeline. V4 Flash as default sub-agent ($0.28/M output). Escalate to V4 Pro ($0.87/M) only when Flash fails.
Strategy 4: Stress-test the 1M context window.
Use deepseek-v4-pro[1m] model designation. Processing 1M tokens input costs $0.435 at promo rate vs ~$15 for Claude Opus.
Strategy 5: Build evals during the discount. Eval runs cost 75% less too. Build your 20-50 task eval suite, run hundreds of trials, establish baseline metrics.
- 24% timeout rate on difficult reasoning tasks
- SimpleQA-Verified: 57.9% (vs Gemini 75.6% — factual retrieval gap)
- Instruction following on complex multi-constraint prompts lags frontier
- Promo may not extend beyond May 31
Sources:
- DeepSeek 75% Price Cut (The Next Web)
- DeepSeek V4 Pro Review & Benchmarks
- DeepSeek Official Pricing
- DeepSeek V4 Promo Tweet
- Simon Willison on DeepSeek V4
"Software, research, education, infrastructure, and knowledge work are all becoming variations of the same pattern: define the context, define the tools, define the feedback loop, define the guardrails, let agents work, preserve human understanding."
| Framework | Strength | Setup Time | Token Overhead |
|---|---|---|---|
| LangGraph | Graph viz, time-travel debug, audit trails | Medium | Low |
| CrewAI | Fastest prototyping, role-based crews | 2-4 hours | Up to 3x on simple tasks |
| AutoGen/AG2 | Multi-agent debate, iterative refinement | Medium | Medium |
| Claude Agent SDK | Safety, computer use, MCP protocol | Medium | Low |
| Google ADK | Google ecosystem integration | Medium | Medium |
LangGraph surpassed CrewAI in GitHub stars in early 2026.
Subagent Patterns (from Philipp Schmid)
| Pattern | Control | When to Use |
|---|---|---|
| Inline Tool | Minimal | Most tasks. Start here. |
| Fan-Out | Moderate | Parallel independent work |
| Agent Pool | High | Multi-turn state tracking |
| Teams | Minimal | Persistent collaboration |
"Start with Pattern 1. Most tasks that feel like they need multi-agent work fine with a well-prompted inline tool call."
Three-layer architecture:
Conversations → Daily Logs (raw transcripts)
Daily Logs → Wiki (compiled knowledge base)
Wiki → Next Session (context injection)
Creates a compounding brain — knowledge compiles once, then updates. Architecture mirrors an OS: main context is RAM, archival memory is disk.
Implementation lesson: Replace complex background automation with one 30-line script + manual /close-day invocation. Date-tagged entries with grep beat complex merge logic.
| Company | System | Result |
|---|---|---|
| Wells Fargo | Orchestrated agent delegation | 35K bankers access 1,700 procedures in 30s (was 10min) |
| HCLTech | Dynamic agent handoff | 40% faster case resolution |
| Stripe | Minion agents | 1,300 PRs/week, zero human-written |
| Claude Code | Agent Teams (experimental) | 3-5 instances, peer-to-peer messaging |
Sources:
- Subagent Patterns 2026
- Addy Osmani: The Code Agent Orchestra
- Karpathy LLM Wiki Concept
- Multi-Agent Orchestration Patterns
Recommended pipeline:
User Request
│
▼
┌────────────────┐
│ RouteLLM Router│ (classify complexity)
└────────┬───────┘
│
┌────┴────┐
│ │
▼ ▼
┌────────┐ ┌──────────┐
│DeepSeek│ │Claude/GPT│ (frontier for complex)
│V4 Flash│ │Opus/5 │
│$0.28/M │ │$10-25/M │
└────────┘ └──────────┘
RouteLLM results: 95% GPT-4 performance with 26% of calls routed to expensive model = 48% savings. With augmented training: only 14% expensive calls = 75% savings.
| Layer | Savings | Effort |
|---|---|---|
| Model routing (cheap → expensive) | 48-85% | Medium (RouteLLM setup) |
| Prompt caching | 50-90% | Low (restructure prompts) |
| Semantic caching | 86% on hits | Medium (vector DB) |
| Quantization (self-hosted) | 50-75% memory | Low (FP8 is free quality-wise) |
| Feature-level cost tracking | 20-40% (optimization target) | Low (Helicone/LiteLLM) |
Request → Semantic Cache Check ($0)
→ Model Router (cheap model $0.28/M vs expensive $10/M)
→ Prompt Cache (90% input savings)
→ Full Inference (last resort)
Immediate action items:
- Sign up for DeepSeek API (5M free tokens)
- Build 20-50 task eval suite (costs 75% less during promo)
- Run benchmark suite: V4 Flash → V4 Pro → Claude Opus/GPT-5
- Identify tier boundaries: which tasks Flash handles, which need Pro, which need frontier
- Front-load cache-heavy workloads before May 31
- Build multi-model pipeline with routing logic
- Set up cost attribution (Helicone or LiteLLM, 2 minutes)
Expected combined savings: 70-90% vs naive single-frontier-model approach.
Sources:
- Intelligent LLM Routing: 85% Cost Cuts
- ngrok: Prompt Caching 10x Cheaper
- Don't Break the Cache (arxiv)
- Braintrust Agent Observability
The entire agentic engineering discipline reduces to Karpathy's meta-pattern:
1. Define the context → Context engineering: what goes in the window
2. Define the tools → Tool routing: what the agent can do
3. Define the feedback → Evals: how you know it worked
4. Define the guardrails → Permissions, security, quality gates
5. Let agents work → Orchestrator-subagent delegation
6. Preserve understanding → "You can outsource thinking, not understanding"
From generation to verification.
Agents produce at incredible speed. Knowing with confidence whether output is correct is the hard part. That is where the 100x engineer lives — not in writing more code, but in building the harness that makes agent output trustworthy at scale.
| Multiplier | Impact | How |
|---|---|---|
| Harness engineering | 6-10x performance on same model | Edit formats, tool routing, phase gates |
| Model routing | 75-85% cost reduction | Cheap model default, expensive escalation |
| Eval-driven development | Prevents quality regression | 20-50 tasks, pass@k metrics, CI integration |
An agentic engineer who combines:
- CLAUDE.md rules (directionally reduce errors)
- Harness engineering (6x performance multiplier)
- Model routing (75% cost reduction)
- Prompt caching (90% input savings)
- Eval loops (quality maintenance)
- Cost attribution (optimization targeting)
- Loop budgets (prevents $47K incidents)
...operates at a fundamentally different level than someone writing prompts in a chat box.
That's the 100x.
Report compiled from 150+ web searches across academic papers, production case studies, official documentation, and community discussions. All claims are sourced. Unverified claims are explicitly flagged.