A curated list of strategies, tools, papers, and resources for reducing LLM token costs and improving efficiency in production.
Building with LLMs is expensive. An agent processing 10 reasoning steps can consume 50K-100K tokens per task. This list collects everything you need to cut costs by 80-99% without sacrificing quality.
- Quick Wins
- Prompt Caching
- Batch APIs
- Model Routing
- Prompt Compression
- Context Window Management
- KV Cache Optimization
- Browser Tool Efficiency
- Cost Tracking Tools
- Pricing Comparison
- Prompt Engineering for Efficiency
- Comprehensive Guides
- Academic Papers
- Community Resources
The highest-impact strategies ranked by effort-to-savings ratio:
| Strategy | Savings | Effort | Link |
|---|---|---|---|
| Prompt caching | 90% input tokens | Add cache headers | Prompt Caching |
| Token-efficient tool use | 70% output reduction | Flip a flag | Prompt Engineering for Efficiency |
| Batch API | 50% | Queue non-urgent work | Batch APIs |
| Model routing | 60-95% | Route by task complexity | Model Routing |
| Response caching | 100% on repeats | Add a cache layer | Comprehensive Guides |
| Prompt compression | 5-20x | Use LLMLingua | Prompt Compression |
Combined pipeline: Cache prefix (90%) + route to cheapest model (60-95%) + batch non-urgent (50%) + compress prompts (5-20x) + cache responses (100% on repeats) = 95-99% cost reduction vs. naive approach.
Reuse previously-processed prompt prefixes to avoid re-computing the same tokens.
- Anthropic Prompt Caching - 90% discount, 5min/1hr TTL. Minimum cacheable prefix: 4,096 tokens on Opus 4.6/Haiku 4.5, 1,024 on Sonnet 4.6/Opus 4.8/Sonnet 5.
- Anthropic Caching Announcement - Blog post explaining economics.
- Anthropic Token-Saving Updates - Cache-aware rate limits, simplified caching.
- Anthropic Extended Thinking + Caching - Thinking blocks get cached in tool-use loops.
- OpenAI Prompt Caching - 50% discount, automatic for 1024+ token prompts. Extended cache retention now defaults to 24h on the GPT-5 series (mandatory on GPT-5.5+), keeping prefixes warm far longer.
- OpenAI Prompt Caching Cookbook - Advanced techniques with code.
- Google Gemini Context Caching - Implicit (auto) and explicit caching, 90% discount.
- Google Vertex AI Caching - Enterprise context caching.
- DeepSeek KV Cache - Disk-based, 64-token granularity. V4 Flash cache hits: $0.0028/M vs $0.14/M base (98% savings). Migration:
deepseek-chatanddeepseek-reasoneraliases retire July 24, 2026 — update todeepseek-v4-flashordeepseek-v4-pro. - DeepSeek Context Caching on Disk - Announcement of disk-based context caching cutting input cost ~10x on cache hits.
Structure prompts so the system prompt + user profile is the first ~2,000 tokens. All subsequent calls share this prefix. For bulk operations (e.g., scoring 50 items): 1x full + 49x at 10% = 88% total savings.
- autocache - Transparent Anthropic proxy that auto-injects
cache_controlbreakpoints at optimal positions; up to 90% cost and 85% latency reduction.
50% discounts for non-time-critical requests. Combine with caching for 95% savings.
- Anthropic Message Batches - Up to 10,000 requests, 24hr turnaround.
- Anthropic Batches Announcement - Use cases and GA details.
- OpenAI Batch API - 50% discount, 50K requests per file.
- OpenAI Batch API FAQ - Limits and behavior.
- Google Gemini Batch API - 50% discount, combinable with context caching.
- Google Vertex Batch Prediction - Enterprise batch.
- Curator - Batch-inference and synthetic-data library;
batch=Truegives ~50% savings across OpenAI/Anthropic/Gemini/Mistral with built-in caching and retries.
Route simple tasks to cheaper models. 80% of typical LLM calls don't need the most expensive model.
- RouteLLM - Open-source LLM router by LMSYS. Trains routers from preference data; 2x+ cost reduction. Note: last commit Aug 2024; LMSYS team shifted focus to Chatbot Arena.
- LiteLLM - SDK + proxy for 100+ LLMs with routing, cost tracking. Strategies: least-busy, cost-based, latency-based.
- NotDiamond - Per-query best-model selection. Note: the Python SDK was archived Dec 2025 (read-only); the notdiamond.ai service remains active.
- Bifrost - 50x faster than LiteLLM; adaptive load balancer, 1000+ models.
- vLLM Semantic Router - System-level signal-driven router for Mixture-of-Models across cloud, data center, and edge. v0.3 "Themis" (June 2026); SAAR adds session-aware model selection cutting model switches 79% in multi-agent deployments.
- LLMRouter - Open-source routing library with 16+ routers (single-round, multi-turn, agentic, personalized) and a unified CLI.
- Portkey AI Gateway - Open-source AI gateway routing to 1,600+ LLMs with guardrails, caching, and load balancing. Acquired by Palo Alto Networks (May 2026); gateway remains open-source under Apache 2.0.
- OpenRouter - Unified API for 300+ models with auto-router.
- Martian Router - Patent-pending; cuts costs 20-97% via "Model Mapping".
- Awesome AI Model Routing - Comprehensive list of routing approaches.
- RouteLLM paper - LMSYS blog on cost-quality tradeoffs.
- IBM LLM Routers - IBM's research on training routers.
- vLLM Session-Aware Agentic Routing (SAAR) - Router-owned session memory with hard locks around tool loops cuts model switches 79% and estimated cost 78.7% in multi-agent deployments.
- LLM Routing Explained - Intuitive guide.
- vLLM Semantic Router paper - Signal-driven decision routing for Mixture-of-Modality models; composable signal orchestration across heuristic and neural classifiers.
Reduce prompt size while preserving information quality.
- LLMLingua - Up to 20x compression. Coarse-to-fine iterative method. Integrates with LangChain/LlamaIndex.
- Headroom - Compress tool outputs, logs, files, and RAG chunks before they reach the LLM (60-95% fewer tokens); library, proxy, and MCP server. Claude Code/Cursor/Aider compatible.
- code2prompt - Codebase to LLM prompt with token counting.
- RTK - Single-binary Rust CLI proxy that compresses dev-command output 60-90% before it reaches a coding agent's context. Works with Claude Code, Cursor, Copilot, Gemini CLI.
- TOON - Token-Oriented Object Notation: a compact, schema-aware encoding for passing JSON-like data to LLMs; 30-60% fewer tokens than JSON on uniform arrays of objects.
- llmtrim - Quality-gated local proxy and MCP server that compresses prompts, tool outputs, and replies before they reach the LLM, reverting any step that doesn't save tokens (project-reported -31% input / -74% output across 112 A/B cases). Rust CLI plus multi-language library bindings and a WebAssembly/JS package.
- lean-ctx - Rust binary context intelligence layer for AI coding agents; 60-90% fewer tokens via shell-output compression and 10 cached-read modes; MCP server with 76 tools and cross-session memory. Works with Claude Code, Cursor, Copilot, Windsurf, Gemini CLI, and 30+ others.
- CompactPrompt - Unified prompt + data compression pipeline.
- Efficient Prompting Survey - Survey of efficient prompting methods.
The full prompt-compression paper table lives in the Academic Papers section below.
- LLMLingua Research Blog - Microsoft Research deep dive.
- Prompt Compression Tutorial (FreeCodeCamp) - Practical guide with code.
- Prompt Compression Overview (MLM) - 6x to 480x compression ratios.
- Awesome LLM Compression - Curated paper list.
Rule-based lossless distillation achieves 3-4:1 compression without any model:
Strip prose transitions, hedging, rhetoric, and common knowledge. Preserve numbers, entities, decisions, constraints, and risks. Transform prose into dense bullets and verbose text into semicolon-joined clauses. Split into 3,000-5,000 token self-contained sections that load independently.
- Context Rot toolkit (Chroma) - Toolkit reproducing how LLMs degrade well before context limits (18 models tested).
- RAG vs Long Context (Elastic) - RAG is 1250x cheaper for many queries.
- Long Context RAG (Databricks) - Degradation after 32K-64K tokens.
- Context Extension Survey - All context extension techniques surveyed.
- Anthropic Long Context Tips - Place docs at top, use XML tags.
- Anthropic Context Windows - How context works, server-side compaction.
- Anthropic Context Engineering - Finding the smallest high-signal token set.
- Anthropic Long-Running Agents - Managing context across extended workflows.
- Pinecone Chunking Guide - Fixed-length, semantic, hierarchical.
- Advanced Chunking (Galileo) - Agentic and LLM-based.
- Context Engineering Guide - Curated papers and tools.
- Efficient Context Management (JetBrains) - Observation masking vs summarization.
- OpenProvence - Open reranker-pruner that drops ~99% off-topic sentences and compresses 80-90% of relevant RAG text; CPU-friendly.
Server-side optimizations for inference efficiency.
- vLLM - PagedAttention, high-throughput inference.
- SGLang - RadixAttention for automatic KV cache reuse.
- GPUStack - GPU cluster manager for vLLM/SGLang.
- NVIDIA Dynamo - Datacenter-scale distributed inference with KV-cache-aware routing and disaggregated prefill/decode; ~2x faster TTFT, 7x throughput/GPU.
- llm-d - Kubernetes-native distributed serving with prefix-cache-aware routing and tiered KV offload to CPU/disk (3x output throughput).
- Mooncake - Distributed KVCache engine (the serving platform behind Moonshot AI's Kimi); integrated into vLLM for high-bandwidth KV-cache transfer and cross-instance prefix reuse across disaggregated prefill/decode.
- NVIDIA kvpress - KV cache compression made easy.
- R-KV - Redundancy-aware compression (NeurIPS 2025).
- llm-compressor - Compression for deployment with vLLM.
- NVIDIA Model Optimizer - Quantization, pruning, distillation, speculative decoding.
- TurboQuant - Google's ICLR 2026; 5x KV cache compression.
- aibrix - Cost-efficient infrastructure for GenAI inference.
- LMCache - KV cache layer for vLLM/SGLang; offloads and reuses caches across engines (CPU/disk/S3) to cut TTFT for long-context, multi-turn, and RAG.
- kvcached - Virtualized elastic KV cache decoupling virtual/physical GPU memory for dynamic GPU sharing; plugs into vLLM/SGLang.
- KVzip - Query-agnostic KV cache eviction via context reconstruction (NeurIPS 2025 Oral); 3-4x memory reduction, 2x lower latency.
- DeepSpec - DeepSeek's open-source speculative decoding stack powering DSpark; 60-85% faster on V4-Flash, 57-78% on V4-Pro; outperforms Eagle-3. MIT license.
- Speculative Sampling - Fast inference via speculative decoding.
- Awesome KV Cache Compression - Must-read paper list.
The full KV-cache paper table lives in the Academic Papers section below.
- mini-sglang - Learn LLM serving internals.
- tiny-llm - Build a tiny vLLM on Apple Silicon.
Different browser automation approaches consume vastly different context.
| Agent | Output Size | Efficiency | Link |
|---|---|---|---|
| WebFetch | ~1.5 KB (AI-summarized) | 20x better | Docs |
| Playwright MCP | ~10-33 KB (accessibility tree) | Baseline | GitHub |
| Agent Browser |
~28 KB (accessibility tree) | Project unmaintained 2026-05 — superseded by browser-use direct mode + Playwright MCP | GitHub |
| Lightpanda | ~16 KB (raw markdown) | 2x better | GitHub |
For 10-page workflows: WebFetch = ~15KB vs Playwright = ~330KB total context consumed.
The accessibility tree strips visual styling to retain only semantic structure (name, role, state, value). 10-50x smaller than raw HTML. See: Token cost analysis in browser MCPs.
- WebFetch vs WebSearch analysis - Deep comparison.
- browser-use - Foundation library for AI browser agents.
- Chrome full accessibility tree - DevTools feature.
- mcp-compressor - MCP proxy that shows a compressed tool surface first and fetches full schemas on demand, cutting tokens spent on large MCP tool descriptions.
- Langfuse - Open-source LLM observability + cost tracking. Cost tracking docs. Acquired by ClickHouse (Jan 2026); still actively developed, MIT-licensed.
- Helicone - LLM observability, 300+ models, SOC 2. Cost tracking cookbook. Acquired by Mintlify (Mar 2026); now maintenance-only — security/bug fixes and new-model support continue, no new feature work.
- LiteLLM Spend Tracking - Per-key/team spend tracking and budget routing for the LiteLLM proxy across 100+ LLMs.
- tokencost - USD cost estimates for 400+ LLMs.
- AgentOps - Agent monitoring with LLM cost tracking.
- agenttrace - Local-first TUI that reads Claude Code / Codex / Gemini / Aider / Cursor sessions to surface tokens, cost, cache use, retries, and latency.
- Future AGI traceAI - OpenTelemetry-based AI tracing capturing per-call tokens, cost, and latency across 35+ frameworks.
- ccusage - Fast local CLI reporting tokens and cost across 14+ coding agents (Claude Code, Codex, Gemini CLI, Copilot); offline, no upload.
- OpenLLMetry - OpenTelemetry-based GenAI observability instrumenting LLM and vector-DB calls with per-call token and latency telemetry.
- MLflow - Open-source AI/ML platform with GenAI observability in MLflow 3.x: LLM call tracing with per-span token tracking, prompt optimization tooling, and AI Gateway integration for cost control.
- Helicone AI Gateway - Fastest open-source AI gateway (Rust).
- Anthropic Token Counter - Free pre-flight token counting endpoint.
- tiktoken - OpenAI's fast BPE tokenizer (Python/Rust), 3-6x faster.
- LangSmith Cost Tracking - Automatic recording with dashboards.
- LlamaIndex Cost Analysis - Estimate costs before calls.
- Price Per Token - Daily-updated, 300+ models.
- Artificial Analysis Calculator - Free calculator, 100+ models.
- Artificial Analysis Leaderboard - Quality + price + speed.
- Simon Willison's LLM Prices - Interactive calculator.
- Helicone LLM Cost Comparison - 300+ model calculator.
- CostGoat - 302+ APIs from 10+ providers.
- Langtail - Side-by-side comparison.
- WhatLLM - 256 models, 43+ providers, weekly updates.
- Anthropic Pricing - Official Claude model pricing.
- OpenAI Pricing - Official OpenAI API pricing.
- Google Gemini Pricing - Official Gemini API pricing.
- DeepSeek Pricing - Official DeepSeek pricing.
- Mistral Pricing - Official Mistral pricing.
| Model | Input /MTok | Output /MTok | Notes |
|---|---|---|---|
| Claude Fable 5 | $10.00 | $50.00 | Anthropic's most capable model; 1M context (June 2026). Access suspended June 12 via US export-control directive; restored July 1, 2026. |
| Claude Opus 4.8 | $5.00 | $25.00 | 1M context at standard pricing |
| Claude Sonnet 5 | $2.00 | $10.00 | Introductory pricing through Aug 31, 2026 (standard: $3/$15 per MTok); 1M context; most agentic Sonnet; launched June 30, 2026. |
| GPT-5.5 | $5.00 | $30.00 | OpenAI flagship; 1M context; 90% cached-input discount |
| GPT-5.4 | $2.50 | $15.00 | Half the cost of GPT-5.5; 50% Batch API discount |
| DeepSeek V4 Flash | $0.14 | $0.28 | Cheapest frontier; 98% cache savings |
| DeepSeek V4 Pro | $0.435 | $0.87 | 1M context; thinking + non-thinking modes |
| Gemini 3.1 Pro | $2.00 | $12.00 | Preview since Feb 2026; ≤200K context; doubles to $4/$18 above 200K tokens |
| Gemini 3.5 Flash | $1.50 | $9.00 | Launched May 19, 2026; 1M context window |
| Gemini 2.5 Flash-Lite | $0.10 | $0.40 | Budget option |
Tokenizer note (Anthropic): Claude Opus 4.7+, Sonnet 5, and Fable 5 use a newer tokenizer that produces roughly 30% more tokens for the same text; per-token prices are unchanged, so the effective cost of a fixed input rises proportionally (Sonnet 4.6 and earlier keep the previous tokenizer). Benchmark your real workload before assuming a newer model lowers cost — confirmed on Anthropic's official pricing docs (already linked under Provider Pricing Pages above).
- Anthropic Prompt Engineering - Master guide.
- Anthropic Claude 4 Best Practices - Model-specific.
- Anthropic Interactive Tutorial - 9-chapter course.
- Anthropic Tool Search - 85% token reduction for large tool libraries.
- OpenAI Prompt Engineering - Strategies and tactics.
- OpenAI Cost Optimization - Input minimization, model selection, caching.
- OpenAI Optimization Cookbook - Collection of notebooks.
- Token-Efficient Tool Use (Anthropic) - 70% output token reduction.
- PromptingGuide: Optimizing - Compression, abstraction, filtering.
- Prompt Bloat Impact (MLOps) - Quality degrades with bloat.
- Token Complexity - Each task has intrinsic minimum tokens for success.
- Verbosity != Veracity - Demystifying verbosity in LLM outputs.
- Incorporating Token Usage - Token usage as prompting strategy metric.
- 8 Strategies to Cut API Spend 80% (2026)
- Redis Token Optimization - Semantic caching, ~73% cost reduction.
- How I Reduced Token Costs by 90%
- LLM Token Optimization Strategies
- Monitor and Cut LLM Costs 90% (Helicone)
- LLM Caching Strategies (CostLens) - "90% savings most developers don't know about."
- AI Agent Cost Optimization - 60-70% of agent calls suit small models.
- Practical Cost + Latency Reduction
- Vantage LLM Cost Guide - Enterprise monitoring.
- Semantic Highlight for RAG (Zilliz) - 70-80% token reduction.
- Optimizing LLM in Production (Hugging Face) - Quantization, Flash Attention.
- Hugging Face Inference Optimization - Transformers library.
- Epoch AI: LLM Inference Price Trends - Data showing inference cost per fixed performance halving roughly every two months.
- ProjectDiscovery: Cut LLM Costs 59% With Caching - Raised cache hit rate from 7% to 84% across 9.8B cached tokens.
- Cockroach Labs: Agentic AI Costs at Scale - Re-sent context can be 62% of agent inference bills; mitigations.
- Together AI: Serving DeepSeek-V4 Long Context - Compressed KV layouts and eviction expanding single-node KV capacity 1.2M to 3.7M tokens.
- GitHub: Token Efficiency in Agentic Workflows - Pruning unused MCP tool schemas and swapping MCP calls for the
ghCLI cut agentic-CI token spend 43-62% across real workflows.
| Paper | Year | Key Result |
|---|---|---|
| Prompt Compression Survey | 2024 | Comprehensive survey of all techniques |
| LLMLingua | 2023 | Up to 20x compression (EMNLP) |
| LLMLingua-2 | 2024 | 3-6x faster via BERT distillation (ACL) |
| LongLLMLingua | 2023 | 4x fewer tokens in long contexts |
| Selective Context | 2023 | 50% reduction via self-information pruning |
| RECOMP | 2023 | 5% token ratio for retrieved docs |
| 500xCompressor | 2024 | 6-480x compression ratios |
| LoPace | 2026 | Lossless; 72.2% savings |
| SCOPE | 2025 | Training-free generative rewriting |
| Dynamic Compressing | 2025 | MDP-based adaptive token removal |
| Empirical Study | 2025 | Benchmarks 6 methods across 13 datasets |
| Telegraph English | 2026 | Symbolic rewriting protocol; ~50% token reduction at 99.1% accuracy; outperforms LLMLingua-2 at matched ratio |
| Prompt Compression in the Wild | 2026 | First large-scale production study (30K queries) of the latency vs. quality tradeoff |
| Production Compression RCT | 2026 | Pre-registered randomized trial: moderate compression −27.9% cost; over-compression backfires |
| LongCodeZip | 2025 | Code-aware two-stage compression; up to 5.6x with no performance loss (ASE 2025) |
| Behavior-Equivalent Token | 2025 | Distills a long system prompt into one learned token; no aux model or labels |
| SAC (Semantic Anchors) | 2025 | Autoencoding-free context compression via selected anchor tokens; no compression-token pretraining |
| Paper | Year | Key Result |
|---|---|---|
| FrugalGPT | 2023 | Seminal cascade paper; up to 98% cost reduction |
| RouteLLM | 2024 | 2x+ cost reduction without quality loss |
| Hybrid LLM | 2024 | 40% fewer calls to large model |
| Unified Routing + Cascading | 2024 | +14% over individual strategies |
| Dynamic Routing Survey | 2026 | Comprehensive survey |
| Pay for Hints | 2026 | Small model gets hints, not full answers |
| RouteProfile | 2026 | Graph-based profiling for cold-start routing; handles unseen models using public benchmark signals |
| MTRouter | 2026 | Cost-aware multi-turn routing via history-model joint embeddings; 58.7% cost reduction (ACL 2026) |
| STEER | 2025 | Confidence-guided stepwise routing between small/large models; no trained router |
| Routing, Cascades & User Choice | 2026 | Game-theoretic analysis: optimal routing is usually static with no cascading; exposes provider/user misalignment |
| Paper | Year | Key Result |
|---|---|---|
| Lost in the Middle | 2023 | Models struggle with mid-context info |
| Context Rot | 2025 | Degradation before context limits |
| RAG vs Long Context | 2025 | Complementary strengths by query type |
| Self-Route Hybrid | 2024 | Adaptive RAG + long context |
| InfiniteICL | 2025 | 90% reduction, 103% performance |
| YaRN Context Extension | 2023 | 10x less tokens for context extension |
| SkyLadder | 2025 | 22% training time savings |
| TRIM | 2024 | 19.4% token savings on GPT-4o |
| ILRe | 2025 | Intermediate-layer retrieval cuts prefill to O(L); ~180x speedup, 1M tokens in ~30s |
| Context Length Alone Hurts | 2025 | Input length itself degrades performance even with perfect retrieval |
| ContextBudget (BACM) | 2026 | Budget-aware context management as constrained sequential decision; curriculum RL learns when/how much history to compress |
| LCLMs (End-to-End Context Compression) | 2026 | 0.6B encoder compresses input blocks into latents a 4B decoder consumes directly; ~16x input compression with little accuracy loss |
| Paper | Year | Key Result |
|---|---|---|
| PagedAttention (vLLM) | 2023 | Near-zero KV cache waste |
| RadixAttention (SGLang) | 2023 | Auto KV cache reuse |
| KV Cache Survey (2026) | 2026 | Comprehensive techniques survey |
| VectorQ Semantic Caching | 2025 | Up to 100x latency reduction |
| KV-Compress | 2024 | Variable-head-rate compression |
| vAttention | 2024 | 1.99x throughput over vLLM |
| LazyLLM | 2024 | Dynamic token pruning at prefill |
| SlimInfer | 2025 | 1.88x latency reduction |
| Mirror Speculative Decoding | 2025 | Breaks serial barrier |
| LongSpec | 2025 | Constant memory speculative decoding |
| Speculative Speculative Decoding | 2026 | Parallelizes speculation+verification; 30% faster than standard SD (ICLR 2026) |
| IceCache | 2026 | Semantic clustering for KV pages; 99% accuracy at 25% token budget |
| Can I Buy Your KV Cache? | 2026 | KV cache marketplace: publishers precompute, agents load instead of prefill; 9-50x cheaper compute on Qwen3-4B |
| LMCache | 2025 | KV cache across GPU/CPU/disk/network; up to 15x throughput with vLLM |
| KV-Fold | 2026 | One-step KV-cache recurrence; training-free long-context inference |
| Thin Keys, Full Values | 2026 | SVD-based key-cache compression; up to 16x combined with GQA + quantization |
| Make Each Token Count | 2026 | Learnable retention gates for KV eviction that improve long-context accuracy |
| Meta-Soft | 2026 | Composable meta-tokens for context-preserving KV cache compression |
| KeepKV | 2025 | Adaptive lossless merging; 2x+ throughput at 10% KV budget |
| FreeKV | 2025 | Training-free speculative KV retrieval; up to 13x speedup, near-lossless |
| SmallKV | 2025 | Small-model-assisted eviction compensation; 1.75-2.56x higher throughput |
| Semantic Caching (Microsoft) | 2025 | Optimal semantic cache is NP-hard; Reverse Greedy + bandit learning |
| SpecFormer | 2025 | Lossless non-autoregressive drafting that holds up under large-batch serving |
| LaProx | 2026 | Output-aware, layer-wise KV eviction modeling attention×value interaction; beats prior eviction across 19 LongBench/NIAH datasets |
| Continuous Semantic Caching | 2026 | Theory for semantic caching in continuous embedding space; dynamic ε-net + kernel ridge regression |
| Learning to Draft (LTD) | 2026 | RL co-adapts draft+verify policies to optimize true throughput, not acceptance length (ICLR 2026) |
| DDTree (Block Diffusion) | 2026 | Block-diffusion draft tree for speculative decoding; outperforms EAGLE-3 at matched node budget |
| Graft | 2026 | Training-free prune-then-retrieve framework for speculative decoding draft trees; 5.41× speedup, 21.8% over EAGLE-3 on Qwen3-235B |
| Paper | Year | Key Result |
|---|---|---|
| APE (Automatic Prompt Engineer) | 2022 | LLMs generate optimal prompts |
| Concise Chain-of-Thought | 2024 | 48.7% shorter, negligible quality loss |
| Chain of Draft | 2025 | Only 7.6% of CoT tokens used |
| Semantic Compression | 2023 | Semantic compression with LLMs |
| Tokenomics | 2026 | Code review = 59.4% of tokens in agentic SE; input context dominates at 53.9% |
| IAPO | 2026 | Information-aware policy optimization; 36% reasoning-length reduction |
| SelfBudgeter | 2025 | Self-estimated reasoning budget via budget-guided GRPO; ~61% length cut |
| Step Pruner | 2025 | Step-aware RL reward; 33% of tokens at equal accuracy |
| BudgetThinker | 2025 | Budget-signaling control tokens for precise reasoning-length control |
| Extra-CoT | 2026 | Mixed-ratio SFT + RL for extreme-ratio CoT compression; ~73% token cut on MATH-500 with +0.6% accuracy |
| CROP | 2026 | Length-regularized automatic prompt optimization; up to ~80.6% output-token reduction (Google/Purdue) |
- LLM Safe Haven - Security toolkit for AI coding agents.
npx llm-safe-havenhardens Claude Code, Cursor, Windsurf in 60 seconds. Companion project — agent retries from security failures waste tokens. - Awesome AI Efficiency (Pruna) - Curated list on making AI faster, cheaper, smaller, and greener.
- Awesome Efficient LLM - Large curated list of efficient-LLM papers and tools.
- Simon Willison: LLM Pricing - Ongoing coverage of cost collapse.
- Simon Willison: LLMs in 2024 - MoE efficiency, cost trends.
- Eugene Yan: LLM Patterns - Caching (50%+ savings), fine-tuning, RAG, guardrails.
- Chip Huyen: AI OSS Analysis - 900 most popular AI tools analyzed.
- goodailist.com - Daily-updated tracker of 15K+ AI repos.
- HN: How Are You Handling LLM API Costs in Production?
- HN: The LLM Agent Cost Curve
- HN: Genosis - LLM Cost Optimization
- Latent Space: Artificial Analysis - The "smiling curve of AI costs".