Skip to content

MENTIONS.md

Latest commit

 

History

History
103 lines (94 loc) · 58.8 KB

MENTIONS.md

File metadata and controls

103 lines (94 loc) · 58.8 KB
Raw

Eval Mentions — evals in the wild

Resources where evaluation is discussed but not the primary focus — agent-building posts and talks that carry a genuinely good eval segment. Kept separate from the main list to preserve its signal density; still vetted (the eval insight must clear the bar), just not eval-first.

💬 Eval insights inside general agent posts

Good eval commentary mined from agent-BUILDING writeups (not eval-primary) — each kept only if a strict judge rated the eval insight excellent/good. Takeaway + verbatim excerpt.

  • How we compare model quality in Cursor (CursorBench) — Naman Jain (Cursor) — https://cursor.com/blog/cursorbench · excellent — To avoid benchmark contamination, derive eval tasks from real committed code traced back to the agent request that produced it (Cursor Blame), and pair offline suites with controlled live-traffic analysis to catch regressions where outputs grade well but the user experience degrades — tracking a basket of outcome… (excerpt: "We source tasks for CursorBench using Cursor Blame, which traces committed code back to the agent request that produced it. ... We supplement CursorBench with controlled analysis on live traffic. These online evals…")

  • How we built our multi-agent research system — Jeremy Hadfield, Barry Zhang, Kenneth Lien, Florian Scholz, Jeremy Fox, and Daniel Ford — https://www.anthropic.com/engineering/multi-agent-research-system · excellent — Start evals with ~20 real-usage queries rather than waiting for a large suite—early on a prompt tweak can move success from 30% to 80%, so small samples already reveal big effects. A single LLM-judge call scoring a multi-dimensional rubric (factual/citation accuracy, completeness, source quality, tool efficiency) on… (excerpt: "We started with a set of about 20 queries representing real usage patterns. Evaluating these queries often required human judgment, but we found that an LLM judge that evaluated each output against criteria in a…")

  • Demystifying evals for AI agents — Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares, and Jiri De Jonghe — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents · excellent — Low eval scores frequently measure broken graders and harnesses, not weak models: rigid string-matching, ambiguous specs, and non-reproducible stochastic tasks can suppress a score from 95% to 42%, so you must read transcripts and audit the eval before trusting any number. (excerpt: "Opus 4.5 initially scored 42% on CORE-Bench, until an Anthropic researcher found multiple issues: rigid grading that penalized '96.12' when expecting '96.124991…', ambiguous task specs, and stochastic tasks that were…")

  • τ-Bench: Benchmarking AI agents for the real-world — Sierra (AI Research) — https://sierra.ai/blog/benchmarking-ai-agents · excellent — Reliability, not single-shot accuracy, is the real bar for agents: pass^k (success on all k independent trials of the same task) collapses GPT-4o from ~50% pass^1 to ~25% pass^8 in τ-retail, meaning only a 1-in-4 chance of handling 8 different customers with the same issue. Measure consistency across repeated trials,… (excerpt: "pass^k, which measures the agent's reliability and determines if it can successfully complete the same task multiple times (k representing the number of different trials). ... the agent powered by GPT-4o drops to ~25%…")

  • From AI agent prototype to product: Lessons from building AWS DevOps Agent — Efe Karakus — https://aws.amazon.com/blogs/devops/from-ai-agent-prototype-to-product-lessons-from-building-aws-devops-agent/ · excellent — Separate "capability" (pass@k: passed at least once in k tries) from "reliability" (pass^k: fraction of the k tries that passed) — a high pass@k with low pass^k means the agent CAN solve a task but does so unreliably, which is the metric that actually matters for shipping a non-deterministic agent. (excerpt: "Key metrics that we keep track of are capability (pass@k – whether the agent passed at least once in k attempts), reliability (pass^k – how many times the agent passed across k attempts, e.g., 0.33 means passed 1 out of…")

  • Notion's Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future — Simon Last & Sarah Sachs (Notion), interviewed on Latent Space — https://www.latent.space/p/notion · excellent — Notion runs a three-tier eval system with distinct pass-rate targets: regression/unit tests gated in CI, launch "report card" evals requiring 80-90% across user journeys to ship, and deliberately hard "frontier/headroom" evals targeted at ~30% pass rate so the suite keeps giving signal instead of saturating. The… (excerpt: "we have the equivalent of unit test. Regression test. Those live in ci, those have to pass a certain percent ... we have a report card and we need to, on these categories, you know, be it 80 or 90% of all of these user…")

  • A practical blueprint for evaluating conversational AI at scale (Dash) — Dropbox (Dropbox Engineering / ML team) — https://dropbox.tech/machine-learning/practical-blueprint-evaluating-conversational-ai-at-scale-dash · excellent — Tier your eval metrics by enforcement: boolean gates as hard blockers (citations present?), scalar budgets with concrete thresholds (Source F1 >= 0.85, p95 latency <= 5s) that block merges, and rubric scores (tone/formatting) that are only dashboard-monitored, not gating. This separates "must never regress" from… (excerpt: "we defined three types of metrics, each with a clear role in the development pipeline: Boolean gates ("Citations present?", "Source present?") | Hard fail, changes can't move forward; Scalar budgets (Source F1 ≥ 0.85,…")

  • Continually improving our agent harness — Stefan Heule & Jediah Katz (Cursor) — https://cursor.com/blog/continually-improving-agent-harness · good — Pair offline benchmarks (CursorBench) with online signals that proxy real satisfaction: a "Keep Rate" measuring what fraction of agent-written code survives in the codebase after fixed time intervals, plus an LLM-judge reading user follow-up messages to infer satisfaction, validated via side-by-side A/B tests of… (excerpt: "The first is the "Keep Rate" of agent-generated code. For a given set of code changes that the agent proposed, we track what fraction of those remain in the user's codebase after fixed intervals of time. ... Second, we…")

  • Enabling Agent 3 to Self-Test at Scale with REPL-Based Verification — Peter Zhong, Jacky Zhao, Ryan Carelli (Replit) — https://replit.com/blog/automated-self-testing · good — Verification scales with autonomy: as an agent runs longer unattended (Replit went from ~20 min to 200+ min of productive autonomous work), robust self-testing becomes the gating factor because errors compound — and they isolate testing into a separate subagent to avoid context pollution, reaching multi-hundred-step… (excerpt: "we've created a self-testing flow for the Agent that is able to perform complex, multi-hundred step testing at a median cost of $0.20 per session.")

  • A review of OpenAI's o1 and how we evaluate coding agents — Cognition (Devin team) — https://cognition.com/blog/evaluating-coding-agents · good — When your judge is itself an agent (with shell/browser/code-editing tools autonomously deciding pass/fail), you must validate the judge: measure its precision and recall against a labeled ground-truth set and keep humans continuously reviewing the "proof of success" it surfaces. They also average over multiple Devin… (excerpt: "We evaluate our evaluators in two ways: (1) Measuring precision and recall on ground truth sets (2) Continuous human review of the proof of success discovered by the evaluator agents.")

  • How we built a high-quality AI code review agent — Akshay Utture (Augment Code) — https://www.augmentcode.com/blog/how-we-built-high-quality-ai-code-review-agent · good — They run a fast offline benchmark (LLM-as-judge comparing generated comments to human-authored "golden comments" on 10 PRs across 5 repos) using F-score as the hill-climbing metric, then map each offline metric to a production proxy: recall to "bugs fixed per PR" and precision to "percentage of comments addressed."… (excerpt: "F-score acts as the primary hill-climbing metric for offline improvements. ... Bugs fixed per PR | Recall | Measures real-world bug prevention and review coverage | Percentage of comments addressed | Precision |…")

  • Zed now predicts your next edit with Zeta, our new open model — Antonio Scandurra & Nathan Sobo (Zed) — https://zed.dev/blog/edit-prediction · good — When the correct output is non-deterministic and admits many valid forms (e.g. code edits), replace brittle token/string assertions with an LLM judge checking plain-English intent assertions (e.g. "ensure quicksort recurses left and right of the pivot"); this tolerates run-to-run variation while still catching wrong… (excerpt: "instead of strict assertions, we used a larger LLM to evaluate Zeta's edits. By writing our test assertions in plain English and having Claude check if the results matched our intent, we could validate that Zeta was…")

  • Code Droid: A Technical Report — Factory.ai — https://factory.ai/news/code-droid-technical-report · good — They decompose agent failures into distinct stages of the localization pipeline — file not retrieved (8%), retrieved but not ranked top-5 (8%), and ranked top but not edited (6%) — which tells practitioners exactly where to invest (retrieval recall vs. ranking vs. edit selection) rather than treating a failed task as… (excerpt: "In 8% of the tasks, Code Droid failed to include the target file in its list of analyzed files. Additionally, even when the target file was analyzed, it was not prioritized as a top-5 file in another 8% of cases.…")

  • Lessons from building AI coding assistants: context retrieval and evaluation — Jan Hartman (Sourcegraph) — https://sourcegraph.com/blog/lessons-from-building-ai-coding-assistants-context-retrieval-and-evaluation · good — When you can't get ground-truth labels for "relevant context," end-to-end user feedback can't tell you whether a bad answer came from retrieval or from the LLM — so substitute cheap automatic proxy checks (code compiles/passes tests for generation; referenced symbols actually exist for code Q&A) and separately… (excerpt: "Since users primarily interact with the LLM's responses rather than the context items themselves, it's hard to know if context retrieval is making a difference. We might get feedback that a response was unhelpful, but…")

  • Introducing Scorers in Mastra — Yujohn Nattrass — https://mastra.ai/blog/mastra-scorers · good — Don't ask an LLM judge to emit a raw 0-1 score directly — it's high-variance and irreproducible. Instead have the LLM emit structured intermediate data (e.g. extract claims/opinions, label each), then compute the score deterministically in code (proportion that pass), keeping the LLM's nuance but making the number… (excerpt: "LLMs are terrible at producing consistent numerical scores—ask the same model to rate something from 0-1 five times and you'll get five different numbers. So we have LLMs output structured data instead, then use a…")

  • Benchmarking AI Agent Memory: Is a Filesystem All You Need? — Letta — https://www.letta.com/blog/benchmarking-ai-agent-memory/ · good — A simple filesystem-backed agent (search_files → grep/open → answer) beats a specialized graph-memory system on LoCoMo, supporting their thesis that what matters for memory eval is whether the agent knows WHEN and HOW to call a retrieval tool, not the underlying retrieval mechanism (vector DB vs knowledge graph). They… (excerpt: "This simple agent achieves 74.0% on LoCoMo with GPT-4o mini and minimal prompt tuning, significantly above Mem0's reported 68.5% score for their top-performing graph variant.")

  • Testing Agent Skills Systematically with Evals — Dominik Kundel, Gabriel Chua — https://developers.openai.com/blog/eval-skills · good — Structure skill evals as deterministic trace checks first (parse the --json JSONL stream: assert specific commands ran, count command_execution items to catch looping/re-run regressions, track usage tokens to catch prompt bloat), then layer a model-assisted --output-schema rubric step only for the qualitative parts… (excerpt: "Deterministic checks answer 'did it do the basics?' but they don't answer 'did it do it the way you wanted?' For skills like setup-demo-app, many requirements are qualitative: component structure, styling conventions,…")

  • Evaluating Deep Agents: Our Learnings — The LangChain Team — https://www.langchain.com/blog/evaluating-deep-agents-our-learnings · good — For multi-turn agent evals, you can't hardcode a fixed sequence of user inputs because once the agent diverges from the expected path the later scripted inputs become incoherent; pair this with per-test fresh/temporary environments so runs stay reproducible and non-flaky, and lean on single-step evals since… (excerpt: "if you naively hardcode a sequence of inputs and the agent deviates from the expected path, the subsequent hardcoded user input may not make sense.")

  • Eval-driven development: Build better AI faster — Malte Ubl, Alice Alexandra Moore, Ido Pesok — https://vercel.com/blog/eval-driven-development-build-better-ai-faster · good — Tier your graders by cost/objectivity (code checks first, LLM grading reserved for subjective calls since it runs 1.5-2x more expensive), hold a hard 100% pass bar on refusal/safety, and deliberately seed the eval set with prompts that currently fail so improvements are tracked and regressions caught as prompts… (excerpt: "Our multi-faceted evaluation strategy includes fast, reliable code checks, end user and internal human feedback, and LLM-based grading for complex judgments at scale. [...] Some of our checks for code quality include:…")

  • Letta Leaderboard: Benchmarking LLMs on Agentic Memory — Letta — https://www.letta.com/blog/letta-leaderboard · good — A good agentic-memory eval must penalize unnecessary memory tool calls, not just reward correct answers: models that are strong at archival retrieval tend to over-call memory tools even when the answer is already in context, which is a real failure mode you only catch if your scoring includes an extraneous-operation… (excerpt: "Models that perform well on archival memory (e.g., Claude Haiku 3-5) might overuse memory operations when unnecessary and receive a lower score on core memory due to penalties.")

  • The evaluation engine behind Decagon's AI agents — Decagon — https://decagon.ai/blog/evaluation-engine-ai-agents · good — A two-stage eval gate (offline LLM-as-judge over query/context/response triplets plus an expert-labeled ground-truth set, then online A/B with gradual traffic ramp) keeps unreliable variants out of production; auditing a subset of judge scores with human labellers validates the judge itself, and online success is… (excerpt: "Using an LLM-as-judge system, we evaluate structured triplets consisting of a user query, the context provided to the model, and the model's generated response. ... We evaluate responses against a ground truth…")

  • AI prompt design at Parahelp — Anker & Mads (Parahelp co-founders) — https://parahelp.com/blog/prompt-design · good — Designing the agent to emit structured XML output is a deliberate "design-for-evaluability" tactic: rigid, parseable output lets you programmatically grade each decision, and pairing it with an outcome metric like "% of tickets resolved end-to-end" grounds eval in real production results rather than proxy scores. (excerpt: "This made the model more strict (and let us parse XML for evals)")

  • How we Built a State-of-the-Art Research Agent for Call Center Conversation Analytics — Iwona Bialynicka-Birula, Ryan Muir, Binoy Robin Dalal, Hagyeong Shin, Nikolai Glushnev — https://cresta.com/blog/how-we-built-a-state-of-the-art-research-agent-for-call-center-conversation-analytics · good — They isolated the dominant hallucination driver (questions whose answer simply isn't in the conversation) and pulled counting/aggregation out of the LLM into deterministic code so report statistics are guaranteed correct, while tracking two concrete report-quality metrics (relevance-classification accuracy and… (excerpt: "Human experts scrutinized a wide range of AI Analyst reports and identified two key metrics that were key drivers of report quality: relevance classification accuracy and the factuality of claims about the…")

  • Why Speech to Text Is the Hidden Engine Behind Contact Center AI Performance — Cresta — https://cresta.com/blog/why-speech-to-text-is-the-hidden-engine-behind-contact-center-ai-performance · good — STT quality is the upstream bottleneck for downstream agent tasks: WER should be measured on a domain-stratified corpus (here 2,703 files / 81.69 hours / 9 domains) because small WER deltas compound at scale (1% WER over 1M minutes = ~10,000 fewer errors), and targeted fine-tuning or keyterm prompting moves the needle… (excerpt: "WER benchmarking was based on a dataset comprising 2,703 audio files across nine distinct domains, totaling 81.69 hours")

  • Your Voice Agents Need Tests. Now They Have Them. — Vapi (Vapi Editorial Team) — https://vapi.ai/blog/evals · good — Convert real production failures into regression tests by capturing the bad transcript and annotating the correct behavior, and match different criteria with different judges: regex/JSON/exact for deterministic outputs (e.g., a tool call must include particular arguments), LLM-as-judge for fuzzy qualities like tone,… (excerpt: "When you discover a bad call in your logs, you can turn that transcript into a test. In the dashboard, pull up the call, click the thumbs down button to use it as an eval, specify what the assistant should have done…")

  • Agents — Chip Huyen — https://huyenchip.com/2025/01/07/agents.html · good — Decompose agent planning evaluation into a concrete (task, tool-inventory) dataset and sample K plans per task, then track plan-level metrics (fraction valid, retries-to-valid) and tool-call-level metrics (invalid tool, valid tool with wrong params, valid tool with wrong values) — separating the distinct failure modes… (excerpt: "To evaluate an agent for planning failures, you can create a dataset of (task, tool inventory) pairs. For each task, use an agent to generate K plans. Compute the following metrics: Out of all generated plans, how many…")

  • LLM Powered Autonomous Agents — Lilian Weng — https://lilianweng.github.io/posts/2023-06-23-agent/ · good — LLM-as-judge can silently fail in expert domains: in ChemCrow, an LLM evaluator rated GPT-4 and ChemCrow as roughly equal, while domain experts judging chemical correctness found ChemCrow far superior. The takeaway is that an LLM judge lacking domain expertise cannot detect flaws it doesn't understand, so… (excerpt: "Interestingly, while the LLM-based evaluation concluded that GPT-4 and ChemCrow perform nearly equivalently, human evaluations with experts oriented towards the completion and chemical correctness of the solutions…")

  • Can AI agents build real Stripe integrations? We built a benchmark to find out — Carol Liang and Kevin Ho (Stripe, API Standards) — https://stripe.com/blog/can-ai-agents-build-real-stripe-integrations · good — Don't grade an agent on its own self-reported success or surface-level API/UI responses; verify the real side effects in the system of record (here, the actual Stripe API object the action should have created). This catches the documented failure where an agent saw a 400 error on invalid test data and declared "Good,… (excerpt: "Some graders also validated the Stripe artifacts of a run by inspecting created Stripe API objects. For example, in a full-stack challenge, the agent might complete a payment in the UI, then verify success by testing…")

  • Developing Rapidly with Generative AI — Discord — https://discord.com/blog/developing-rapidly-with-generative-ai · good — Use a separate LLM (a "critic") to score your agent's outputs against criteria, and structure the judge prompt to force constrained outputs — yes/no or a numeric scale — rather than free-form critique, which makes the eval signal aggregable and lets you compare prompt variants quickly. (excerpt: "AI-assisted evaluation uses best-in-class LLMs (like GPT-4) to automatically critique how well the AI's outputs match what we expected or how they score against a set of criteria. ... This method uses GPT-4 in a way…")

  • What it takes to build AI agents at scale — Gayatri Sabharwal — https://ramp.com/leading-indicators/what-it-takes-to-build-ai-agents-at-scale · good — Build eval ground truth from a domain expert's spec, then use a frontier model to generate adversarial edge cases the expert missed, and validate with beta-user feedback; the genuinely hard problem is deciding when eval coverage is sufficient to remove the human from the loop. The post also draws a useful line:… (excerpt: "At Ramp, the eval suite starts with a human expert — often an accountant — who writes down how the task should go. A frontier model then stress-tests it, surfacing edge cases or the scenarios the expert didn't think of.…")

  • How we made v0 an effective coding agent — Max Leiter — https://vercel.com/blog/how-we-made-v0-an-effective-coding-agent · good — Define the agent's primary metric as a binary user-visible outcome (does the generated site actually render, not error/blank) rather than text-similarity, then attack the ~10% LLM error baseline with a streaming autofix layer targeting specific named failure modes (stale APIs, nonexistent icons, missing providers,… (excerpt: "The primary metric we optimize for is the percentage of successful generations. A successful generation is one that produces a working website in v0's preview instead of an error or blank screen. ... In our experience,…")

  • Estimating the Productivity of an Autonomous AI Software Engineer — The Cognition Team — https://cognition.com/blog/ai-productivity · blog — Automated system converting Devin production sessions to equivalent human engineering hours via an AI classifier: r_log = 0.74 on 233 held-out sessions from 126 users; "around 50% of sessions fall within a factor of 2 of the true estimate" but aggregate estimates converge. Novel production-measurement methodology: reason about the human's path, not the agent's. 🆕

  • Bugbot now self-improves with learned rules — Cursor — https://cursor.com/blog/bugbot-learning · excellent — Implement an online rule-promotion loop fed by three post-merge signals — developer downvotes on agent comments, developer correction replies, and reviewer comments flagging missed issues — driving rules through a candidate → active → disabled lifecycle. Result: resolution rate ~80%, 15 points above the next-closest competitor (Greptile: 63.5%, CodeRabbit: 49.0%); 110k+ repositories, 44k+ custom rules generated. (excerpt: "each review is a natural experiment that Bugbot can use to self-improve based on whether the developer acted or not on its report.") 🆕

  • Why coding agents fail in large codebases (and what to do about it) — Sourcegraph — https://sourcegraph.com/blog/why-coding-agents-fail-large-codebases · excellent — From 1,281 scored agent runs across 40+ large repos, five repeatable infrastructure failure patterns dominate: (1) lost navigation without convergence, (2) lexical search without structural ranking, (3) missing cross-repo files, (4) tool thrashing / retry loops, (5) context overflow from irrelevant reads. MCP-augmented agents cut cost 30%, cut time 38%, and improved retrieval precision 2–3×. Core thesis: these are infrastructure failures, not intelligence failures. (excerpt: "Data from 1,281 agent runs across 40+ large open source repos reveals five repeatable failure patterns in coding agents, and the infrastructure fixes for each. Agent failures in large codebases are not random. They cluster into five repeatable patterns.") 🆕

  • Decision-Time Guidance: Keeping Replit Agent Reliable — Peter Zhong et al. (Replit) — https://replit.com/blog/decision-time-guidance · good — Rather than encoding all guidance in the system prompt (which degrades as context grows), inject short situational instructions at agent decision points identified by a lightweight multi-label trajectory classifier. Recency bias makes end-of-trace injection far more effective: the same guidance injected at the trace end caused the agent to execute 15% more tools per loop vs. system-prompt placement. Cache hit rate stays at 90%+ since the core prompt never changes. (excerpt: "injecting a short prompt at the bottom of the trace led the agent to execute 15% more tools per loop ... reducing cost by 90% compared to dynamic system prompt modification") 🆕

  • Harness design for long-running application development — Anthropic — https://www.anthropic.com/engineering/harness-design-long-running-apps · good — Self-evaluation by a builder agent reliably fails: models "confidently praise their work — even when, to a human observer, the quality is obviously mediocre." Delegating evaluation to a separate Evaluator agent that uses Playwright to click through the running application proved far more tractable. Also: as model capabilities improve, load-bearing harness components can be removed — frameworks require regular reassessment. (excerpt: "When asked to evaluate work they've produced, agents tend to respond by confidently praising the work — even when, to a human observer, the quality is obviously mediocre. ... The evaluator used the Playwright MCP to click through the running application the way a user would, testing UI features, API endpoints, and database states.") 🆕

  • How Capital One Delivers Multi-Agent Systems (TWIML #765) — Rashmi Shetty — Rashmi Shetty (Capital One) with Sam Charrington (TWIML AI Podcast) — https://twimlai.com/podcast/twimlai/how-capital-one-delivers-multi-agent-systems · podcast (good) — A senior Capital One platform leader describes evaluating a real deployed multi-agent system (Chat Concierge for auto dealerships) by shifting from per-model ML metrics to end-to-end task-outcome evaluation, treating evals for stochastic multi-agent workflows plus observability as first-class… 🆕

  • Don't Build Agents, Build Skills Instead — Barry Zhang & Mahesh Murag (Anthropic) — https://www.youtube.com/watch?v=CEvIs9y1uog · talk (good) — Anthropic's case for packaging procedural knowledge as composable "Skills" (organized folders + self-documenting scripts, loaded via progressive disclosure so metadata is cheap until a skill.md is invoked) rather than building bespoke domain agents. For evals specifically, the speakers name the… 🆕

  • Teaching agents product design at Vercel — Vercel — https://vercel.com/blog/teaching-agents-product-design-at-vercel · excellent — Score rule correctness separately from similarity-to-shipped-result — "shipped code can contain a flaw that the agent should improve instead of reproduce" — and run fixtures WITHOUT the skill as an explicit no-skill control baseline to verify the instruction is doing any causal work at all. Evals are drawn from real shipped examples with holdout sets hiding expected edits so coverage generalizes. (excerpt: "We score rule correctness separately from similarity to the shipped result. Shipped code can contain a flaw that the agent should improve instead of reproduce. ... We also run fixtures without the skill to measure whether it changed the agent's behavior.") 🆕

🎯 Eval segments inside agent-building talks

Talks about building agents (Devin, Claude Code, Cursor, Replit, OpenAI Deep Research, Karpathy…) with a substantive eval segment — the eval part is noted.