Make an LLM agent's reliability measurable — and keep it that way. A small, vendor-neutral, framework-light reference that wires a working agent together with OpenTelemetry tracing, an eval harness, and guardrails, so evals run as CI and a regression fails the build like a broken test.
Everyone can build an agent. Almost no one can prove it still works after the next change — and the tooling that helps is mostly paid SaaS (Braintrust, LangSmith, DeepEval). This is the open, adapt-it-yourself version of that idea.
Runs end-to-end with zero API keys and zero network via a deterministic
mockprovider, so the whole thing is free to run and reproducible in CI.
python -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"
agent-eval run -q "Who created Python and when?" # watch the trace + answer
agent-eval eval # score the suite -> scorecard
make demo # green, then a regression turns it red
pytest -q # unit + data gatesBaseline is green:
metric rate baseline delta
-----------------------------------------------
task_success 95.00% 95.00% +0.00%
tool_correctness 100.00% 100.00% +0.00%
schema_validity 100.00% 100.00% +0.00%
groundedness 100.00% 100.00% +0.00%
PASS: no regressions vs baseline.
Now break grounding (AES_DISABLE_FETCH=1 — the agent answers from search
snippets and cites documents it never actually read):
metric rate baseline delta
-----------------------------------------------
task_success 75.00% 95.00% -20.00% <-- REGRESSION
tool_correctness 15.00% 100.00% -85.00% <-- REGRESSION
schema_validity 100.00% 100.00% +0.00%
groundedness 15.00% 100.00% -85.00% <-- REGRESSION
FAIL: 3 metric(s) regressed vs baseline: task_success, tool_correctness, groundedness
agent-eval eval exits non-zero, so CI goes red and
posts the scorecard on the PR. Your agent's accuracy is now a gate, not a vibe.
flowchart LR
Q[question] --> A[agent loop]
A -->|JSON action| P[provider: mock / claude / ollama]
A --> T[tools: search / fetch / finish]
A -. OpenTelemetry spans .-> O[(trace: tokens, cost, latency)]
D[labelled dataset] --> E[eval harness]
A --> E
E --> S[scorecard.json/.md]
S --> G{regressed vs baseline?}
G -->|yes| R[CI fails]
G -->|no| M[CI passes]
The metrics (each a pass-rate over the dataset):
- task_success — did it answer correctly (or correctly abstain on the unanswerable cases)?
- tool_correctness — did it use tools honestly, and only cite documents it fetched?
- schema_validity — is the final answer well-formed (pydantic-validated)?
- groundedness — the hallucination guard: is every claim backed by fetched text?
- JSON-action protocol, not native tool-calling → portable across any provider and trivially mockable. No framework lock-in.
- OpenTelemetry, not a bespoke tracer → the same spans that draw the trace tree roll up into a cost/latency/token scorecard, and export to any backend. No vendor lock-in.
- Evals as CI → a committed
evals/baseline.jsonturns "did this change make the agent worse?" into a red build. - Two-layer guardrails → the lesson from production is that prompt-level "don't say X" is unreliable, so a deterministic post-filter (
guardrails.py) is the dependable fallback, with an optional LLM-as-verifier review pass (harness/review.py). mockprovider → the suite runs deterministically in CI with no keys, no spend, no flakiness.
| provider | use | needs |
|---|---|---|
mock (default) |
deterministic runs + CI | nothing |
claude |
real runs on Anthropic | pip install '.[claude]' + ANTHROPIC_API_KEY |
ollama |
local models | a running Ollama; captures prompt_eval_count/eval_count |
AES_PROVIDER=claude AES_MODEL=claude-haiku-4-5-20251001 agent-eval run -q "..."src/agent_eval/
agent.py # the instrumented run loop (emits OTel spans)
tools.py # search / fetch / finish (finish is schema-validated)
guardrails.py # deterministic banned-phrase post-filter
providers/ # base + mock + claude + ollama
harness/ # metrics, runner (CI gate), review, span roll-up
data/corpus.json # ~10 synthetic docs (offline, deterministic)
evals/dataset.jsonl # ~20 labelled cases
evals/baseline.json # committed regression gate
tests/ # unit tests + data gates
.github/workflows/eval.yml
Swap the corpus/dataset for your domain, add metrics in harness/metrics.py,
point the provider at your model, and commit a new baseline. The pattern is the
product; the example is just a runnable seed.
MIT © Richard Atkins
