Skip to content

About

Make an LLM agent's reliability measurable: a vendor-neutral reference wiring an agent to OpenTelemetry tracing, an eval harness, and guardrails — with evals running as a CI gate. Runs offline, zero keys.

Topics

Resources

Stars

3 stars

Watchers

1 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

Repository files navigation

agent-eval-starter

demo — evals as a CI gate: a green scorecard, then a seeded regression turns it red

Make an LLM agent's reliability measurable — and keep it that way. A small, vendor-neutral, framework-light reference that wires a working agent together with OpenTelemetry tracing, an eval harness, and guardrails, so evals run as CI and a regression fails the build like a broken test.

Everyone can build an agent. Almost no one can prove it still works after the next change — and the tooling that helps is mostly paid SaaS (Braintrust, LangSmith, DeepEval). This is the open, adapt-it-yourself version of that idea.

Runs end-to-end with zero API keys and zero network via a deterministic mock provider, so the whole thing is free to run and reproducible in CI.


Quickstart (< 5 minutes)

python -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"

agent-eval run -q "Who created Python and when?"   # watch the trace + answer
agent-eval eval                                     # score the suite -> scorecard
make demo                                           # green, then a regression turns it red
pytest -q                                           # unit + data gates

The 60-second demo — evals as a CI gate

Baseline is green:

metric                  rate  baseline    delta
-----------------------------------------------
task_success          95.00%    95.00%   +0.00%
tool_correctness     100.00%   100.00%   +0.00%
schema_validity      100.00%   100.00%   +0.00%
groundedness         100.00%   100.00%   +0.00%
PASS: no regressions vs baseline.

Now break grounding (AES_DISABLE_FETCH=1 — the agent answers from search snippets and cites documents it never actually read):

metric                  rate  baseline    delta
-----------------------------------------------
task_success          75.00%    95.00%  -20.00%  <-- REGRESSION
tool_correctness      15.00%   100.00%  -85.00%  <-- REGRESSION
schema_validity      100.00%   100.00%   +0.00%
groundedness          15.00%   100.00%  -85.00%  <-- REGRESSION
FAIL: 3 metric(s) regressed vs baseline: task_success, tool_correctness, groundedness

agent-eval eval exits non-zero, so CI goes red and posts the scorecard on the PR. Your agent's accuracy is now a gate, not a vibe.

What's inside

flowchart LR
    Q[question] --> A[agent loop]
    A -->|JSON action| P[provider: mock / claude / ollama]
    A --> T[tools: search / fetch / finish]
    A -. OpenTelemetry spans .-> O[(trace: tokens, cost, latency)]
    D[labelled dataset] --> E[eval harness]
    A --> E
    E --> S[scorecard.json/.md]
    S --> G{regressed vs baseline?}
    G -->|yes| R[CI fails]
    G -->|no| M[CI passes]
Loading

The metrics (each a pass-rate over the dataset):

  • task_success — did it answer correctly (or correctly abstain on the unanswerable cases)?
  • tool_correctness — did it use tools honestly, and only cite documents it fetched?
  • schema_validity — is the final answer well-formed (pydantic-validated)?
  • groundedness — the hallucination guard: is every claim backed by fetched text?

Why these choices

  • JSON-action protocol, not native tool-calling → portable across any provider and trivially mockable. No framework lock-in.
  • OpenTelemetry, not a bespoke tracer → the same spans that draw the trace tree roll up into a cost/latency/token scorecard, and export to any backend. No vendor lock-in.
  • Evals as CI → a committed evals/baseline.json turns "did this change make the agent worse?" into a red build.
  • Two-layer guardrails → the lesson from production is that prompt-level "don't say X" is unreliable, so a deterministic post-filter (guardrails.py) is the dependable fallback, with an optional LLM-as-verifier review pass (harness/review.py).
  • mock provider → the suite runs deterministically in CI with no keys, no spend, no flakiness.

Providers

provider use needs
mock (default) deterministic runs + CI nothing
claude real runs on Anthropic pip install '.[claude]' + ANTHROPIC_API_KEY
ollama local models a running Ollama; captures prompt_eval_count/eval_count
AES_PROVIDER=claude AES_MODEL=claude-haiku-4-5-20251001 agent-eval run -q "..."

Layout

src/agent_eval/
  agent.py         # the instrumented run loop (emits OTel spans)
  tools.py         # search / fetch / finish (finish is schema-validated)
  guardrails.py    # deterministic banned-phrase post-filter
  providers/       # base + mock + claude + ollama
  harness/         # metrics, runner (CI gate), review, span roll-up
data/corpus.json   # ~10 synthetic docs (offline, deterministic)
evals/dataset.jsonl  # ~20 labelled cases
evals/baseline.json  # committed regression gate
tests/             # unit tests + data gates
.github/workflows/eval.yml

Extending it

Swap the corpus/dataset for your domain, add metrics in harness/metrics.py, point the provider at your model, and commit a new baseline. The pattern is the product; the example is just a runnable seed.

License

MIT © Richard Atkins

About

Make an LLM agent's reliability measurable: a vendor-neutral reference wiring an agent to OpenTelemetry tracing, an eval harness, and guardrails — with evals running as a CI gate. Runs offline, zero keys.

Topics

Resources

Stars

3 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages