eval-view/llms.txt at main · hidai25/eval-view · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
# EvalView

> Regression testing for AI agents. Snapshot behavior, detect regressions, block broken agents before production.

EvalView is an open-source testing and regression detection framework for AI agents. It sends test queries to your agent, records everything (tool calls, parameters, sequence, output, cost, latency), and diffs it against a golden baseline. When something changes, you know immediately.

## Key Facts

- Name: EvalView
- Tagline: "Proof that your agent still works."
- Category: AI Agent Testing / Regression Detection / LLM CI/CD
- License: Apache 2.0 (free and open source)
- Language: Python 3.9+
- Install: `pip install evalview`
- Version: 0.6.1
- Author: EvalView Team (hidai@evalview.com)

## Links

- Homepage: https://evalview.com
- GitHub: https://github.com/hidai25/eval-view
- PyPI: https://pypi.org/project/evalview/
- Documentation: https://github.com/hidai25/eval-view#readme
- Issues: https://github.com/hidai25/eval-view/issues
- Discussions: https://github.com/hidai25/eval-view/discussions
- Starter Template: https://github.com/hidai25/evalview-support-automation-template

## What Problem Does EvalView Solve?

AI agents break silently. You change a prompt, swap a model, or update a tool, and the agent degrades without any error. EvalView captures golden baselines of known-good behavior and automatically detects when behavior drifts. Normal tests catch crashes; tracing shows what happened after the fact; EvalView catches when the agent returns 200 but silently takes the wrong tool path.

## Quick Start

```
pip install evalview
evalview init        # Detect agent, create starter suite
evalview snapshot    # Save current behavior as baseline
evalview check       # Catch regressions after every change
evalview model-check # Detect silent drift in closed models (Claude, GPT, ...)
evalview demo        # See it live, no API key needed
```

## Four Scoring Layers

| Layer | Cost |
|-------|------|
| Tool calls + sequence | Free |
| Code-based checks (regex, JSON schema) | Free |
| Semantic similarity via embeddings | ~$0.00004/test |
| LLM-as-judge (GPT, Claude, Gemini, DeepSeek, Ollama) | ~$0.01/test |

## How EvalView Compares

EvalView is built for a different job than most adjacent tools, and pairs well with them:

- Langfuse, LangSmith: observability and tracing
- Braintrust, DeepEval: eval scoring and experiments
- Promptfoo: prompt and model comparison
- EvalView: behavior-regression gating for tool-calling agents in CI

Where EvalView focuses: full-trajectory diffing (tool calls, parameters, and order — not just final output), golden baselines with multi-variant support for non-determinism, silent model/runtime change detection, auto-heal with variant proposals, hermetic record/replay, and a deterministic tool/sequence diff that runs without an API key.

EvalView complements observability and eval platforms: use them for visibility and scoring, use EvalView for the merge-time regression gate.

Positioning reflects each tool's primary focus as of June 2026, based on public documentation; capabilities change over time. Product names are trademarks of their respective owners; EvalView is independent and not affiliated with them.

## Supported Frameworks

LangGraph, CrewAI, OpenAI Assistants, Anthropic Claude, HuggingFace, Ollama, MCP servers, and any HTTP API.

## Documentation

- Getting Started: https://github.com/hidai25/eval-view/blob/main/docs/GETTING_STARTED.md
- CLI Reference: https://github.com/hidai25/eval-view/blob/main/docs/CLI_REFERENCE.md
- FAQ: https://github.com/hidai25/eval-view/blob/main/docs/FAQ.md
- YAML Schema: https://github.com/hidai25/eval-view/blob/main/docs/YAML_SCHEMA.md
- Framework Support: https://github.com/hidai25/eval-view/blob/main/docs/FRAMEWORK_SUPPORT.md
- Golden Traces: https://github.com/hidai25/eval-view/blob/main/docs/GOLDEN_TRACES.md
- Evaluation Metrics: https://github.com/hidai25/eval-view/blob/main/docs/EVALUATION_METRICS.md
- CI/CD Integration: https://github.com/hidai25/eval-view/blob/main/docs/CI_CD.md
- Skills Testing: https://github.com/hidai25/eval-view/blob/main/docs/SKILLS_TESTING.md
- Statistical Mode: https://github.com/hidai25/eval-view/blob/main/docs/STATISTICAL_MODE.md
- MCP Contracts: https://github.com/hidai25/eval-view/blob/main/docs/MCP_CONTRACTS.md

## Comparisons

- EvalView vs LangSmith: https://github.com/hidai25/eval-view/blob/main/docs/VS_LANGSMITH.md
- EvalView vs Langfuse: https://github.com/hidai25/eval-view/blob/main/docs/VS_LANGFUSE.md
- EvalView vs Braintrust: https://github.com/hidai25/eval-view/blob/main/docs/VS_BRAINTRUST.md
- EvalView vs DeepEval: https://github.com/hidai25/eval-view/blob/main/docs/VS_DEEPEVAL.md

## Optional

- Full documentation: https://evalview.com/llms-full.txt