Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 8 additions & 12 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,23 +95,19 @@ To schedule a suite, add a team-owned workflow that calls [eval-refresh.yml](.gi

Include refreshed results for PRs with new/changed evals so a reviewer can inspect them in Braintrust.

## Publishing and consuming results
## Consuming results

To publish a benchmark, upload complete runs with `--runs` and update the pins and rationales in [publication.ts](apps/web/src/data/publication.ts). Include every experiment in `experiments/benchmark/`.
Local eval runs write results to disk as described in [Running evals](README.md#running-evals). Uploaded results are in the [Evals project in Braintrust](https://www.braintrust.dev/app/supabase.io/p/Evals).

For local UI work with a partial run, temporarily keep only the uploaded experiment IDs and their rationales in the catalog and start the server with `EVALS_ALLOW_PARTIAL=1 pnpm web`.
Use the [Braintrust CLI](https://docs.braintrust.dev/reference/cli/quickstart) or [MCP server](https://docs.braintrust.dev/integrations/developer-tools/mcp) to query results. See the [Braintrust SQL reference](https://docs.braintrust.dev/reference/sql) for query syntax.

The published benchmark is at https://supabase.com/evals/eval-results.json. For other suites or history, query the Evals Braintrust project. For example, replace `EXPERIMENT_ID` below to read an experiment's scored runs:
For internal dashboards, see the [guide to reading eval results in Hex](https://app.notion.com/p/41c5004b775f815eb9c2f0485e74de5b).

```sql
SELECT root_span_id, metadata.eval, metadata.run, scores.passed
FROM experiment('EXPERIMENT_ID')
WHERE is_root = true
ORDER BY _pagination_key DESC
LIMIT 1000
```
## Publishing the benchmark

Follow the returned cursor as described in https://www.braintrust.dev/docs/api-reference/query. The [web fetcher](apps/web/scripts/fetch-braintrust.ts) shows how to read checks and token usage from child spans.
The [evals web app](https://supabase.com/evals) fetches benchmark results from Braintrust at build time. To publish a benchmark, refresh every experiment in `experiments/benchmark/` in CI, then update the pins and rationales in [publication.ts](apps/web/src/data/publication.ts).

For local UI work with a partial run, temporarily keep only the uploaded experiment IDs and their rationales in the catalog and start the server with `EVALS_ALLOW_PARTIAL=1 pnpm web`.

## Reviewing an eval

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Set `BRAINTRUST_API_KEY` and `BRAINTRUST_PROJECT_ID` in the repository's `.env`,
pnpm web
```

The server fetches results once at startup. See [CONTRIBUTING.md](CONTRIBUTING.md#publishing-and-consuming-results) for publishing and querying results.
The server fetches results once at startup. See [consuming results](CONTRIBUTING.md#consuming-results) and [publishing the benchmark](CONTRIBUTING.md#publishing-the-benchmark).

## Eval Shape

Expand Down