diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c59cb558..cae39b62 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -95,23 +95,19 @@ To schedule a suite, add a team-owned workflow that calls [eval-refresh.yml](.gi Include refreshed results for PRs with new/changed evals so a reviewer can inspect them in Braintrust. -## Publishing and consuming results +## Consuming results -To publish a benchmark, upload complete runs with `--runs` and update the pins and rationales in [publication.ts](apps/web/src/data/publication.ts). Include every experiment in `experiments/benchmark/`. +Local eval runs write results to disk as described in [Running evals](README.md#running-evals). Uploaded results are in the [Evals project in Braintrust](https://www.braintrust.dev/app/supabase.io/p/Evals). -For local UI work with a partial run, temporarily keep only the uploaded experiment IDs and their rationales in the catalog and start the server with `EVALS_ALLOW_PARTIAL=1 pnpm web`. +Use the [Braintrust CLI](https://docs.braintrust.dev/reference/cli/quickstart) or [MCP server](https://docs.braintrust.dev/integrations/developer-tools/mcp) to query results. See the [Braintrust SQL reference](https://docs.braintrust.dev/reference/sql) for query syntax. -The published benchmark is at https://supabase.com/evals/eval-results.json. For other suites or history, query the Evals Braintrust project. For example, replace `EXPERIMENT_ID` below to read an experiment's scored runs: +For internal dashboards, see the [guide to reading eval results in Hex](https://app.notion.com/p/41c5004b775f815eb9c2f0485e74de5b). -```sql -SELECT root_span_id, metadata.eval, metadata.run, scores.passed -FROM experiment('EXPERIMENT_ID') -WHERE is_root = true -ORDER BY _pagination_key DESC -LIMIT 1000 -``` +## Publishing the benchmark -Follow the returned cursor as described in https://www.braintrust.dev/docs/api-reference/query. The [web fetcher](apps/web/scripts/fetch-braintrust.ts) shows how to read checks and token usage from child spans. +The [evals web app](https://supabase.com/evals) fetches benchmark results from Braintrust at build time. To publish a benchmark, refresh every experiment in `experiments/benchmark/` in CI, then update the pins and rationales in [publication.ts](apps/web/src/data/publication.ts). + +For local UI work with a partial run, temporarily keep only the uploaded experiment IDs and their rationales in the catalog and start the server with `EVALS_ALLOW_PARTIAL=1 pnpm web`. ## Reviewing an eval diff --git a/README.md b/README.md index b859d7ee..e3f6cd99 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ Set `BRAINTRUST_API_KEY` and `BRAINTRUST_PROJECT_ID` in the repository's `.env`, pnpm web ``` -The server fetches results once at startup. See [CONTRIBUTING.md](CONTRIBUTING.md#publishing-and-consuming-results) for publishing and querying results. +The server fetches results once at startup. See [consuming results](CONTRIBUTING.md#consuming-results) and [publishing the benchmark](CONTRIBUTING.md#publishing-the-benchmark). ## Eval Shape