Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
935ccc0
Fix benchmark health-check starvation and capture server diagnostics
bradleyshep Oct 28, 2026
e304d0f
Reject benchmark uploads with unresolved provider failures
bradleyshep Oct 28, 2026
3d9f7c3
Limit benchmark model concurrency to reduce credit reservations
bradleyshep Oct 28, 2026
d8e317d
Merge branch 'master' into bradley/llm-benchmark-health-check-fix
bradleyshep Oct 29, 2026
ead2965
Handle nullable provider responses and continue independent benchmark…
bradleyshep Oct 29, 2026
3d56bfa
Finish remaining benchmark modes after a batch failure
bradleyshep Oct 29, 2026
274f46f
Isolate C# benchmark servers and handle startup failures
bradleyshep Oct 30, 2026
0cd339c
Preserve incomplete benchmark results and refusal evidence
bradleyshep Oct 30, 2026
214e8cc
Bound C# benchmark reference database lifetimes
bradleyshep Oct 31, 2026
a92afc7
Score rejected benchmark migrations as code failures
bradleyshep Oct 31, 2026
b680ad8
Score SDK-generated compile errors as code failures
bradleyshep Oct 31, 2026
c4a2f1c
Match schema parity tables across naming styles
bradleyshep Oct 31, 2026
c9b6622
Fail schema parity for the C# 1.x table form
bradleyshep Nov 2, 2026
e11e0b8
Score Rust compile errors and invalid modules as code failures
bradleyshep Nov 2, 2026
ad820d9
Score candidate publish failures unless the environment failed
bradleyshep Nov 2, 2026
9f1c231
Allow 12 hours for periodic benchmark jobs
bradleyshep Nov 2, 2026
bd0e97d
Keep scorer CLI identities per benchmark server
bradleyshep Nov 2, 2026
db052ea
Stop C# benchmark builds losing their SDK generators
bradleyshep Nov 2, 2026
be59dd7
Check behaviour on 27 tasks and stop scoring valid code as wrong
bradleyshep Nov 3, 2026
2da7c5a
Resolve subscribed table names and use C# names in the t_093 prompt
bradleyshep Nov 3, 2026
3e0062a
Use each language's declared names in the advanced prompts
bradleyshep Nov 3, 2026
223dbc3
Fix C# examples that do not compile in the server skill and lifecycle…
bradleyshep Nov 3, 2026
7ead14b
Make the delete checks fail a module that does nothing
bradleyshep Nov 3, 2026
4d670cb
State requirements in the prompts instead of naming APIs, and fix wha…
bradleyshep Nov 4, 2026
0581850
State failure and query requirements in prompts instead of how to wri…
bradleyshep Nov 4, 2026
0cd35ea
Test error delivery in its own task and say when init runs
bradleyshep Nov 4, 2026
7af3511
Update the benchmark README's task count
bradleyshep Nov 4, 2026
836badb
Say that t_023 defines no reducers
bradleyshep Nov 4, 2026
0a8878e
Score content rejections as failed tasks
bradleyshep Nov 4, 2026
6ef60e4
Remove the added caller sentences from t_074 and t_079
bradleyshep Nov 4, 2026
6fb0ee2
Stop failing C# tables that set Name without Accessor
bradleyshep Nov 4, 2026
77f5232
Give Grok the longer LLM call timeout
bradleyshep Nov 5, 2026
63d2908
Give Grok a 20 minute LLM call timeout
bradleyshep Nov 5, 2026
e085f0c
Count a blank reply at the output limit as a failed task
bradleyshep Nov 5, 2026
815d6c5
Retry a blank reply before counting it as a failed task
bradleyshep Nov 5, 2026
7c9b313
Add rules for writing LLM benchmark tasks
bradleyshep Nov 5, 2026
04616dd
Remove a product name from the benchmark README
bradleyshep Nov 5, 2026
8dcb79b
Merge remote-tracking branch 'origin/master' into bradley/llm-benchma…
bradleyshep Nov 6, 2026
175ecfa
Match the benchmark's C# project to the new .NET 10 module shape
bradleyshep Nov 6, 2026
2206be9
Format benchmark task specs and sync the codex-plugin skills copy
bradleyshep Nov 10, 2026
f8cef11
Keep guard unchanged, run the migration seed off the async thread, al…
bradleyshep Nov 10, 2026
835e5d0
Drop the timeout comment
bradleyshep Nov 10, 2026
ac70695
Use one timeout for every reference check
bradleyshep Nov 10, 2026
abe6fcd
Merge branch 'master' into bradley/llm-benchmark-health-check-fix
bradleyshep Nov 10, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions .github/workflows/llm-benchmark-periodic.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ jobs:
name: Run benchmarks (${{ matrix.lang }})
needs: prepare_matrix
runs-on: spacetimedb-linux
timeout-minutes: 1440
env:
CARGO_TARGET_DIR: ${{ github.workspace }}/target
TARGET_TRIPLE: x86_64-unknown-linux-gnu
Expand Down Expand Up @@ -180,11 +181,13 @@ jobs:
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: "1"
MSBUILDDISABLENODEREUSE: "1"
DOTNET_CLI_USE_MSBUILD_SERVER: "0"
# Bound in-flight provider credit reservations across the three language jobs.
LLM_BENCH_CONCURRENCY: "8"
LLM_BENCH_RUST_CONCURRENCY: "4"
LLM_BENCH_CSHARP_CONCURRENCY: "2"
LLM_BENCH_ROUTE_CONCURRENCY: ${{ matrix.lang == 'typescript' && '4' || matrix.lang == 'csharp' && '1' || '2' }}
LLM_BENCH_CSHARP_CONCURRENCY: "4"
LLM_BENCH_ROUTE_CONCURRENCY: "1"
LLM_BENCHMARK_REPORT_DIR: ${{ runner.temp }}/llm-benchmark-reports
LLM_BENCHMARK_DIAGNOSTICS_DIR: ${{ runner.temp }}/llm-benchmark-diagnostics
INPUT_LANGUAGE: ${{ matrix.lang }}
INPUT_MODEL_SET: ${{ inputs.model_set || 'website_active' }}
INPUT_MODELS: ${{ inputs.models || '' }}
Expand Down Expand Up @@ -271,6 +274,15 @@ jobs:
if-no-files-found: ignore
retention-days: 14

- name: Upload server diagnostics
if: always()
uses: actions/upload-artifact@v4
with:
name: llm-benchmark-diagnostics-${{ matrix.lang }}
path: ${{ runner.temp }}/llm-benchmark-diagnostics
if-no-files-found: ignore
retention-days: 14

summarize:
name: Summarize benchmark analysis
if: always()
Expand Down
14 changes: 12 additions & 2 deletions .github/workflows/llm-benchmark-validate-goldens.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ concurrency:
jobs:
validate-goldens:
runs-on: spacetimedb-linux
timeout-minutes: 60
timeout-minutes: 180
env:
CARGO_TARGET_DIR: ${{ github.workspace }}/target
TARGET_TRIPLE: x86_64-unknown-linux-gnu
Expand Down Expand Up @@ -122,5 +122,15 @@ jobs:
MSBUILDDISABLENODEREUSE: "1"
DOTNET_CLI_USE_MSBUILD_SERVER: "0"
LLM_BENCH_CSHARP_CONCURRENCY: "1"
LLM_BENCHMARK_DIAGNOSTICS_DIR: ${{ runner.temp }}/llm-golden-diagnostics
run: |
llm_benchmark run --goldens-only --lang ${{ matrix.lang }}
llm_benchmark run --goldens-only --lang ${{ matrix.lang }} --dry-run --skip-task-catalog-upload

- name: Upload server diagnostics
if: always()
uses: actions/upload-artifact@v4
with:
name: llm-golden-diagnostics-${{ matrix.lang }}
path: ${{ runner.temp }}/llm-golden-diagnostics
if-no-files-found: ignore
retention-days: 14
15 changes: 10 additions & 5 deletions codex-plugin/plugins/spacetimedb/skills/csharp-server/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,15 +187,17 @@ The pattern is `ctx.Db.{Accessor}.{ColumnName}.{Method}(value)` for all indexed

```csharp
[SpacetimeDB.Reducer(ReducerKind.Init)]
public static void OnInit(ReducerContext ctx) { ... }
public static void Init(ReducerContext ctx) { ... }

[SpacetimeDB.Reducer(ReducerKind.ClientConnected)]
public static void OnConnect(ReducerContext ctx) { ... }
public static void ClientConnected(ReducerContext ctx) { ... }

[SpacetimeDB.Reducer(ReducerKind.ClientDisconnected)]
public static void OnDisconnect(ReducerContext ctx) { ... }
public static void ClientDisconnected(ReducerContext ctx) { ... }
```

Reducer and procedure method names must not start with `On`, `on`, or `__`; the code generator rejects them (STDB0010), so name lifecycle reducers `ClientConnected`, not `OnConnect`.

`ctx.ConnectionId` is `ConnectionId?`, including in connection lifecycle reducers. Check or unwrap it before storing it in a non-nullable column or passing it to an index accessor.

## Views
Expand Down Expand Up @@ -333,7 +335,7 @@ public partial struct TickTimer
[SpacetimeDB.Reducer]
public static void Tick(ReducerContext ctx, TickTimer timer)
{
// timer row is auto-deleted after this reducer runs
// a one-time row is deleted after this runs; an interval row stays
}

// One-time: fires once at a specific time
Expand All @@ -344,7 +346,7 @@ var at = new ScheduleAt.Interval(TimeSpan.FromSeconds(5));
ctx.Db.TickTimer.Insert(new TickTimer { ScheduledId = 0, ScheduledAt = at });
```

Scheduled reducer callbacks use the ordinary `[SpacetimeDB.Reducer]` attribute. There is no `ReducerKind.Scheduled`; the table's `Scheduled` option associates the callback.
Scheduled reducer callbacks use the ordinary `[SpacetimeDB.Reducer]` attribute. There is no `ReducerKind.Scheduled`; the table's `Scheduled` option associates the callback. To let only the scheduler run a callback, reject calls where `ctx.Sender != ctx.DatabaseIdentity`.

To construct a `ConnectionId` from a 128-bit numeric representation, encode it as exactly 16 little-endian bytes and call `ConnectionId.From(bytes)`. The result is nullable and must be checked before use.

Expand All @@ -364,6 +366,7 @@ public static ResultValue Inspect(ProcedureContext ctx, string input) =>
Outbound HTTP is available through `ctx.Http`; handle its success/error result before using the response. Open short database transactions with `ctx.WithTx`. Perform network I/O before opening the transaction, and keep only database work inside its callback.

```csharp
string uri = "https://example.com/data"; // the C# HTTP API takes the URI as a string
var result = ctx.Http.Get(uri);
var text = result.Match(
response => response.Body.ToStringUtf8Lossy(),
Expand Down Expand Up @@ -435,3 +438,5 @@ public partial record Shape : SpacetimeDB.TaggedEnum<(Circle Circle, Rectangle R
var a = new Shape.Circle(new Circle { Radius = 10 });
var b = new Shape.Rectangle(new Rectangle { Width = 4, Height = 6 });
```

Declare `TaggedEnum` records inside your `Module` class. At file scope, a variant named after its own type, such as `Circle Circle`, fails with CS8910.
4 changes: 3 additions & 1 deletion codex-plugin/plugins/spacetimedb/skills/rust-server/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ pub struct TickTimer {

#[spacetimedb::reducer]
pub fn tick(ctx: &ReducerContext, timer: TickTimer) {
// timer row is auto-deleted after this reducer runs
// a one-time row is deleted after this runs; an interval row stays
}

// One-time: fires once at a specific time
Expand All @@ -284,6 +284,8 @@ let at = ScheduleAt::Interval(std::time::Duration::from_secs(5).into());
ctx.db.tick_timer().insert(TickTimer { scheduled_id: 0, scheduled_at: at });
```

To let only the scheduler run a callback, reject calls where `ctx.sender() != ctx.database_identity()`.

Construct a connection ID from a numeric representation with `ConnectionId::from_u128(value)`.

## Inter-Database Calls (Async IDC)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,7 @@ const tickTimer = table({
export const tick = spacetimedb.reducer(
{ onSchedule: tickTimer },
{ timer: tickTimer.rowType },
(ctx, { timer }) => { /* timer row auto-deleted after this runs */ }
(ctx, { timer }) => { /* a one-time row is deleted after this runs; an interval row stays */ }
);

// `onSchedule` also works for scheduled procedures whose return type is `t.unit()`.
Expand All @@ -261,6 +261,8 @@ export const tick = spacetimedb.reducer(
// const micros = at.tag === 'time' ? at.value : at.value.microsSinceUnixEpoch; // bigint
```

To let only the scheduler run a callback, reject calls where `!ctx.sender.isEqual(ctx.databaseIdentity)`.

## Submodules

Import the library's full module-namespace object, not just its default schema:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ export const onConnect = spacetimedb.clientConnected((ctx) => {

```csharp
[SpacetimeDB.Reducer(ReducerKind.ClientConnected)]
public static void OnConnect(ReducerContext ctx)
public static void ClientConnected(ReducerContext ctx)
{
Log.Info($"Client connected: {ctx.Sender}");

Expand Down Expand Up @@ -318,7 +318,7 @@ export const onDisconnect = spacetimedb.clientDisconnected((ctx) => {

```csharp
[SpacetimeDB.Reducer(ReducerKind.ClientDisconnected)]
public static void OnDisconnect(ReducerContext ctx)
public static void ClientDisconnected(ReducerContext ctx)
{
Log.Info($"Client disconnected: {ctx.Sender}");

Expand Down
15 changes: 10 additions & 5 deletions skills/csharp-server/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,15 +187,17 @@ The pattern is `ctx.Db.{Accessor}.{ColumnName}.{Method}(value)` for all indexed

```csharp
[SpacetimeDB.Reducer(ReducerKind.Init)]
public static void OnInit(ReducerContext ctx) { ... }
public static void Init(ReducerContext ctx) { ... }

[SpacetimeDB.Reducer(ReducerKind.ClientConnected)]
public static void OnConnect(ReducerContext ctx) { ... }
public static void ClientConnected(ReducerContext ctx) { ... }

[SpacetimeDB.Reducer(ReducerKind.ClientDisconnected)]
public static void OnDisconnect(ReducerContext ctx) { ... }
public static void ClientDisconnected(ReducerContext ctx) { ... }
```

Reducer and procedure method names must not start with `On`, `on`, or `__`; the code generator rejects them (STDB0010), so name lifecycle reducers `ClientConnected`, not `OnConnect`.

`ctx.ConnectionId` is `ConnectionId?`, including in connection lifecycle reducers. Check or unwrap it before storing it in a non-nullable column or passing it to an index accessor.

## Views
Expand Down Expand Up @@ -333,7 +335,7 @@ public partial struct TickTimer
[SpacetimeDB.Reducer]
public static void Tick(ReducerContext ctx, TickTimer timer)
{
// timer row is auto-deleted after this reducer runs
// a one-time row is deleted after this runs; an interval row stays
}

// One-time: fires once at a specific time
Expand All @@ -344,7 +346,7 @@ var at = new ScheduleAt.Interval(TimeSpan.FromSeconds(5));
ctx.Db.TickTimer.Insert(new TickTimer { ScheduledId = 0, ScheduledAt = at });
```

Scheduled reducer callbacks use the ordinary `[SpacetimeDB.Reducer]` attribute. There is no `ReducerKind.Scheduled`; the table's `Scheduled` option associates the callback.
Scheduled reducer callbacks use the ordinary `[SpacetimeDB.Reducer]` attribute. There is no `ReducerKind.Scheduled`; the table's `Scheduled` option associates the callback. To let only the scheduler run a callback, reject calls where `ctx.Sender != ctx.DatabaseIdentity`.

To construct a `ConnectionId` from a 128-bit numeric representation, encode it as exactly 16 little-endian bytes and call `ConnectionId.From(bytes)`. The result is nullable and must be checked before use.

Expand All @@ -364,6 +366,7 @@ public static ResultValue Inspect(ProcedureContext ctx, string input) =>
Outbound HTTP is available through `ctx.Http`; handle its success/error result before using the response. Open short database transactions with `ctx.WithTx`. Perform network I/O before opening the transaction, and keep only database work inside its callback.

```csharp
string uri = "https://example.com/data"; // the C# HTTP API takes the URI as a string
var result = ctx.Http.Get(uri);
var text = result.Match(
response => response.Body.ToStringUtf8Lossy(),
Expand Down Expand Up @@ -435,3 +438,5 @@ public partial record Shape : SpacetimeDB.TaggedEnum<(Circle Circle, Rectangle R
var a = new Shape.Circle(new Circle { Radius = 10 });
var b = new Shape.Rectangle(new Rectangle { Width = 4, Height = 6 });
```

Declare `TaggedEnum` records inside your `Module` class. At file scope, a variant named after its own type, such as `Circle Circle`, fails with CS8910.
4 changes: 3 additions & 1 deletion skills/rust-server/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ pub struct TickTimer {

#[spacetimedb::reducer]
pub fn tick(ctx: &ReducerContext, timer: TickTimer) {
// timer row is auto-deleted after this reducer runs
// a one-time row is deleted after this runs; an interval row stays
}

// One-time: fires once at a specific time
Expand All @@ -284,6 +284,8 @@ let at = ScheduleAt::Interval(std::time::Duration::from_secs(5).into());
ctx.db.tick_timer().insert(TickTimer { scheduled_id: 0, scheduled_at: at });
```

To let only the scheduler run a callback, reject calls where `ctx.sender() != ctx.database_identity()`.

Construct a connection ID from a numeric representation with `ConnectionId::from_u128(value)`.

## Inter-Database Calls (Async IDC)
Expand Down
4 changes: 3 additions & 1 deletion skills/typescript-server/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,7 @@ const tickTimer = table({
export const tick = spacetimedb.reducer(
{ onSchedule: tickTimer },
{ timer: tickTimer.rowType },
(ctx, { timer }) => { /* timer row auto-deleted after this runs */ }
(ctx, { timer }) => { /* a one-time row is deleted after this runs; an interval row stays */ }
);

// `onSchedule` also works for scheduled procedures whose return type is `t.unit()`.
Expand All @@ -261,6 +261,8 @@ export const tick = spacetimedb.reducer(
// const micros = at.tag === 'time' ? at.value : at.value.microsSinceUnixEpoch; // bigint
```

To let only the scheduler run a callback, reject calls where `!ctx.sender.isEqual(ctx.databaseIdentity)`.

## Submodules

Import the library's full module-namespace object, not just its default schema:
Expand Down
114 changes: 114 additions & 0 deletions tools/xtask-llm-benchmark/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Benchmark tasks

The goal is simple: valid code using the current API passes, and outdated or wrong code fails. A failure must measure the model, not our setup.

Every prompt change alters the test for every model. Change as little as the fix needs, and never rewrite prompts for style.

## Files

A task lives in `src/benchmarks/<category>/t_NNN_name/`:

- `spec.rs`: the graders
- `tasks/rust.txt`, `tasks/typescript.txt`, `tasks/csharp.txt`: the prompts
- `answers/rust.rs`, `answers/typescript.ts`, `answers/csharp.cs`: the reference answers

The build regenerates `src/generated/registry.rs`, which is tracked, so commit it with a new task. Update the task count and the task table in `README.md`.

## Choosing a task

- Test one concept. Compare `advanced/t_087_webhook_ordering` for a hard task and `basics/t_004_insert` for a simple one.
- Test a SpacetimeDB contract, not another product's API.
- Use local fixtures for outside calls, never a paid service.
- Do not name other products in task files, the README or commit messages.

## Prompts

Before writing a prompt, read the prompts of the nearest tasks in the same category and copy their format. Most tasks use `TABLE` and `REDUCERS` sections. The advanced tasks use short paragraphs followed by a table list.

**State requirements, not how to meet them.**

- Say "the caller's identity", "the current time" and "fail with", not `ctx.sender`, a time method or `panic!`. Models are being tested on that syntax.
- Say when something runs, such as "when the module is first published", not which lifecycle API to use.
- Do not list the collections or iterator calls to use.
- Do not reveal values that the checks use, such as a test URL.

**State everything the checks enforce.** A model cannot guess these:

- Table and view names, field names, field types, and which tables are public
- Primary keys, auto-increment and indexes
- Reducer names, argument types and any required return type
- Exact text, whenever a check compares text
- That a module has no tables or no reducers, when an extra one would fail the schema check

**Keep the three languages the same.** Only names and types differ.

- Use each language's declared names in the prose too: `event_id` in Rust, `eventId` in TypeScript, `EventId` in C#.
- Use each language's types. Mark 64-bit TypeScript integers as bigint, as in `u64 (bigint)`.
- Check every name you write against the SDK. Earlier prompts named a C# method that does not exist.

**Add nothing that no model needs.**

- Never add a sentence just to get past a check or a content filter. If a valid answer fails, fix the grader.
- The runner already tells models to output one source file and to copy identifiers exactly. New prompts need not repeat this.
- Start with `Write a SpacetimeDB backend module in <Rust|TypeScript|C#> that ...`. Migration tasks and a few others vary, so follow the nearest task.

## Reference answers

- Write each answer the way the current guidelines in the root `skills/` directory teach it.
- Use the current API in all three languages.
- Include only what the prompt asks for. Extra reducers or tables make the schema check compare against the wrong thing.

## Graders

Start from `default_schema_parity_scorers`, then add a behaviour check. `scenario` in `src/eval/scenario.rs` is the usual tool. A schema check alone passes a module whose reducers do nothing.

**A check must fail a module that does nothing.**

- Assert the rows or replies you expect, including intermediate state.
- Confirm that setup rows exist before a delete, and that rows outside the delete survive.
- Assert explicit expected rows. Agreement between two copies of the reference proves little.
- Cover concurrency with real parallel calls where the task promises it, as `t_087` does.

**A check must pass every design the prompt allows.**

- Trace each assertion to a sentence in the prompt. If none exists, delete the assertion. State the requirement instead only when the task cannot test its concept without it.
- Accept any rejection unless the prompt quotes the error text. A plain panic or throw that rolls back is a valid rejection.
- Test error delivery to the caller in its own task, not as part of atomicity or lease tasks.
- Accept extra indexes and extra HTTP routes. Ignore case and underscores in names. The shared helpers already do this, so use them instead of hand-written SQL.
- Do not treat a transport error as an expected rejection.

## Validate

Check only the tasks you touched. A prompt-only change does not need a full reference run.

1. Run the crate's tests, `cargo fmt` and clippy.
2. Run the reference answers for each language, after the local builds in `README.md`. This makes no model calls and uploads nothing.

```text
cargo run -p xtask-llm-benchmark --bin llm_benchmark --locked -- run --lang rust --goldens-only --dry-run --skip-task-catalog-upload --tasks t_NNN
```

3. Prove the behaviour check can fail. Temporarily replace the answer with a version that does nothing, then with one that breaks a single requirement. Run the same command and confirm both fail. Restore the answer. The schema check compares an answer with itself here, so only the behaviour check can catch these.
4. Run real models without uploading. Use the `Periodic LLM benchmarks` workflow with `model_set=explicit`, one or two cheap models, the `tasks` input and `dry_run=true`. Enabling, dispatching and disabling that workflow needs the owner's approval.
5. Read the failures and sort them with the table below. Do not assume a failure is the model's fault.

## Whose fault is a failure?

| What happened | What to do |
| --- | --- |
| Valid current-API code fails | Our bug. Fix the grader. |
| Outdated or wrong code passes | Our bug. Fix the grader. |
| Many models fail the same way on a requirement | Suspect the prompt. Check that it states the requirement and that the requirement is real. |
| The prompt names something wrong | Our bug. Fix the prompt. |
| Models use outdated syntax | A model failure. Keep it. |
| Guidelines mode misses a pattern that models need | Fix the root `skills/` files, only where a run shows a benefit. `llms.md` is generated from them. |
| The SDK itself is at fault | Keep the failure and report it to the core team. Do not work around it. |
| The provider rejects the content | Keep the failure. Do not reword the prompt to avoid it. |
| The provider or the setup fails | The batch is refused. Fix the cause and rerun the batch. |

## Changing a published task

- Scores from before the change no longer compare with scores after it. The README should say so when the suite changes.
- Website uploads only add rows. Old rows stay on Trends until someone deletes them from the admin page.
- A refused batch uploads nothing, so a rerun has to cover the whole batch. Group changes so that one rerun covers them all.
- Weakening or deleting a check must be stated in the pull request, as the root `AGENTS.md` requires.
1 change: 1 addition & 0 deletions tools/xtask-llm-benchmark/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
@AGENTS.md
Loading
Loading