Skip to content

Bradley/stackbench test - #5766

Draft
bradleyshep wants to merge 763 commits into
masterfrom
bradley/stackbench-test
Draft

bradleyshep wants to merge 763 commits into
masterfrom
bradley/stackbench-test

Conversation

@bradleyshep

Copy link
Copy Markdown
Contributor

Description of Changes

This PR adds Stack Bench, a reproducible benchmark for coding agents building applications against SpacetimeDB, PostgreSQL, and MongoDB.

Benchmark definitions and grading

  • Adds versioned tracks, levels, feature packs, specification packs, recipes, prompts, checks, and dependency declarations.
  • Supports selecting complete recipes, individual packs, or individual checks.
  • Separates what the agent is asked to build from what the finished application is evaluated against.
  • Adds typed scenario actions for browser interaction, authenticated server calls, concurrent operations, direct database writes, reconnects, reloads, and service lifecycle operations.
  • Records each check as passed, failed, inconclusive, or harness_failure.
  • Keeps the declared score denominator fixed when evidence is missing or inconclusive.
  • Adds reference, mutation, and null-control qualification:
    • reference applications for all three stacks;
    • source-bound mutation manifests;
    • exact mutation-target and collateral-failure validation;
    • blank-application controls;
    • versioned calibrations and promotion catalogs.
  • Adds immutable identities for recipes, prompts, fixtures, mutations, checks, runtime images, adapters, and evidence.

Ecommerce benchmark

  • Adds ecommerce L1 coverage for accounts, sessions, storefronts, carts, checkout, purchases, warehouses, reviews, authorization, accounting, concurrency, live updates, persistence, reconnects, and direct database writes.
  • Adds ecommerce L2 coverage for fulfilment, shipping, warehouse transfers, cancellations, returns, historical pricing, operational views, recommendations, authorization, stock conservation, and refund accounting.
  • Adds complete ecommerce reference applications for MongoDB, PostgreSQL, and SpacetimeDB.
  • Promotes qualified ecommerce L1 modular 2.4.
  • Keeps ecommerce L2 1.4 as the promoted L2 release.
  • Adds L2 1.5 as a draft cumulative successor based on L1 2.4:
    • 76 checks;
    • 117 available points;
    • static mutation coverage for all 74 scored checks on every supported stack;
    • candidate reference fixtures, mutation manifests, and calibration;
    • live qualification still required before promotion.

Runs, repairs, and campaigns

  • Adds bounded correction rounds with rollback when a repair loses previously established evidence.
  • Stops cumulative execution when a lower level remains incomplete.
  • Adds source checkpoints and operator-granted continuation rounds without rewriting the original result.
  • Adds campaign manifests containing exact stacks, repetitions, parallelism, recipes, models, pricing, prompt treatments, and correction budgets.
  • Supports concurrent attempts of the same or different stacks using isolated ports, databases, modules, containers, work directories, leases, and result directories.
  • Adds durable campaign scheduling, retries, locking, interruption recovery, reconciliation, and cleanup.
  • Preserves invalid and interrupted attempts instead of presenting them as comparison data.
  • Adds reproducible JSON and HTML reports with scores, coverage, cost, tokens, elapsed time, correction rounds, raw evidence, screenshots, videos, traces, and prompts.

CLI, dashboard, and appliance

  • Adds Stack Bench CLI commands for runs, campaigns, preflight, qualification, composition, repair continuations, recovery, release verification, and reporting.
  • Adds a Dockerized local dashboard that reads and controls the same durable campaign state as the CLI.
  • Adds a Linux/amd64 Docker appliance with pinned controller and build images, preflight checks, resource leases, reference qualification, release identity verification, and recovery tooling.
  • Adds model-free reference adapters and campaign trials for testing orchestration without provider spend.
  • Adds retained grader stdout/stderr and structured browser-initialization evidence.
  • Hardens process cleanup, destructive filesystem operations, campaign locks, recovery authority, credential redaction, and failed teardown handling.
  • Prevents local notes, generated reports, media, runtime state, and mutation scratch files from entering controller image builds.

SpacetimeDB TypeScript SDK

  • Changes exported TableRef from an anonymous readonly type alias to a readonly interface with the same members.
  • Produces shorter, named TypeScript diagnostics when a TableRef is used incorrectly.
  • Changes automatic ConnectionManager reconnects to reuse the identity token previously issued by the server.
  • Keeps explicit connection rebuilds bound to the token supplied by the caller.
  • Adds unit tests for reconnect token reuse and TableRef diagnostics.
  • Adds an opt-in live-host reconnect identity test.

SpacetimeDB CLI

  • Changes spacetime dev to retain configuration changes made during direct-login publishing.
  • Reuses that authenticated configuration for log streaming and subsequent rebuilds.

SpacetimeDB skills

  • Expands the TypeScript, Rust, and C# server skills with examples and guidance for client visibility filters, procedures, HTTP endpoints, indexes, and uniqueness constraints.
  • Updates the TypeScript client skill’s authentication and reconnect guidance.

API and ABI breaking changes

This PR changes the public Rust signature of:

spacetimedb_cli::publish::exec_from_entry

The function now accepts &mut Config instead of taking Config by value. External Rust callers must pass a mutable reference. The API-breaking label should be applied unless the previous entry point is preserved through a compatibility wrapper.

The exported TypeScript TableRef declaration changes from a type alias to an interface while retaining the same readonly member contract.

Automatic TypeScript client reconnection also changes behavior by presenting the identity token from the previous successful connection.

Expected complexity level and risk

3 / 5

Most of the implementation is isolated under tools/stack-bench. The principal risks are benchmark correctness, evidence integrity, concurrent resource isolation, interrupted-run recovery, Docker cleanup, and credential handling.

The TypeScript reconnect change and spacetimedb-cli helper signature are the only changes outside the benchmark and skill directories.

Testing

  • Stack Bench host suite: 602 tests, 599 passed, 0 failed, 3 Windows-specific skips.
  • Composition validation: 51 packs, 13 recipes, and 677 selected checks.
  • All calibration manifests compile.
  • All registered reference fixtures pass source and registry validation.
  • Scenario validation completes with 0 errors.
  • All 27 prompt snapshots match.
  • All 4 definition goldens match.
  • TypeScript reconnect and TableRef regression tests: 4/4 passed.
  • cargo check -p spacetimedb-cli.
  • Ecommerce L1 2.4 null control: 0/58.
  • Ecommerce L1 2.4 references: 58/58 on MongoDB, PostgreSQL, and SpacetimeDB.
  • Ecommerce L1 2.4 mutation qualification:
    • MongoDB: 44/44 defects caught;
    • PostgreSQL: 45/45 defects caught;
    • SpacetimeDB: 46/46 defects caught.
  • Controller build-context regression coverage.
  • Review whether to preserve the previous exec_from_entry Rust signature.
  • Run the appliance preflight and smoke test on Linux/amd64.
  • Run live Docker qualification for ecommerce L2 1.5 before promotion.
  • Run the opt-in TypeScript reconnect test against a disposable live host.

@github-advanced-security github-advanced-security AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

@bradleyshep
bradleyshep force-pushed the bradley/stackbench-test branch from 6a1eb7d to dc1a416 Compare August 31, 2026 17:38
@bradleyshep
bradleyshep force-pushed the bradley/stackbench-test branch from 32d3018 to 837c1f7 Compare September 7, 2026 20:22
Grading:
- A refusal counts only after the same request was accepted for an
  authorized actor (same named action, or a completed captured write);
  otherwise the check is inconclusive (route-unproven). A proven 404 counts.
  317a, 613b and 620b gain authorized proof steps.
- Signup claim probe: later writes after the patched target are recorded
  instead of voiding the probe; a repeated target still does. The claimant
  signs in if the app left it signed out. Per-account URL segments and
  post/redirect/get responses no longer void write capture.
- expectNumber keeps a matching read that started in time; element counts
  and retried checkout state must hold across a second sample.
- 612d's signed-out reader opens support history when a link exists.
- Staff-role contract states that the select shows the saved role and that
  promoted accounts stay listed; stale-session 621d mutations per stack.

Maintenance:
- Break the auth-request-patch/browser-action-executors import cycle.
- Add expectWarehouseInventory to the action fixture; fix no-unsafe-finally.
- Pin Claude Code 2.1.284 (knows claude-opus-5-5) in the agent image.
- Re-pin calibration hashes; progression evidence and reuse are cleared for
  one fresh full qualification.
Route proofs are keyed by the capabilities' actor accessor. The grader built
a new accessor for every step, so a proof recorded by one step was invisible
to the next and correct refusals became route-unproven. Found by regrading
saved study apps with the candidate controller.
317a's admin route proof re-submitted the setup's pending Webcam/West
restock, so an app that rejects duplicates refused the proof, and the
customer's refusal of the same duplicate never measured authorization.
The admin now proves the route with Desk Lamp/East and the customer
attempts a fresh Keyboard/East restock with every field refilled.

The slice reuse test compiled the live calibration with identity-free
reuse, which is valid only over sliced evidence; give it that evidence.
618a only forged the buyer's username on an item the buyer had already
reviewed, so an app that trusts the claimed username but allows one
review per buyer refused it as a duplicate and passed. The buyer now
also owns an unreviewed Desk Lamp, and a forged review of it must be
refused and absent.
The reference audit read pack budgets from the recipe release document,
which has none, so every timing-only run failed its runtime check and
pack-budget recommend refused the evidence. Pass the plan's pack
definitions with the release.

The access-control pack exceeded its bound after new route proofs, so
its budget is unmeasured until it is measured again.
The scheduled-restock contract makes the POST body the same object as
data-action-input: item and warehouse names with integer quantity and
delay. The MongoDB reference exposed string numbers and posted object
ids, so a direct authorized call was a validation error. 317a's new
admin route proof exposed it; the customer refusal had hidden it.
Budgets follow the recorded policy: twice the largest clean-reference
runtime per pack across the four stacks, measured by timing-only
qualification on c0f6def with all four stacks in parallel.
Restore master's TableRef type and remove its diagnostic test (Core tier; the test only covered that change). Remove the CLI build script's skills folder watch; master already watches each skill file. Restore the concepts, rust-server, csharp-server, and cpp-server skills, which no StackBench profile delivers.
…test

# Conflicts:
#	codex-plugin/plugins/spacetimedb/skills/cli/SKILL.md
#	skills/cli/SKILL.md
…ench

StackBench now uses master's skills, TypeScript SDK and build scripts. The
connection URL fix and the true skill corrections go to master as their own
product PRs. The controller image clears the cached CLI and codegen build
outputs so the embedded commit matches the source, which the build-script
lines did before. The StackBench line-ending rules move into its own
.gitattributes, without the rules for evidence folders that no longer exist.

Deleted test: crates/bindings-typescript/tests/ws_url.test.ts covered only the
URL change removed here; it returns with that fix's own PR.
Restores the cli, typescript-client and typescript-server skill edits that
state product facts, and removes the ones that prescribed an app design: the
paragraphs on presence, sign-out and application accounts. Fixes three
mistakes: owners can read private tables, AnonymousViewCtx has no sender, and
the view example now indexes the column it filters. Adds that dev defaults to
--delete-data=on-conflict, restores the React 18 bigint warning, and restores
"Test connectivity".
Spacetime sandboxes no longer install /deps/spacetime-dev, which only the
unused neutral-managed-dev profile documented; agents run spacetime dev as the
prompt says. The neutral-managed-dev profile and its workflow doc go with it,
and spacetime dev's --ready-file returns to master.

The CLI wrapper now copies the run's read-only login into a fresh HOME for each
call. The CLI saves config during ordinary commands, and master's atomic_write
retries forever on a read-only folder; the copy lets those saves succeed while
no change outlives the call. fs-utils returns to master.

Deleted test: tests/managed-dev.test.ts covered only the removed helper.
cp -R keeps the read-only permissions of the run login, so the copied home was read-only too and config saves still hung. Tested in a container as an unprivileged user with master's CLI: config-saving commands finish, a server added in one call is gone in the next, and the run login is unchanged; the same command against the read-only home without the wrapper hangs.
A process the agent left running could swap a path for a link while the
controller copied or restored files, redirecting the write anywhere the
controller can write. Every agent-user process now stops when a coding
session ends, before the next session starts (so the graded application does
not outlive grading), and whenever StackBench stops a hosted application, not
only the processes listening on its port.

Files read or written while the agent or its application may still run are
opened without following links and must be regular files: the transcript
archive, Antigravity's finished-answer read, and the package-lock write during
accepted startup.
Grading resets the database before each stateful check but left the last checks' rows in place, so the next level's or repair's session could read data the hidden checks created; a transcript search found the L1 support check's ticket in about 500 later sessions across all five stacks. Upgrade and repair sessions now start from an empty database, which the application seeds as on any first start; Spacetime republishes the graded module with its data deleted.
A restored workspace may have no installed dependencies, so publishing the graded module could fail and end the run as a harness failure. The pre-session reset now uses the repair reset, which deletes the module; the agent publishes it again as on a first start.
A clone of the repository searched with grep -r, or read through a script the audit does not parse, returns tools/stack-bench paths without the command naming them. Any tool output containing that path now counts as StackBench source; nothing StackBench gives an agent contains it, and no archived transcript does. Refused calls still return nothing and stay refused.
Coding sessions keep public HTTP and HTTPS but are refused GitHub's published
ranges (api.github.com/meta, 2026-11-19), where StackBench's checks are
public. Grading allows only replies to connections made into the attempt,
loopback and the npm cache, and refuses name lookups, so an application cannot
send out what it recorded of the hidden checks. Grading mode starts with
run-suite; coding mode returns when the next session starts, after the graded
application has stopped. Each change replaces the table in one nft
transaction. The npm cache's address goes into the shared hosts file so
installs during grading need no lookup.

Archived apps (196, all stacks) need nothing outside the npm cache at startup;
only cosmetic Google Fonts and photos stop loading while graded. Checked with
real nft in a user-defined Docker network: coding loads a public site and
refuses github.com and raw.githubusercontent.com; grading refuses all outbound
traffic and lookups while the application still answers an inbound request.
…rs too

Reference, regrade and fault-injection runs prepare the build container with --prepare-only and skip the coding session, so the stop and the return to coding mode now run before that early exit. Every --prepare-only caller prepares a container before putting work in it, never during grading.
A check still running at the limit fails, and so do the checks after it, even when restarts or recoveries used the time. The doc said harness failures never become application failures; this is the exception. No check in a kept study has reached the limit since it was added, so it stays a stated limitation rather than a grader change.
Spacetime and Convex restart their servers; PostgreSQL, MongoDB and Supabase restart the application server. Supabase's services keep running, so state held only in Edge Function memory is not tested. None of the 38 archived Supabase apps has an Edge Function, so this stays a stated limitation rather than a grader change.
On Spacetime the database reset republishes the app's module, and a failed publish is a harness failure, while an app on another stack that cannot start on an empty database fails. No archived run on any stack ended in a reset failure of this kind, so it stays a stated limitation; published results will report excluded runs per stack and model.
A failed stop or start of the app server fails the check unless the control was refused, the container was lost, or the command could not run or timed out, and the app must run in one process group on its port. No archived run has an app-control failure, so it stays a stated limitation rather than a grader change.
When an application stopped answering after an earlier feature passed, the grade bundle's score counted that pass but the conversion compared against current work only and threw, losing the level. The conversion now accepts the bundle score when it covers every current pass and at most the earlier passes graded in the same scope.
Private notes now live outside the repository. The dashboard check-guide and form-input browser tests write their evidence under the system temp folder, and local-notes leaves the skip lists, the Docker ignore file and .gitignore.
The reference apps' start.sh files are allowed as a reviewed exception: they provide the start script StackBench requires of every app it grades. The Linux CLI export becomes npm run build:linux-cli, the same docker build the shell script ran.
…'s own Spacetime page

docs/fairness.md lists what every stack shares, why only Spacetime receives SDK skills, which Spacetime build runs, and where grading differs by stack. The Spacetime setup no longer says skills are included, which was false in the no-skills profiles. StackBench's workflow page for spacetime dev now gets the same disclosure check as the prompt; only the public skills are exempt. Prompting now states the network rules for coding and grading.
Both launch through sh, so the version lookup ran sh --version and recorded unknown. Each provider now names its own CLI for the lookup.
Claude Code 2.1.329, Codex 0.171.0, Grok 1.0.89 and Antigravity 1.5.13. The Grok checksum matches on both of xAI's hosts; the Antigravity checksum is from Google's release manifest.
Both appear in the account's Antigravity model list at medium effort.
Antigravity 1.5 renews a Google sign-in into a file with the account's email in place of the ID token. The sign-in is now known by its refresh token, which renewal keeps. The test of the ID token's issuer is removed with that check.
Convex moved its GitHub organization from get-convex to convex, and the old image name now refuses pulls. The pinned digest is unchanged.
Backend revision 3d585751a2, built 2026-11-20, replacing the 2026-09-15 build.
Antigravity 1.5 asks Gemini for up to 262,144 output tokens, above the 128,000 a plan could set. The default for a plan that sets none stays 128,000.
Postgres 19 and MongoDB 9. Supabase uses the component versions from its own self-hosting compose file of 2026-11-18.
Every complete event of a Gemini stream carries the running token count. When Google cuts a stream off, the broker now charges that count instead of the request's full reservation, which at Antigravity 1.5's 262,144-token limit is about $6 a request. A response cut off before its first complete event still counts at the reservation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants