Repository navigation
Bradley/stackbench test - #5766
Draft
bradleyshep wants to merge 763 commits into
Draft
bradleyshep wants to merge 763 commits into
bradleyshep wants to merge 763 commits into
Conversation
Contributor
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
bradleyshep
force-pushed
the
bradley/stackbench-test
branch
from
August 31, 2026 17:38
6a1eb7d to
dc1a416
Compare
bradleyshep
force-pushed
the
bradley/stackbench-test
branch
from
September 7, 2026 20:22
32d3018 to
837c1f7
Compare
Grading: - A refusal counts only after the same request was accepted for an authorized actor (same named action, or a completed captured write); otherwise the check is inconclusive (route-unproven). A proven 404 counts. 317a, 613b and 620b gain authorized proof steps. - Signup claim probe: later writes after the patched target are recorded instead of voiding the probe; a repeated target still does. The claimant signs in if the app left it signed out. Per-account URL segments and post/redirect/get responses no longer void write capture. - expectNumber keeps a matching read that started in time; element counts and retried checkout state must hold across a second sample. - 612d's signed-out reader opens support history when a link exists. - Staff-role contract states that the select shows the saved role and that promoted accounts stay listed; stale-session 621d mutations per stack. Maintenance: - Break the auth-request-patch/browser-action-executors import cycle. - Add expectWarehouseInventory to the action fixture; fix no-unsafe-finally. - Pin Claude Code 2.1.284 (knows claude-opus-5-5) in the agent image. - Re-pin calibration hashes; progression evidence and reuse are cleared for one fresh full qualification.
Route proofs are keyed by the capabilities' actor accessor. The grader built a new accessor for every step, so a proof recorded by one step was invisible to the next and correct refusals became route-unproven. Found by regrading saved study apps with the candidate controller.
317a's admin route proof re-submitted the setup's pending Webcam/West restock, so an app that rejects duplicates refused the proof, and the customer's refusal of the same duplicate never measured authorization. The admin now proves the route with Desk Lamp/East and the customer attempts a fresh Keyboard/East restock with every field refilled. The slice reuse test compiled the live calibration with identity-free reuse, which is valid only over sliced evidence; give it that evidence.
618a only forged the buyer's username on an item the buyer had already reviewed, so an app that trusts the claimed username but allows one review per buyer refused it as a duplicate and passed. The buyer now also owns an unreviewed Desk Lamp, and a forged review of it must be refused and absent.
The reference audit read pack budgets from the recipe release document, which has none, so every timing-only run failed its runtime check and pack-budget recommend refused the evidence. Pass the plan's pack definitions with the release. The access-control pack exceeded its bound after new route proofs, so its budget is unmeasured until it is measured again.
The scheduled-restock contract makes the POST body the same object as data-action-input: item and warehouse names with integer quantity and delay. The MongoDB reference exposed string numbers and posted object ids, so a direct authorized call was a validation error. 317a's new admin route proof exposed it; the customer refusal had hidden it.
Budgets follow the recorded policy: twice the largest clean-reference runtime per pack across the four stacks, measured by timing-only qualification on c0f6def with all four stacks in parallel.
Restore master's TableRef type and remove its diagnostic test (Core tier; the test only covered that change). Remove the CLI build script's skills folder watch; master already watches each skill file. Restore the concepts, rust-server, csharp-server, and cpp-server skills, which no StackBench profile delivers.
…test # Conflicts: # codex-plugin/plugins/spacetimedb/skills/cli/SKILL.md # skills/cli/SKILL.md
…ench StackBench now uses master's skills, TypeScript SDK and build scripts. The connection URL fix and the true skill corrections go to master as their own product PRs. The controller image clears the cached CLI and codegen build outputs so the embedded commit matches the source, which the build-script lines did before. The StackBench line-ending rules move into its own .gitattributes, without the rules for evidence folders that no longer exist. Deleted test: crates/bindings-typescript/tests/ws_url.test.ts covered only the URL change removed here; it returns with that fix's own PR.
Restores the cli, typescript-client and typescript-server skill edits that state product facts, and removes the ones that prescribed an app design: the paragraphs on presence, sign-out and application accounts. Fixes three mistakes: owners can read private tables, AnonymousViewCtx has no sender, and the view example now indexes the column it filters. Adds that dev defaults to --delete-data=on-conflict, restores the React 18 bigint warning, and restores "Test connectivity".
Spacetime sandboxes no longer install /deps/spacetime-dev, which only the unused neutral-managed-dev profile documented; agents run spacetime dev as the prompt says. The neutral-managed-dev profile and its workflow doc go with it, and spacetime dev's --ready-file returns to master. The CLI wrapper now copies the run's read-only login into a fresh HOME for each call. The CLI saves config during ordinary commands, and master's atomic_write retries forever on a read-only folder; the copy lets those saves succeed while no change outlives the call. fs-utils returns to master. Deleted test: tests/managed-dev.test.ts covered only the removed helper.
cp -R keeps the read-only permissions of the run login, so the copied home was read-only too and config saves still hung. Tested in a container as an unprivileged user with master's CLI: config-saving commands finish, a server added in one call is gone in the next, and the run login is unchanged; the same command against the read-only home without the wrapper hangs.
A process the agent left running could swap a path for a link while the controller copied or restored files, redirecting the write anywhere the controller can write. Every agent-user process now stops when a coding session ends, before the next session starts (so the graded application does not outlive grading), and whenever StackBench stops a hosted application, not only the processes listening on its port. Files read or written while the agent or its application may still run are opened without following links and must be regular files: the transcript archive, Antigravity's finished-answer read, and the package-lock write during accepted startup.
Grading resets the database before each stateful check but left the last checks' rows in place, so the next level's or repair's session could read data the hidden checks created; a transcript search found the L1 support check's ticket in about 500 later sessions across all five stacks. Upgrade and repair sessions now start from an empty database, which the application seeds as on any first start; Spacetime republishes the graded module with its data deleted.
A restored workspace may have no installed dependencies, so publishing the graded module could fail and end the run as a harness failure. The pre-session reset now uses the repair reset, which deletes the module; the agent publishes it again as on a first start.
A clone of the repository searched with grep -r, or read through a script the audit does not parse, returns tools/stack-bench paths without the command naming them. Any tool output containing that path now counts as StackBench source; nothing StackBench gives an agent contains it, and no archived transcript does. Refused calls still return nothing and stay refused.
Coding sessions keep public HTTP and HTTPS but are refused GitHub's published ranges (api.github.com/meta, 2026-11-19), where StackBench's checks are public. Grading allows only replies to connections made into the attempt, loopback and the npm cache, and refuses name lookups, so an application cannot send out what it recorded of the hidden checks. Grading mode starts with run-suite; coding mode returns when the next session starts, after the graded application has stopped. Each change replaces the table in one nft transaction. The npm cache's address goes into the shared hosts file so installs during grading need no lookup. Archived apps (196, all stacks) need nothing outside the npm cache at startup; only cosmetic Google Fonts and photos stop loading while graded. Checked with real nft in a user-defined Docker network: coding loads a public site and refuses github.com and raw.githubusercontent.com; grading refuses all outbound traffic and lookups while the application still answers an inbound request.
…rs too Reference, regrade and fault-injection runs prepare the build container with --prepare-only and skip the coding session, so the stop and the return to coding mode now run before that early exit. Every --prepare-only caller prepares a container before putting work in it, never during grading.
A check still running at the limit fails, and so do the checks after it, even when restarts or recoveries used the time. The doc said harness failures never become application failures; this is the exception. No check in a kept study has reached the limit since it was added, so it stays a stated limitation rather than a grader change.
Spacetime and Convex restart their servers; PostgreSQL, MongoDB and Supabase restart the application server. Supabase's services keep running, so state held only in Edge Function memory is not tested. None of the 38 archived Supabase apps has an Edge Function, so this stays a stated limitation rather than a grader change.
On Spacetime the database reset republishes the app's module, and a failed publish is a harness failure, while an app on another stack that cannot start on an empty database fails. No archived run on any stack ended in a reset failure of this kind, so it stays a stated limitation; published results will report excluded runs per stack and model.
A failed stop or start of the app server fails the check unless the control was refused, the container was lost, or the command could not run or timed out, and the app must run in one process group on its port. No archived run has an app-control failure, so it stays a stated limitation rather than a grader change.
When an application stopped answering after an earlier feature passed, the grade bundle's score counted that pass but the conversion compared against current work only and threw, losing the level. The conversion now accepts the bundle score when it covers every current pass and at most the earlier passes graded in the same scope.
Private notes now live outside the repository. The dashboard check-guide and form-input browser tests write their evidence under the system temp folder, and local-notes leaves the skip lists, the Docker ignore file and .gitignore.
The reference apps' start.sh files are allowed as a reviewed exception: they provide the start script StackBench requires of every app it grades. The Linux CLI export becomes npm run build:linux-cli, the same docker build the shell script ran.
…'s own Spacetime page docs/fairness.md lists what every stack shares, why only Spacetime receives SDK skills, which Spacetime build runs, and where grading differs by stack. The Spacetime setup no longer says skills are included, which was false in the no-skills profiles. StackBench's workflow page for spacetime dev now gets the same disclosure check as the prompt; only the public skills are exempt. Prompting now states the network rules for coding and grading.
Both launch through sh, so the version lookup ran sh --version and recorded unknown. Each provider now names its own CLI for the lookup.
Claude Code 2.1.329, Codex 0.171.0, Grok 1.0.89 and Antigravity 1.5.13. The Grok checksum matches on both of xAI's hosts; the Antigravity checksum is from Google's release manifest.
Both appear in the account's Antigravity model list at medium effort.
Antigravity 1.5 renews a Google sign-in into a file with the account's email in place of the ID token. The sign-in is now known by its refresh token, which renewal keeps. The test of the ID token's issuer is removed with that check.
Convex moved its GitHub organization from get-convex to convex, and the old image name now refuses pulls. The pinned digest is unchanged.
Backend revision 3d585751a2, built 2026-11-20, replacing the 2026-09-15 build.
Antigravity 1.5 asks Gemini for up to 262,144 output tokens, above the 128,000 a plan could set. The default for a plan that sets none stays 128,000.
Postgres 19 and MongoDB 9. Supabase uses the component versions from its own self-hosting compose file of 2026-11-18.
Every complete event of a Gemini stream carries the running token count. When Google cuts a stream off, the broker now charges that count instead of the request's full reservation, which at Antigravity 1.5's 262,144-token limit is about $6 a request. A response cut off before its first complete event still counts at the reservation.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
This PR adds Stack Bench, a reproducible benchmark for coding agents building applications against SpacetimeDB, PostgreSQL, and MongoDB.
Benchmark definitions and grading
passed,failed,inconclusive, orharness_failure.Ecommerce benchmark
Runs, repairs, and campaigns
CLI, dashboard, and appliance
SpacetimeDB TypeScript SDK
TableReffrom an anonymous readonly type alias to a readonly interface with the same members.TableRefis used incorrectly.ConnectionManagerreconnects to reuse the identity token previously issued by the server.TableRefdiagnostics.SpacetimeDB CLI
spacetime devto retain configuration changes made during direct-login publishing.SpacetimeDB skills
API and ABI breaking changes
This PR changes the public Rust signature of:
spacetimedb_cli::publish::exec_from_entryThe function now accepts
&mut Configinstead of takingConfigby value. External Rust callers must pass a mutable reference. The API-breaking label should be applied unless the previous entry point is preserved through a compatibility wrapper.The exported TypeScript
TableRefdeclaration changes from a type alias to an interface while retaining the same readonly member contract.Automatic TypeScript client reconnection also changes behavior by presenting the identity token from the previous successful connection.
Expected complexity level and risk
3 / 5
Most of the implementation is isolated under
tools/stack-bench. The principal risks are benchmark correctness, evidence integrity, concurrent resource isolation, interrupted-run recovery, Docker cleanup, and credential handling.The TypeScript reconnect change and
spacetimedb-clihelper signature are the only changes outside the benchmark and skill directories.Testing
TableRefregression tests: 4/4 passed.cargo check -p spacetimedb-cli.exec_from_entryRust signature.