Skip to content

feat: add supabase.sh (docs over SSH) as an experiment - #540

Open
Akshat-Mishra101 wants to merge 5 commits into
supabase:mainfrom
Akshat-Mishra101:supabase-sh-experiment
Open

Akshat-Mishra101 wants to merge 5 commits into
supabase:mainfrom
Akshat-Mishra101:supabase-sh-experiment

Conversation

@Akshat-Mishra101

Copy link
Copy Markdown

Adds supabase.sh as something the harness can measure, plus two Claude Code experiments that use it.

Why: in the published benchmark results, Claude Code checks the docs in 6–8 of 69 runs without skills, while Codex and Grok check them in nearly every run. supabase.sh is a one-line CLAUDE.md or AGENTS.md setup aimed at exactly that, and I couldn't find any measurements of it, so I ran some.

Changes

  • Counts ssh supabase.sh <command> calls as a new supabase_sh docs source in docs-results.ts. Pages under /supabase/docs map to their supabase.com urls. A call counts as reading a page only when it just printed pages (cat, sed, head). It handles pipes, chained commands, bash -lc wrappers and shell loops, and skips ssh supabase.sh agents. Adds a label in the results UI.
  • Adds an instructions option to ExperimentConfig. It appends text to CLAUDE.md (Claude Code) or AGENTS.md (other CLI agents) in the agent's workspace before the run, the way a user sets up supabase.sh. The prompt is unchanged.
  • Adds openssh-client and supabase.sh's pinned host key to the sandbox image, so agents can connect without a host-key prompt.
  • Adds claude-code-sonnet-5.5-supabase-sh and claude-code-sonnet-5.5-no-skills-supabase-sh, using the output of ssh supabase.sh agents as instructions (experiments/supabase-sh.ts).

Results

I ran these locally on a Mac with Docker: Sonnet 5.5 without skills, with and without supabase.sh, 3 runs each, graded by the existing scorers. The tasks are every benchmark eval where claude-code-sonnet-5.5-no-skills failed at least once in the published results.

Eval Without With Used supabase.sh
build-database-001-migrate-postgres-to-supabase 0/3 3/3 3/3
build-functions-005-dual-auth-user-secret 1/3 2/3 3/3
build-dataapi-002-restock-alert-report 0/3 2/3 0/3
build-cli-003-pg-cron-queue-workflow 1/3 1/3 3/3
build-dataapi-001-relational-report 0/3 0/3 0/3
investigate-auth-001-deleted-user-access 0/3 0/3 3/3
resolve-database-001-migration-history-mismatch 1/3 0/3 2/3
build-cli-002-declarative-schema 3/3 0/3 3/3
Total 6/24 8/24 17/24
  • Docs lookups: 5/24 runs without (MCP search_docs), 17/24 with (all supabase.sh). Claude never looked anything up on the two dataapi "finish the script" tasks, so the dataapi-002 change isn't a docs effect.
  • build-database-001: without docs, Claude ran supabase init and supabase start for a second project in a temp folder, which broke the eval's own stack. With supabase.sh, it followed the "migrating to Supabase" guide into the existing project.
  • build-cli-002 looks like a scorer issue. With supabase.sh, Claude read guides/local-development/declarative-database-schemas.md, which says to generate migrations with supabase db schema declarative sync and "Don't use supabase db diff here". It did that, and the column was applied, but the supabase db diff used to generate the migration check fails. Should the scorer also accept db schema declarative sync? Happy to send a separate PR.
  • Caveats: 3 runs per task, so 6 vs 8 is within noise. It's one model, run locally rather than in CI.

Also found: when an agent starts its own stack under a new project id (build-database-001 without docs), the harness doesn't stop it. Locally, that stack keeps ports 54321 and up, and every later local-stack run fails at supabase start with "port is already allocated". CI doesn't hit this since each run gets its own sandbox. I worked around it by removing those stacks between runs. I'm happy to look at a fix if useful.

Question: these are in experiments/benchmark/, so run-evals would pick them up. If you'd rather not add benchmark experiments, I can drop the experiment files and keep the lookup counting and the instructions option.

Testing

  • pnpm test, pnpm typecheck and pnpm format:check pass.
  • New tests are in docs-results.test.ts (supabase.sh calls) and agent-instructions.test.ts.
  • The counter picks up all 36 supabase.sh commands Claude Code ran in an earlier pilot.
pnpm eval -- --suite benchmark --experiment claude-code-sonnet-5.5-no-skills,claude-code-sonnet-5.5-no-skills-supabase-sh --eval build-database-001-migrate-postgres-to-supabase --runs 3

Record `ssh supabase.sh <command>` shell calls as a supabase_sh docs source. Pages under /supabase/docs map to their supabase.com urls, and a call is marked as delivering page text only when it just printed pages (cat, sed, head).
Appends an experiment's instructions to CLAUDE.md (Claude Code) or AGENTS.md (other CLI agents) in the agent's workspace before it starts, the way a user sets up a tool like supabase.sh. The prompt is unchanged.
Installs openssh-client and pins supabase.sh's host key, so agents can read the docs over SSH without a host-key prompt.
Adds the `ssh supabase.sh agents` snippet as project instructions, with and without skills, to compare against the existing Sonnet 5.5 experiments.
A call after `do`, `then` or `else` now counts, and a path built from a shell variable or glob (`$f.md`) no longer becomes a page.
@vercel

vercel Bot commented Nov 21, 2026

Copy link
Copy Markdown

@Akshat-Mishra101 is attempting to deploy a commit to the Supabase Team on Vercel.

A member of the Team first needs to authorize it.

@Akshat-Mishra101
Akshat-Mishra101 marked this pull request as ready for review November 21, 2026 08:31
@Akshat-Mishra101
Akshat-Mishra101 requested a review from a team as a code owner November 21, 2026 08:31

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant