Repository navigation
feat: add supabase.sh (docs over SSH) as an experiment - #540
Open
Akshat-Mishra101 wants to merge 5 commits into
Open
Akshat-Mishra101 wants to merge 5 commits into
Akshat-Mishra101 wants to merge 5 commits into
Conversation
Record `ssh supabase.sh <command>` shell calls as a supabase_sh docs source. Pages under /supabase/docs map to their supabase.com urls, and a call is marked as delivering page text only when it just printed pages (cat, sed, head).
Appends an experiment's instructions to CLAUDE.md (Claude Code) or AGENTS.md (other CLI agents) in the agent's workspace before it starts, the way a user sets up a tool like supabase.sh. The prompt is unchanged.
Installs openssh-client and pins supabase.sh's host key, so agents can read the docs over SSH without a host-key prompt.
Adds the `ssh supabase.sh agents` snippet as project instructions, with and without skills, to compare against the existing Sonnet 5.5 experiments.
A call after `do`, `then` or `else` now counts, and a path built from a shell variable or glob (`$f.md`) no longer becomes a page.
|
@Akshat-Mishra101 is attempting to deploy a commit to the Supabase Team on Vercel. A member of the Team first needs to authorize it. |
Akshat-Mishra101
marked this pull request as ready for review
November 21, 2026 08:31
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds supabase.sh as something the harness can measure, plus two Claude Code experiments that use it.
Why: in the published benchmark results, Claude Code checks the docs in 6–8 of 69 runs without skills, while Codex and Grok check them in nearly every run. supabase.sh is a one-line CLAUDE.md or AGENTS.md setup aimed at exactly that, and I couldn't find any measurements of it, so I ran some.
Changes
ssh supabase.sh <command>calls as a newsupabase_shdocs source indocs-results.ts. Pages under/supabase/docsmap to their supabase.com urls. A call counts as reading a page only when it just printed pages (cat,sed,head). It handles pipes, chained commands,bash -lcwrappers and shell loops, and skipsssh supabase.sh agents. Adds a label in the results UI.instructionsoption toExperimentConfig. It appends text to CLAUDE.md (Claude Code) or AGENTS.md (other CLI agents) in the agent's workspace before the run, the way a user sets up supabase.sh. The prompt is unchanged.openssh-clientand supabase.sh's pinned host key to the sandbox image, so agents can connect without a host-key prompt.claude-code-sonnet-5.5-supabase-shandclaude-code-sonnet-5.5-no-skills-supabase-sh, using the output ofssh supabase.sh agentsas instructions (experiments/supabase-sh.ts).Results
I ran these locally on a Mac with Docker: Sonnet 5.5 without skills, with and without supabase.sh, 3 runs each, graded by the existing scorers. The tasks are every benchmark eval where
claude-code-sonnet-5.5-no-skillsfailed at least once in the published results.search_docs), 17/24 with (all supabase.sh). Claude never looked anything up on the two dataapi "finish the script" tasks, so the dataapi-002 change isn't a docs effect.supabase initandsupabase startfor a second project in a temp folder, which broke the eval's own stack. With supabase.sh, it followed the "migrating to Supabase" guide into the existing project.guides/local-development/declarative-database-schemas.md, which says to generate migrations withsupabase db schema declarative syncand "Don't usesupabase db diffhere". It did that, and the column was applied, but thesupabase db diff used to generate the migrationcheck fails. Should the scorer also acceptdb schema declarative sync? Happy to send a separate PR.Also found: when an agent starts its own stack under a new project id (build-database-001 without docs), the harness doesn't stop it. Locally, that stack keeps ports 54321 and up, and every later local-stack run fails at
supabase startwith "port is already allocated". CI doesn't hit this since each run gets its own sandbox. I worked around it by removing those stacks between runs. I'm happy to look at a fix if useful.Question: these are in
experiments/benchmark/, sorun-evalswould pick them up. If you'd rather not add benchmark experiments, I can drop the experiment files and keep the lookup counting and theinstructionsoption.Testing
pnpm test,pnpm typecheckandpnpm format:checkpass.docs-results.test.ts(supabase.sh calls) andagent-instructions.test.ts.pnpm eval -- --suite benchmark --experiment claude-code-sonnet-5.5-no-skills,claude-code-sonnet-5.5-no-skills-supabase-sh --eval build-database-001-migrate-postgres-to-supabase --runs 3