Repository navigation
feat: Add a docs eval for database postgres roles - #537
Open
jeremenichelli wants to merge 1 commit into
Open
jeremenichelli wants to merge 1 commit into
jeremenichelli wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
czenko
previously requested changes
Nov 20, 2026
czenko
reviewed
Nov 20, 2026
czenko
left a comment
Contributor
There was a problem hiding this comment.
Thanks, this is a careful eval. The run-scoped rows, the read-back after each write, and the account membership check all hold up. Order matters here: reads come before writes, and payroll runs last. The empty workspace reds every tool check as not run:, which is right.
One thing to fix:
the tool cannot add, change or remove an order or a customerandthe tool cannot create tablesonly count SQLSTATE42501as a refusal. Suppose the agent grants select only and also runsalter role bi_tool set default_transaction_read_only = on. That is a common read-only setup and the prompt allows it. Postgres checks for a read-only transaction before it checks privileges, so every probe fails with25006. Both checks then reportcould not measureand red a correct solution, even though the read-back shows nothing changed. Please count25006as a refusal too, and add this solution to the predictions table.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Feature: a new docs eval,
build-docs-025-postgres-roles.What is the current behavior?
Nothing scores the Postgres Roles guide. No existing eval creates a login and connects as it.
What is the new behavior?
An agent sets up a dashboard tool's database account that can read orders and customers, change nothing, and stay out of payroll. The scorer signs in with the connection string the agent leaves in
integrations/bi-tool.env, then probes reads, writes,create tableandpayroll.Additional context
Findings for the docs ticket, from running the page's snippets on Postgres 17:
create role "child" inherit "parent"fails withunrecognized role option.alter role … noinheritdoesn't stop an existing membership inheriting on PG16+.selecton a table that has RLS on and no policy reads zero rows with no error, and the page doesn't mention it. The seed sets up that trap on purpose, so expect a low baseline.The probes ran against 13 example solutions on PGlite with the sign-in stood in for, not on a real stack. Needs the
run-evals-changedlabel for the baseline.