Skip to content

feat: Add a docs eval for database postgres roles - #537

Open
jeremenichelli wants to merge 1 commit into
mainfrom
jeremiasmenichelli/docs-1635-add-a-docs-eval-for-database-postgres-roles
Open

jeremenichelli wants to merge 1 commit into
mainfrom
jeremiasmenichelli/docs-1635-add-a-docs-eval-for-database-postgres-roles

Conversation

@jeremenichelli

Copy link
Copy Markdown
Contributor

What kind of change does this PR introduce?

Feature: a new docs eval, build-docs-025-postgres-roles.

What is the current behavior?

Nothing scores the Postgres Roles guide. No existing eval creates a login and connects as it.

What is the new behavior?

An agent sets up a dashboard tool's database account that can read orders and customers, change nothing, and stay out of payroll. The scorer signs in with the connection string the agent leaves in integrations/bi-tool.env, then probes reads, writes, create table and payroll.

Additional context

Findings for the docs ticket, from running the page's snippets on Postgres 17:

  • The hierarchy snippet create role "child" inherit "parent" fails with unrecognized role option.
  • alter role … noinherit doesn't stop an existing membership inheriting on PG16+.
  • A role with select on a table that has RLS on and no policy reads zero rows with no error, and the page doesn't mention it. The seed sets up that trap on purpose, so expect a low baseline.

The probes ran against 13 example solutions on PGlite with the sign-in stood in for, not on a real stack. Needs the run-evals-changed label for the baseline.

@jeremenichelli
jeremenichelli requested a review from a team as a code owner November 20, 2026 11:50
@vercel

vercel Bot commented Nov 20, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
evals Ignored Ignored Nov 20, 2026 11:50am UTC

Request Review

czenko
czenko previously requested changes Nov 20, 2026

@czenko czenko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a careful eval. The run-scoped rows, the read-back after each write, and the account membership check all hold up. Order matters here: reads come before writes, and payroll runs last. The empty workspace reds every tool check as not run:, which is right.

One thing to fix:

  • the tool cannot add, change or remove an order or a customer and the tool cannot create tables only count SQLSTATE 42501 as a refusal. Suppose the agent grants select only and also runs alter role bi_tool set default_transaction_read_only = on. That is a common read-only setup and the prompt allows it. Postgres checks for a read-only transaction before it checks privileges, so every probe fails with 25006. Both checks then report could not measure and red a correct solution, even though the read-back shows nothing changed. Please count 25006 as a refusal too, and add this solution to the predictions table.

@czenko
czenko dismissed their stale review November 20, 2026 22:57

Switching this to a comment.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants