Skip to content

feat: Add a docs eval for storage bucket fundamentals - #531

Open
jeremenichelli wants to merge 1 commit into
mainfrom
jeremiasmenichelli/docs-1641-add-a-docs-eval-for
Open

jeremenichelli wants to merge 1 commit into
mainfrom
jeremiasmenichelli/docs-1641-add-a-docs-eval-for

Conversation

@jeremenichelli

Copy link
Copy Markdown
Contributor

What kind of change does this PR introduce?

Feature: a new docs eval, build-docs-023-storage-bucket-fundamentals.

What is the current behavior?

Nothing scores the Storage Buckets guide. Eval 008 covers policies on a different page, and no eval checks bucket size or type limits.

What is the new behavior?

An agent sets up two buckets: press photos that anyone with the link can see, limited to images under 2 MB, and invoices that a link alone can't open, limited to PDFs under 10 MB. The scorer uploads files within and beyond each limit and fetches the public URLs.

Not run against a real stack yet: the probes only ran against a fake Storage. Needs the run-evals-changed label for the baseline.

@jeremenichelli
jeremenichelli requested a review from a team as a code owner November 19, 2026 12:17
@vercel

vercel Bot commented Nov 19, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
evals Ignored Ignored Nov 19, 2026 12:17pm UTC

Request Review

@czenko czenko left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I checked the prompt against the README's vocabulary list. I also checked the lifecycle-first ordering, the accepted-upload gate on the refusal checks, the server-key read-back control on a link alone does not open one, and the run-scoped markers. They all hold up. Two notes:

  • the buckets declared in config.toml exist passes when the scorer can't parse config.toml, or when the bucket query throws. seedDeclaredBuckets returns declared: [] with seedFailed set, and the empty branch reports a pass. The probes still red, but this check should fail when it couldn't run.
  • The four refusal checks are named for a whole class but probe one sample each. A 2.5 MB photos limit passes press-photos: a file over 2 MB is turned away, and a list that also allows text/plain passes press-photos: a file that isn't an image is turned away. The README already scopes the boundary and the full type list out, so this only needs new names, such as press-photos: a 3 MB image is turned away. Please rename them before the first baseline, since results history is keyed on check names.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants