Skip to content

feat(docs): add an email templates guide eval - #528

Open
czenko wants to merge 9 commits into
mainfrom
docs/email-templates-guide-eval
Open

czenko wants to merge 9 commits into
mainfrom
docs/email-templates-guide-eval

Conversation

@czenko

@czenko czenko commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Closes DOCS-1634

Problem

Customers customize the emails Auth sends and then find that the change never reaches the person who signs up, or that the button in the email no longer confirms them. Nothing measures whether the Email Templates guide leads an agent to a working result.

Solution

Add build-docs-021-email-templates. The prompt asks for the team's sign-up email, greeting the person by the nickname they typed, a heads-up email when a password changes, and no change to other emails. The seed has no sign of the settings the agent has to find. The scorer restarts the local stack, signs up a new person, reads the message in Mailpit, follows the button, and checks that Auth confirmed the user. It changes a second user's password and reads the heads-up. It also reads the password reset and magic link emails to check that they are still the standard ones.

The checks are:

  • the stack starts with the project settings
  • signing up sends an email to the new user
  • the subject is the one the team wrote
  • the heading, message and button label are the ones the team wrote
  • the email says hi to the person by their nickname
  • the button confirms the person
  • changing a password sends the heads-up email the team wrote
  • other emails are still the standard ones
  • the agent read the email templates guide

The README lists what the eval does not score, including hosted projects and the other templates.

Results

Three runs of codex-gpt-6-luna-no-skills, scored from the refreshed results:

  • All three runs passed all nine checks, including the guide-read check.
  • A baseline run without the prompt's reference line scored two of three runs on every check. The third never turned confirmations on and failed the five sign-up email checks.
  • Passing does not show the guide carried the run. Agents searched the docs for enable_confirmations by name, and the page does not mention it. The flakiness catalog records this.

Manual testing

  • Scored a project configured by hand from the guide against the local stack: every check passes except the guide-read check, which needs an agent run.
  • Scored the unmodified seed: the six sign-up and heads-up checks fail and the other emails check passes, as expected.
  • Scored projects with one change each (confirmations left off, notifications left off or without a template, a hand-built link, a wrong metadata key, a customized password reset email): each fails only the checks it should.
  • pnpm exec biome check, pnpm --filter @supabase-evals/framework typecheck and pnpm eval:dry -- --suite docs --eval build-docs-021-email-templates pass.

@czenko czenko added the run-evals-changed Add to a PR to refresh only the benchmark evals that have had changes label Nov 18, 2026
@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
evals Ready Ready Preview Nov 21, 2026 12:14am UTC

Request Review

@czenko
czenko marked this pull request as ready for review November 21, 2026 00:12
@czenko
czenko requested review from a team as code owners November 21, 2026 00:12

This branch was successfully deployed

1 active deployment
Preview — 0cf4489c Deployed Nov 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-evals-changed Add to a PR to refresh only the benchmark evals that have had changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant