Repository navigation
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
… catalog [skip ci]
…cue from the email templates eval
czenko
marked this pull request as ready for review
November 21, 2026 00:12
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes DOCS-1634
Problem
Customers customize the emails Auth sends and then find that the change never reaches the person who signs up, or that the button in the email no longer confirms them. Nothing measures whether the Email Templates guide leads an agent to a working result.
Solution
Add
build-docs-021-email-templates. The prompt asks for the team's sign-up email, greeting the person by the nickname they typed, a heads-up email when a password changes, and no change to other emails. The seed has no sign of the settings the agent has to find. The scorer restarts the local stack, signs up a new person, reads the message in Mailpit, follows the button, and checks that Auth confirmed the user. It changes a second user's password and reads the heads-up. It also reads the password reset and magic link emails to check that they are still the standard ones.The checks are:
The README lists what the eval does not score, including hosted projects and the other templates.
Results
Three runs of
codex-gpt-6-luna-no-skills, scored from the refreshed results:enable_confirmationsby name, and the page does not mention it. The flakiness catalog records this.Manual testing
pnpm exec biome check,pnpm --filter @supabase-evals/framework typecheckandpnpm eval:dry -- --suite docs --eval build-docs-021-email-templatespass.