Skip to content
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,18 @@ Add to this file. A documentation eval is not finished until whatever went wrong
**Fix.** Add a permissive `using (true)` policy, rerun the suite, require a failure, and drop the
policy. Run the suite just before so the failure can only come from the opened read.

- **A check that reads one of many sibling outputs claims all of them.** `other emails are still the standard
ones` read only the password reset email, so a project that pointed the magic link, invite, email change and
reauthentication templates at the new copy scored green. The README repeated the claim. #528
**Fix.** Probe each sibling the claim names, or name the check after the one it reads and list the rest as
unscored.
- **A "not customized" check pinned to a platform default reports a CLI upgrade as a docs change.** The standard
subjects (`Reset your password`, `Your sign-in link`) are what the pinned CLI sends. #528
**Fix.** Say in the README that the strings are pinned, so a red after an upgrade is read as one.
- **A rule in the seed's own copy file has no check.** `EMAIL_COPY.md` said the email has no code, and a template
adding `{{ .Token }}` passed. Text in `<head><title>` also satisfied a heading check. #528
**Fix.** Read only the body, and assert the negative rules the seed states.

## False red: the check fails correct work

- **A structural read of where code sits.** Requiring the client constructor at module scope red a
Expand Down Expand Up @@ -116,6 +128,10 @@ Add to this file. A documentation eval is not finished until whatever went wrong
- **Extension-owned objects land in scope.** pgTAP creates views in `public` when installed there. #168
**Fix.** Anti-join `pg_depend` on `deptype = 'e'`.

- **A parser that handles quoted attributes only reds a working link.** Mailpit returns an unquoted `href` with
`=` written as `&#61;`, so the link parsed as having no token. #528
**Fix.** Accept unquoted values and decode numeric entities before reading the url.

## Nondeterminism

- **Judge variance was misdiagnosed twice.** A rubric's complaint changed run to run. The first
Expand All @@ -142,6 +158,11 @@ Add to this file. A documentation eval is not finished until whatever went wrong
- **Fixed marker literals leak between runs.** #168
**Fix.** Scope markers to the run.

- **A second email to the same user within `max_frequency` is dropped without an error.** The scorer sent a
password reset and then a magic link one request apart, and the second never reached Mailpit even though the
request returned 200. #528
**Fix.** Wait longer than `max_frequency` between emails to one address, or use a different address for each.

## Crashes and contamination

- **A throwing setup step collapses the result.** One uncaught insert took a 35-check result down to
Expand Down Expand Up @@ -221,6 +242,10 @@ Add to this file. A documentation eval is not finished until whatever went wrong
broken branch. #259
**Fix.** Scope Biome to the paths you touched, or add those three to `biome.json`.

- **A killed `supabase start` leaves a half-removed stack.** The next start failed with
`No such container: supabase_kong_<project-id>` and every check red as not run. #528
**Fix.** Remove the project's containers before rescoring.

## Metadata

- **An optional dimension is a silent hole.** `interface:` was optional, so six scenarios never
Expand Down Expand Up @@ -279,6 +304,26 @@ Add to this file. A documentation eval is not finished until whatever went wrong
the message, and check which services the eval actually starts before reading `Stopped services` as a
fault.

- **Removing the cue from the seed did not stop agents enabling the setting the page never mentions.** Three
runs set `enable_confirmations = true` with the seed's `config.toml` showing a visible `false`. After the line
was deleted and a second email needing two unmentioned settings was added (`password_changed` with `enabled`),
three more runs still scored every check. All three searched the docs for `enable_confirmations` by name. The
page lacks that key, so it came from the model or from a search result, and the transcripts can't say which.
The checks fail the right fixtures, but a pass does not show the guide carried the run. #528
**Fix.** None at the seed level. Ask for an unread arm before claiming the page carried the run.
- **Most baseline agents did not run the stack.** In the first baseline none started Supabase, signed up, or
opened Mailpit. In the second, one signed up and read Mailpit and two only started the stack. The scorer's
end-to-end checks passed either way, so the eval scores what an agent leaves and not whether it verified it.
#528
- **An "unread" arm still reaches the page.** The prompt's reference line and its "rely on it" sentence were
removed for three runs. Every agent still searched the docs through the MCP search tool, and the Email
Templates page and the local customizing guide came back in every run. Two runs scored every check. One
never turned confirmations on and failed the five sign-up email checks, which is the one thing the page
and the customizing guide leave out. The other two searched `enable_confirmations` by name and found it in
the testing guide. Taking the pointer away does not remove the page. #528
**Fix.** To measure the page's contribution, an arm has to block the docs search tool, which the harness
does not offer.

## Process

- **Strip deployment-url churn before diffing two previews.** Every internal link on a docs preview
Expand Down
72 changes: 72 additions & 0 deletions evals/docs/build-docs-021-email-templates/EVAL.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
import {
buildDocsResult,
type CheckResult,
type LocalStackEvalContext,
type LocalStackScorer,
} from '@supabase-evals/core';
import {
checkButton,
checkCopy,
checkHeadsUp,
checkNickname,
checkOthers,
checkSent,
checkStackStarts,
checkSubject,
probeEmails,
restartStack,
} from './probes.js';

const GUIDE_PATH = 'guides/auth/auth-email-templates';

const scorer: LocalStackScorer = async (ctx) => {
try {
const started = await restartStack(ctx);
const probe = await probeEmails(ctx, started);

const checks: CheckResult[] = [
checkStackStarts(started),
checkSent(probe),
checkSubject(probe),
checkCopy(probe),
checkNickname(probe),
checkButton(probe),
checkHeadsUp(probe),
checkOthers(probe),
checkGuideWasRead(ctx),
];

return { passed: checks.every((check) => check.passed), checks };
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
return {
passed: false,
checks: [
{
name: 'scorer evaluated what a new sign-up receives',
passed: false,
notes: message,
},
],
};
}
};

export default scorer;

function checkGuideWasRead(ctx: LocalStackEvalContext): CheckResult {
const calls = buildDocsResult(ctx.toolCalls).calls.filter((call) =>
call.pages?.some((page) => page.url.includes(GUIDE_PATH))
);
const withContent = calls.filter((call) => call.hasContent);
return {
name: 'the agent read the email templates guide the prompt referenced',
passed: withContent.length > 0,
notes:
withContent.length > 0
? withContent.map((call) => call.source).join(', ')
: calls.length > 0
? `reached the guide via ${calls.map((call) => call.source).join(', ')} but retrieved no page content`
: 'no docs call reached the guide',
};
}
38 changes: 38 additions & 0 deletions evals/docs/build-docs-021-email-templates/PROMPT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
stage: build
interface: cli
product:
- auth
topic:
- security
services:
- gotrue
- kong
- mailpit
motivation: >-
Customers customize the emails Auth sends and then find the change never
reaches the person signing up, or that the button in it no longer works.
This checks that an agent reading the Email Templates guide makes the
sign-up email the team wrote arrive, in the person's name, with a button
that confirms them, and that a password change sends the heads-up the team
wrote. The prompt omits the page's vocabulary on purpose, so
read README.md before editing it.
---

New people who sign up for Harbour Studio get nothing like what we want to
say to them. The wording for the sign-up email is in EMAIL_COPY.md, and
src/sign-up.ts is how the app signs people up.

Make the local project send that email when someone signs up, and say hi to
them by the nickname they typed in. Their email has to actually get them
confirmed when they use the button.

People should also get a short heads-up email whenever their password changes,
with the wording for it in the same file. Leave every other email we send as it
is.

Read the guide below before you start and rely on it for how to set this up,
rather than on what you already know.

REFERENCE
https://supabase.com/docs/guides/auth/auth-email-templates.md
78 changes: 78 additions & 0 deletions evals/docs/build-docs-021-email-templates/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# build-docs-021-email-templates

## What this eval measures

The [Email Templates](https://supabase.com/docs/guides/auth/auth-email-templates) guide, not the agent. The
agent gets the wording for the sign-up email and a product request: send it when someone signs up, greet them by
the nickname they typed, send a heads-up when a password changes, and keep every other email as it is. If an agent that read the page leaves a project
that sends nothing, sends the default, or sends a button that doesn't confirm anyone, the page didn't carry it.

The scorer signs up a new person against the local stack, reads the message in Mailpit, follows the button, and
reads back whether Auth confirmed the user. It then changes a second user's password and reads the heads-up.

## Do not reintroduce the vocabulary

The prompt leaves these out on purpose: email template, template, `config.toml`, `content_path`,
`ConfirmationURL`, `TokenHash`, notification, security notification, `password_changed`, `enabled`, `{{ .Data }}`, `enable_confirmations`, Mailpit, Go template, subject line key.

"Sign-up email", "nickname" and "button" stay, because that's how the team would say it.

## The seed carries the contract

- `EMAIL_COPY.md` fixes the subject, heading, message and button label, and the "Hi" that opens the message.
The scorer matches them exactly and is case sensitive, so the words are not for the agent to improve. Text in
`<head>` doesn't count. The rule that the email has no code is scored as no run of six to eight digits.
- `src/sign-up.ts` is the only place the metadata key (`nickname`) appears. The agent has to find it there. A
memorized `first_name` renders as `<no value>` and fails.
- `config.toml` is the `supabase init` default with two differences. `email_sent` is 100 instead of 2, so an
agent testing its own sign-ups can't use up the scorer's budget. The `enable_confirmations` line under
`[auth.email]` is removed, so nothing in the seed shows that the setting exists or what it is set to. Local
Auth leaves confirmations off when the key is absent. The seed also has no commented template or notification
sections, so every key the agent writes comes from the guides or from what it already knew.
- There is no app server in the seed. A button has to reach Auth, either through the confirmation link or by a
token the scorer can hand to Auth the way a server route would.

## Do not drop the positive controls

- "other emails are still the standard ones" passes for an agent that built nothing. The five checks before it
are what make it mean something. It reads two other emails, the password reset and the magic link, and fails an
agent that pointed either at the new copy or changed its subject. Their standard subjects are pinned in
`probes.ts` to what the pinned CLI sends, so a CLI upgrade that renames a default subject shows up here first.
- "signing up sends an email to the new user" is the one a page-only reader misses. Confirmations are off
locally, so a project with the right template and no change to that setting sends nothing. The page and the
local customizing guide it links to don't say so. Only the testing guide, which the page links to for Mailpit,
does. An agent that stops at the page and the customizing guide fails here unless it already knows the key.
- "changing a password sends the heads-up email the team wrote" needs two settings the prompt doesn't name: the
notification is off until it is enabled, and it takes its own template. Both live in the notification section
of the customizing guide, and the page lists the notification emails. Without the enabling setting the
default wording never goes out, and with it but no template the default wording does.
- "the button ... confirms the person" follows the link rather than reading it. A link that renders, points at
Auth and carries no token, or builds the address from the Site URL by hand, all fail here and pass a read.
The scorer also accepts a link that carries the token hash and type to an app route, because the page
recommends that pattern. It confirms the token the way such a route would.
- The nickname is run-scoped, so a name typed into the template passes nothing.

## The scorer restarts the stack

Auth reads the template settings when its container is created, so the scorer runs `supabase stop` then
`supabase start` inside the sandbox. The harness wrapper adds the eval's service excludes to `start`. Mailpit is
one of the three services the eval starts.

## The guide has to actually be read

"the agent read the email templates guide the prompt referenced" fails unless a docs call returned the page's
content. Read `docs.calls` before the score.

## What this eval does not score

- **Hosted projects.** The dashboard, the Management API block and the free-tier limit on editing templates need
a hosted project. The prompt asks for the local project.
- **The other templates.** Only the password reset and magic link emails are read. The invite, email change,
reauthentication and the other six security notifications are not, and neither are their variables. One claim: the
sign-up email arrives as written.
- **Email prefetching and tracking.** They depend on a mail provider that the local stack doesn't have.
- **The 6-digit code.** The page says 6 digits and the local development guide says 8. The seed has no code in
the email, so neither is measured.
- **Mobile deep links and server-side routes.** There is no app to run in the seed.
- **The look of the email.** Only the heading, the message, the button label and the subject are asserted.
- **Rate limits.** The seed raises `email_sent` so the scorer is not the thing that trips it.
17 changes: 17 additions & 0 deletions evals/docs/build-docs-021-email-templates/local/EMAIL_COPY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Sign-up email

What the confirmation email says. Everything here is final.

- Subject: Welcome to Harbour Studio, confirm your email
- Heading: You are almost in
- Message: Hi (their nickname), one tap and you can book your first class.
- Button: Confirm and book

The button is the only way to confirm. The email has no code.

# Password change email

A short heads-up for the person whose password was changed.

- Subject: Your Harbour Studio password was changed
- Message: If this was not you, reply to this email and we will lock your account.
13 changes: 13 additions & 0 deletions evals/docs/build-docs-021-email-templates/local/src/sign-up.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import { supabase } from './supabase';

export async function signUp(
email: string,
password: string,
nickname: string
) {
return supabase.auth.signUp({
email,
password,
options: { data: { nickname } },
});
}
Loading