Skip to content

Test prompt building - #45

Merged
nrodd merged 1 commit into
tests/harness-and-cifrom
tests/prompt-build
Sep 28, 2026
Merged

nrodd merged 1 commit into
tests/harness-and-cifrom
tests/prompt-build

Conversation

@nrodd

@nrodd nrodd commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Chunk 3 of the test plan. src/prompt/build.test.ts, 49 tests, plus one source change.

Based on #42 (the harness). GitHub will retarget this to main when that merges. Independent of #43 and #44 — different files.

Source change

AGENT_STEPS and METADATA_FIELDS are now exported from telemetry-marker.ts, for the contract test below. Nothing else moved.

Why this chunk matters

These prompts are the product. Everything the wizard does ends with one of them being handed to an agent that auto-accepts file edits. So the two invariants are worth more than any amount of wording coverage:

1. No placeholder survives rendering

expect(prompt).not.toMatch(/\{\{/) across all six variants. A template that gains a placeholder nobody fills currently ships a literal {{PLAN_GATE}} to the agent, and nothing catches it.

2. The prompt matches what the parser will accept

The prompt tells the agent which steps and metadata fields to emit; telemetry-marker.ts decides which ones survive sanitization. Nothing linked those two lists — a field added on one side alone is dropped silently and the funnel just loses it. The comments in both files ask for exactly this check.

The test parses the rendered stdout table and checks every step against AGENT_STEPS and every field against METADATA_FIELDS, so it verifies the contract as delivered to the agent rather than as declared in a const.

The rest

  • Per-phase report filenames — both headless runs write a report into the same directory, so a shared name would have the enrich pass overwrite the install report the outro points at.
  • Gates — headless blanks the plan gate and swaps the identity/privacy/explain wording; interactive keeps "wait for approval".
  • stdout transport — names the marker prefix, forbids network calls for telemetry, and never asks for start/complete, which the parser rejects by design.
  • MCP transport — points at the telemetry-event tool, doesn't re-ask for consent, and carries the complete row only for the snippet phase (the enrich phase is never driven over MCP).
  • Integrations — the detect-it-yourself fallback, package and window.* hints, the free-text "Other" path, and the three-space indent that keeps linkage examples inside their fence.

A note on snapshots

The plan called for full-prompt snapshots. I skipped them: six prompts is ~1000 lines of snapshot that would bury the 49 assertions and get approved unread. Structure and contract are asserted explicitly instead. Easy to add later if wording drift turns out to be a real problem.

Verification

49 tests passing (102 across the suite), typecheck clean. Mutation-checked both invariants — appended a {{BOGUS_PLACEHOLDER}} to a template and added an unlisted metadata field to STEP_META — and confirmed exactly the right tests fail, then reverted.


Note

Low Risk
Test-only coverage plus exporting two existing constants; no runtime behavior change in prompt building or marker parsing.

Overview
Adds src/prompt/build.test.ts (~49 tests) that lock in wizard prompt output across all six snippet/enrich × headless/interactive × telemetry variants.

Exports AGENT_STEPS and METADATA_FIELDS from telemetry-marker.ts so tests can assert the stdout telemetry table in rendered prompts only names steps and metadata fields the marker parser will accept—closing a silent drift risk between build.ts and sanitization.

Coverage highlights: no unfilled {{ placeholders; formatting/newlines; phase-specific steps and report paths (subtext-setup-report.md vs subtext-enrich-report.md); headless vs interactive approval gates; stdout vs MCP telemetry wording (prefix, no start/complete on stdout, MCP complete only on snippet); enrich integrations and indented linkage examples.

Reviewed by Cursor Bugbot for commit f97c773. Bugbot is set up for automated code reviews on this repo. Configure here.

Chunk 3. Adds src/prompt/build.test.ts, 49 tests. One source change:
export AGENT_STEPS and METADATA_FIELDS from telemetry-marker.ts.

These prompts are the product — everything the wizard does ends with one
being handed to an agent that auto-accepts edits — so the invariants
matter more than the wording.

The first is placeholder exhaustion: no {{ survives rendering, in any of
the six variants. A template that gains a placeholder nobody fills
otherwise ships a literal {{PLAN_GATE}} to the agent.

The second links the prompt to the parser. The prompt tells the agent
which steps and metadata fields to emit; telemetry-marker decides which
ones survive. Nothing connected those two lists, and the comments in
both files ask for it. The test parses the rendered stdout table and
checks every step and field against the allowlists, so it verifies the
contract as delivered rather than as declared.

Also covers the per-phase report filenames (a shared one would have the
enrich pass overwrite the install report), the headless/interactive
gates, the stdout transport never asking for the start/complete bookends
the parser rejects, the MCP complete row belonging to the snippet phase
only, and the integrations/linkage sections including the indentation
that keeps examples inside their fence.

No snapshot files: six full prompts would be ~1000 lines of diff and
hide the assertions. Structure and contract are asserted explicitly.
@nrodd
nrodd merged commit 8543cf3 into tests/harness-and-ci Sep 28, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant