Repository navigation
Test prompt building - #45
Merged
Merged
Conversation
Chunk 3. Adds src/prompt/build.test.ts, 49 tests. One source change:
export AGENT_STEPS and METADATA_FIELDS from telemetry-marker.ts.
These prompts are the product — everything the wizard does ends with one
being handed to an agent that auto-accepts edits — so the invariants
matter more than the wording.
The first is placeholder exhaustion: no {{ survives rendering, in any of
the six variants. A template that gains a placeholder nobody fills
otherwise ships a literal {{PLAN_GATE}} to the agent.
The second links the prompt to the parser. The prompt tells the agent
which steps and metadata fields to emit; telemetry-marker decides which
ones survive. Nothing connected those two lists, and the comments in
both files ask for it. The test parses the rendered stdout table and
checks every step and field against the allowlists, so it verifies the
contract as delivered rather than as declared.
Also covers the per-phase report filenames (a shared one would have the
enrich pass overwrite the install report), the headless/interactive
gates, the stdout transport never asking for the start/complete bookends
the parser rejects, the MCP complete row belonging to the snippet phase
only, and the integrations/linkage sections including the indentation
that keeps examples inside their fence.
No snapshot files: six full prompts would be ~1000 lines of diff and
hide the assertions. Structure and contract are asserted explicitly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chunk 3 of the test plan.
src/prompt/build.test.ts, 49 tests, plus one source change.Source change
AGENT_STEPSandMETADATA_FIELDSare now exported fromtelemetry-marker.ts, for the contract test below. Nothing else moved.Why this chunk matters
These prompts are the product. Everything the wizard does ends with one of them being handed to an agent that auto-accepts file edits. So the two invariants are worth more than any amount of wording coverage:
1. No placeholder survives rendering
expect(prompt).not.toMatch(/\{\{/)across all six variants. A template that gains a placeholder nobody fills currently ships a literal{{PLAN_GATE}}to the agent, and nothing catches it.2. The prompt matches what the parser will accept
The prompt tells the agent which steps and metadata fields to emit;
telemetry-marker.tsdecides which ones survive sanitization. Nothing linked those two lists — a field added on one side alone is dropped silently and the funnel just loses it. The comments in both files ask for exactly this check.The test parses the rendered stdout table and checks every step against
AGENT_STEPSand every field againstMETADATA_FIELDS, so it verifies the contract as delivered to the agent rather than as declared in a const.The rest
start/complete, which the parser rejects by design.telemetry-eventtool, doesn't re-ask for consent, and carries thecompleterow only for the snippet phase (the enrich phase is never driven over MCP).window.*hints, the free-text "Other" path, and the three-space indent that keeps linkage examples inside their fence.A note on snapshots
The plan called for full-prompt snapshots. I skipped them: six prompts is ~1000 lines of snapshot that would bury the 49 assertions and get approved unread. Structure and contract are asserted explicitly instead. Easy to add later if wording drift turns out to be a real problem.
Verification
49 tests passing (102 across the suite),
typecheckclean. Mutation-checked both invariants — appended a{{BOGUS_PLACEHOLDER}}to a template and added an unlisted metadata field toSTEP_META— and confirmed exactly the right tests fail, then reverted.Note
Low Risk
Test-only coverage plus exporting two existing constants; no runtime behavior change in prompt building or marker parsing.
Overview
Adds
src/prompt/build.test.ts(~49 tests) that lock in wizard prompt output across all six snippet/enrich × headless/interactive × telemetry variants.Exports
AGENT_STEPSandMETADATA_FIELDSfromtelemetry-marker.tsso tests can assert the stdout telemetry table in rendered prompts only names steps and metadata fields the marker parser will accept—closing a silent drift risk betweenbuild.tsand sanitization.Coverage highlights: no unfilled
{{placeholders; formatting/newlines; phase-specific steps and report paths (subtext-setup-report.mdvssubtext-enrich-report.md); headless vs interactive approval gates; stdout vs MCP telemetry wording (prefix, nostart/completeon stdout, MCPcompleteonly on snippet); enrich integrations and indented linkage examples.Reviewed by Cursor Bugbot for commit f97c773. Bugbot is set up for automated code reviews on this repo. Configure here.