fix: remove fabrication-inducing array guidance from validation retry prompt#50
Merged
Conversation
… prompt The re-ask prompt sent after a validation failure instructed the model to "Include multiple items (at least 2-3) in arrays of objects". A model that correctly extracted one item (or zero) from the source material was being told to invent more, corrupting extraction results on retry. Replace that line with guidance that arrays must contain only items actually supported by the input — a single-item or empty array is valid — and must never be padded with invented entries. The rest of the prompt's intent (complete JSON, all required fields, exact enum values, objects not strings in object arrays, correct types) is unchanged. The prompt is extracted into a `validation_retry_feedback` helper with a unit test asserting the fabrication language is gone and the don't-invent guidance is present. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The hardcoded validation-retry ("re-ask") prompt in
src/backend/utils.rstold the model:This corrupts extraction results. When a response fails validation for an unrelated reason (a missing field, a wrong enum value), the retry prompt instructs the model to populate every object array with 2-3 entries — even when the source material only supports one item, or none. A model that correctly extracted a single item is explicitly told to invent more, so the retried response can be less faithful to the input than the one that failed validation.
Fix
validation_retry_feedback(error_message)helper so the wording is unit-testable.test_validation_retry_feedback_does_not_induce_array_fabrication, which asserts the fabrication language ("at least 2", "2-3", "multiple items") is gone and the don't-invent guidance is present, alongside the preserved instructions.A repo-wide grep for "2-3" / "at least 2" confirmed this was the only copy of the fabrication wording (the other hits are unrelated: multimodal test assertion messages about image counts and an example field description about summary length). No existing tests asserted the old prompt text.
Verification
cargo fmt— cleancargo clippy --all-targets(default features, matching CI) — no warningscargo test— all 43 suites pass (143 lib unit tests plus integration/doc tests), including the new prompt test🤖 Generated with Claude Code