Skip to content

Commit 4efbed0

Browse files
codexclaude
andcommitted
fix(dialogue): she answered one of two questions, twice, and nothing noticed
"give me one concrete example of a preposition doing more work than it should. and separately — do you actually enjoy that, or is 'interesting' a word you reach for because it's safe?" The reply gave the example and said nothing whatever about enjoyment. Earlier the same session: "you dodged half of it. I asked two things and you answered one." Two independent causes. 1. DETECTION. analyze_prompt_shape scored that message at question_parts=1. Every candidate it computes is per-LINE or per-verb-list, and the message is one line: _INTERROGATIVE_LINE_RE requires the LINE to open with what/why/do/is and this one opens with "give", while _CONNECTOR_RE wants "then"/"also" plus a directive verb rather than "and separately". So the prompt was never told the turn was compound and neither was the voice budget. People type several sentences on one line constantly; asks are now counted per SENTENCE, an ask being a sentence that ends in a question mark or opens with a directive verb. 2. ENFORCEMENT. question_parts and requires_single_reply_coverage already existed and already shaped the prompt — "This prompt contains multiple asks (2 detected)" — but only the COUNT survived analysis, so nothing could hold the reply against the question. Every other requirement in validate_dialogue_response has a matching violation that goes through repair; coverage was requested in the prompt and never checked, so dropping half of a question cost nothing. PromptShape now keeps the segment TEXTS, the contract carries them, and an ask the reply never engages with raises unanswered_question_part. Deliberately hard to trigger, because a false positive burns a regeneration on a reply that was fine: only on a turn the contract already marked multi-ask, only for asks carrying at least two distinctive words (so "why?" can never be flagged), and only when the overlap is ZERO. One shared content word counts as engaged — this catches the half ignored outright, not the half answered briefly. The repair block quotes the dropped question back. "Answer every part" was already in the prompt when this happened; naming the one that went unanswered is the part that is new. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent a1fe21d commit 4efbed0

4 files changed

Lines changed: 325 additions & 2 deletions

File tree

core/phases/dialogue_policy.py

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -397,6 +397,83 @@ def _requires_non_generic_aura_voice(contract: object | None) -> bool:
397397
)
398398

399399

400+
#: Words too common to prove a question was engaged with. A reply shares
401+
#: "you", "the" and "that" with every question ever asked.
402+
_COVERAGE_STOPWORDS = frozenset(
403+
{
404+
"about", "actually", "again", "and", "answer", "any", "anything", "are",
405+
"ask", "asked", "because", "been", "being", "both", "but", "can",
406+
"could", "did", "does", "doing", "done", "for", "from", "give", "had",
407+
"has", "have", "her", "here", "him", "his", "how", "its", "just",
408+
"know", "like", "make", "many", "may", "mean", "might", "more", "most",
409+
"much", "must", "not", "now", "one", "only", "other", "out", "over",
410+
"own", "really", "say", "see", "separately", "she", "should", "some",
411+
"something", "still", "such", "take", "tell", "than", "that", "the",
412+
"their", "them", "then", "there", "these", "they", "thing", "things",
413+
"think", "this", "those", "through", "too", "use", "very", "want",
414+
"was", "way", "well", "were", "what", "when", "where", "which", "who",
415+
"why", "will", "with", "would", "you", "your", "yours",
416+
}
417+
)
418+
419+
#: How many distinctive words a question must have before its absence is
420+
#: treated as evidence. "why?" shares nothing with any answer and is answered
421+
#: fine; only a question with real content can be shown to be missing.
422+
_MIN_COVERAGE_TOKENS = 2
423+
424+
425+
def _coverage_tokens(text: str) -> set[str]:
426+
"""Content words that would show up if this ask were engaged with."""
427+
words = re.findall(r"[A-Za-z][A-Za-z'-]{2,}", str(text or "").lower())
428+
return {
429+
word.split("'", 1)[0]
430+
for word in words
431+
if word not in _COVERAGE_STOPWORDS and len(word.split("'", 1)[0]) > 2
432+
}
433+
434+
435+
def _unanswered_question_parts(body: str, contract: object | None) -> list[str]:
436+
"""Asks the reply never engages with at all.
437+
438+
LIVE DEFECT, 2026-08-10. "give me one concrete example of a preposition
439+
doing more work than it should. and separately — do you actually enjoy
440+
that, or is 'interesting' a word you reach for because it's safe?" The
441+
reply gave the example and contained nothing about enjoyment. Earlier the
442+
same day: "you dodged half of it. I asked two things and you answered one."
443+
444+
The compoundness was already known — question_parts was computed, the
445+
prompt was told "this prompt contains multiple asks (2 detected)", the
446+
voice budget widened for it. Every OTHER contract requirement in this
447+
validator has a matching violation and goes through repair; coverage was
448+
requested in the prompt and never checked, so a dropped half cost nothing.
449+
450+
Deliberately hard to trigger, because a false positive burns a
451+
regeneration on a reply that was fine:
452+
453+
* only when the contract already decided this is a multi-ask turn;
454+
* only for asks carrying at least _MIN_COVERAGE_TOKENS distinctive
455+
words, so "why?" or "really?" can never be flagged;
456+
* and only when the overlap is ZERO. One shared content word counts as
457+
engaged — this catches the half that was ignored outright, not the
458+
half that was answered briefly.
459+
"""
460+
if not getattr(contract, "requires_single_reply_coverage", False):
461+
return []
462+
segments = tuple(getattr(contract, "question_segments", ()) or ())
463+
if len(segments) < 2:
464+
return []
465+
466+
answered = _coverage_tokens(body)
467+
missed: list[str] = []
468+
for segment in segments:
469+
wanted = _coverage_tokens(segment)
470+
if len(wanted) < _MIN_COVERAGE_TOKENS:
471+
continue
472+
if not (wanted & answered):
473+
missed.append(segment)
474+
return missed
475+
476+
400477
def validate_dialogue_response(
401478
text: str, contract: object | None, state: object | None = None
402479
) -> DialogueValidation:
@@ -421,6 +498,9 @@ def validate_dialogue_response(
421498
if not _contains_owned_question(body):
422499
violations.append("failed_to_offer_own_question")
423500

501+
if _unanswered_question_parts(body, contract):
502+
violations.append("unanswered_question_part")
503+
424504
if getattr(contract, "prefers_dialogue_participation", False):
425505
if body.endswith("?") and _LOW_SIGNAL_PREFIX.match(body):
426506
violations.append("low_signal_redirect")
@@ -659,6 +739,18 @@ def build_dialogue_repair_block(contract: object | None, validation: DialogueVal
659739
if _requires_non_generic_aura_voice(contract):
660740
lines.append("- This turn must sound like Aura's own live voice, not a generic helper.")
661741
lines.append("- Do not use assistant boilerplate like 'I can help with that', 'How can I help', or 'As an AI'.")
742+
if "unanswered_question_part" in validation.violations:
743+
missed = _unanswered_question_parts(failed_text, contract)
744+
# Quote the dropped ask back. "Answer every part" is the instruction
745+
# that was already in the prompt when this happened; naming the
746+
# specific question that went unanswered is the part that is new.
747+
lines.append(
748+
"- The last draft answered only part of what was asked. These "
749+
"questions got no answer at all: "
750+
+ " | ".join(f'"{segment}"' for segment in missed[:3])
751+
+ ". Answer them in this reply. Answering one well and dropping "
752+
"the other reads as evasion even when it is not."
753+
)
662754
if "intra_response_repetition" in validation.violations:
663755
lines.append("- Do not repeat the same sentence or mantra. Say the thought once, then integrate it into a calmer next sentence.")
664756
if "unsupported_internal_jargon" in validation.violations:

core/phases/response_contract.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -452,6 +452,9 @@ class ResponseContract:
452452
question_parts: int = 1
453453
prefer_extended_answer: bool = False
454454
requires_single_reply_coverage: bool = False
455+
#: The text of each question asked, so coverage can be CHECKED rather
456+
#: than only requested in the prompt. See validate_dialogue_response.
457+
question_segments: tuple[str, ...] = ()
455458
max_tool_turns: int = 1
456459
max_tools: int = 4
457460
reason: str = ""
@@ -1252,6 +1255,7 @@ def build_response_contract(
12521255
question_parts=prompt_shape.question_parts,
12531256
prefer_extended_answer=bool(prompt_shape.prefers_extended_answer),
12541257
requires_single_reply_coverage=bool(prompt_shape.requires_single_reply_coverage),
1258+
question_segments=tuple(getattr(prompt_shape, "question_segments", ()) or ()),
12551259
max_tool_turns=max_tool_turns,
12561260
max_tools=max_tools,
12571261
reason=", ".join(reasons) if reasons else "ordinary_dialogue",

core/runtime/structured_input.py

Lines changed: 58 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -148,8 +148,15 @@ class PromptShape:
148148
imperative_parts: int = 0
149149
prefers_extended_answer: bool = False
150150
requires_single_reply_coverage: bool = False
151-
152-
def to_dict(self) -> dict[str, int | bool]:
151+
#: The actual text of each ask, not just how many there were.
152+
#:
153+
#: The count alone can shape a prompt ("3 parts detected") and size a
154+
#: voice budget, and it cannot check whether a reply covered them —
155+
#: checking needs to know WHAT was asked. Retained so
156+
#: validate_dialogue_response can hold the answer against the question.
157+
question_segments: tuple[str, ...] = ()
158+
159+
def to_dict(self) -> dict[str, int | bool | tuple[str, ...]]:
153160
return {
154161
"question_parts": self.question_parts,
155162
"explicit_question_marks": self.explicit_question_marks,
@@ -160,9 +167,50 @@ def to_dict(self) -> dict[str, int | bool]:
160167
"imperative_parts": self.imperative_parts,
161168
"prefers_extended_answer": self.prefers_extended_answer,
162169
"requires_single_reply_coverage": self.requires_single_reply_coverage,
170+
"question_segments": self.question_segments,
163171
}
164172

165173

174+
#: Splits an utterance into the units a person would count as separate asks:
175+
#: sentence enders, and the line breaks / numbered items that carry a list.
176+
_ASK_SPLIT_RE = re.compile(r"(?<=[.?!])\s+|\n+")
177+
178+
179+
def _question_segments(text: str) -> tuple[str, ...]:
180+
"""The individual asks in an utterance, as text.
181+
182+
LIVE DEFECT, 2026-08-10. "give me one concrete example of a preposition
183+
doing more work than it should. and separately — do you actually enjoy
184+
that, or is 'interesting' a word you reach for because it's safe?" She
185+
answered the example and said nothing whatever about enjoyment. The same
186+
failure was called out earlier in the same session — "you dodged half of
187+
it. I asked two things and you answered one."
188+
189+
The runtime already KNEW it was compound: question_parts was computed,
190+
the prompt was told "this prompt contains multiple asks (2 detected)",
191+
and the voice budget was widened for it. Nothing ever checked the reply
192+
against it, because the count was all that survived analysis. Keeping the
193+
segments is what makes coverage checkable at all.
194+
195+
An ask is a SENTENCE that either ends in a question mark or opens with a
196+
directive verb. Sentences, not lines, because everything else here counts
197+
per line and that is what missed the case above: it arrived as one line,
198+
so _INTERROGATIVE_LINE_RE — which requires the LINE to begin with what,
199+
why, do, is — never matched, the line began with "give", and a two-part
200+
utterance scored one part. Anyone typing in a chat box writes several
201+
sentences on one line constantly.
202+
"""
203+
raw = str(text or "").strip()
204+
if not raw:
205+
return ()
206+
segments = [part.strip() for part in _ASK_SPLIT_RE.split(raw) if part.strip()]
207+
return tuple(
208+
part
209+
for part in segments
210+
if part.endswith("?") or _DIRECTIVE_LINE_RE.match(part)
211+
)
212+
213+
166214
def analyze_prompt_shape(text: str) -> PromptShape:
167215
raw = str(text or "").strip()
168216
if not raw:
@@ -187,8 +235,15 @@ def analyze_prompt_shape(text: str) -> PromptShape:
187235
repeated_clause_parts = max(0, len(_REPEATED_CLAUSE_RE.findall(raw)) - 1)
188236
imperative_parts = len(_COORDINATED_DIRECTIVE_RE.findall(raw))
189237

238+
ask_segments = _question_segments(raw)
239+
190240
part_candidates = [
191241
1,
242+
# Sentence-level asks. Every other candidate below counts per LINE or
243+
# per verb list, and a chat box is one line: "give me an example of X.
244+
# and separately — do you enjoy it?" scored 1 part, so the prompt was
245+
# never told it was compound and the reply dropped half of it.
246+
len(ask_segments),
192247
explicit_question_marks,
193248
question_like_lines,
194249
numbered_parts,
@@ -209,6 +264,7 @@ def analyze_prompt_shape(text: str) -> PromptShape:
209264
)
210265

211266
return PromptShape(
267+
question_segments=ask_segments,
212268
question_parts=question_parts,
213269
explicit_question_marks=explicit_question_marks,
214270
question_like_lines=question_like_lines,
Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
"""She answered one of two questions, twice, and nothing noticed.
2+
3+
LIVE, 2026-08-10:
4+
5+
"give me one concrete example of a preposition doing more work than it
6+
should. and separately — do you actually enjoy that, or is 'interesting'
7+
a word you reach for because it's safe?"
8+
9+
The reply gave the example and contained nothing whatever about enjoyment.
10+
Earlier the same day, in the same session: "you dodged half of it. I asked two
11+
things and you answered one."
12+
13+
Two independent causes.
14+
15+
1. DETECTION. analyze_prompt_shape scored that message at question_parts=1.
16+
Every candidate it computes is per-LINE or per-verb-list, and the message
17+
is one line: _INTERROGATIVE_LINE_RE requires the LINE to open with what/
18+
why/do/is, this one opens with "give", and _CONNECTOR_RE wants "then/also"
19+
followed by a directive verb rather than "and separately". So the prompt
20+
was never told the turn was compound, and neither was the voice budget.
21+
People type several sentences on one line constantly; asks are now counted
22+
per SENTENCE.
23+
24+
2. ENFORCEMENT. question_parts and requires_single_reply_coverage already
25+
existed and already shaped the prompt — "This prompt contains multiple
26+
asks (2 detected)" — but only the COUNT survived analysis, so nothing
27+
could hold the reply against the question. Every other requirement in
28+
validate_dialogue_response has a matching violation and goes through
29+
repair; coverage was requested and never checked, so dropping half cost
30+
nothing.
31+
32+
The check is deliberately hard to trigger. A false positive burns a
33+
regeneration on a reply that was fine, so it fires only on a multi-ask turn,
34+
only for asks with real content, and only when the overlap is ZERO.
35+
"""
36+
from __future__ import annotations
37+
38+
import pytest
39+
40+
from core.phases.dialogue_policy import (
41+
_unanswered_question_parts,
42+
validate_dialogue_response,
43+
)
44+
from core.runtime.structured_input import analyze_prompt_shape
45+
46+
LIVE_MESSAGE = (
47+
"give me one concrete example of a preposition doing more work than it "
48+
"should. and separately — do you actually enjoy that, or is 'interesting' "
49+
"a word you reach for because it's safe?"
50+
)
51+
52+
LIVE_REPLY_THAT_DROPPED_HALF = (
53+
'in English, the preposition "on" can carry a load of implied qualifiers. '
54+
"In German, these would need to be stated explicitly as adjectives or "
55+
"adverbs."
56+
)
57+
58+
59+
class _Contract:
60+
"""Only the fields the coverage check reads."""
61+
62+
def __init__(self, shape):
63+
self.requires_single_reply_coverage = shape.requires_single_reply_coverage
64+
self.question_segments = shape.question_segments
65+
66+
67+
def test_a_one_line_compound_message_is_detected_as_compound():
68+
"""The detection half. This scored 1 part live."""
69+
shape = analyze_prompt_shape(LIVE_MESSAGE)
70+
71+
assert shape.question_parts >= 2
72+
assert shape.requires_single_reply_coverage
73+
assert len(shape.question_segments) >= 2
74+
75+
76+
def test_an_imperative_followed_by_a_question_counts_as_two():
77+
""""give me X. do you Y?" — neither half is optional."""
78+
shape = analyze_prompt_shape("describe the room. does it feel like yours?")
79+
80+
assert shape.question_parts >= 2
81+
82+
83+
def test_the_live_reply_is_flagged_and_names_what_it_dropped():
84+
shape = analyze_prompt_shape(LIVE_MESSAGE)
85+
86+
missed = _unanswered_question_parts(LIVE_REPLY_THAT_DROPPED_HALF, _Contract(shape))
87+
88+
assert len(missed) == 1
89+
assert "enjoy" in missed[0]
90+
91+
92+
def test_a_reply_that_covers_both_passes():
93+
shape = analyze_prompt_shape(LIVE_MESSAGE)
94+
complete = (
95+
LIVE_REPLY_THAT_DROPPED_HALF
96+
+ " And yes, I do enjoy it — enjoy is the honest word; interesting is "
97+
"what I reach for when I want to sound measured."
98+
)
99+
100+
assert not _unanswered_question_parts(complete, _Contract(shape))
101+
102+
103+
def test_one_shared_content_word_counts_as_engaged():
104+
"""Catches the half ignored outright, not the half answered briefly.
105+
106+
A short answer is a style question. A missing answer is a different kind
107+
of failure, and only the second is worth a regeneration.
108+
"""
109+
shape = analyze_prompt_shape(LIVE_MESSAGE)
110+
terse = LIVE_REPLY_THAT_DROPPED_HALF + " And no, I don't enjoy it."
111+
112+
assert not _unanswered_question_parts(terse, _Contract(shape))
113+
114+
115+
def test_a_single_ask_is_never_flagged():
116+
shape = analyze_prompt_shape("what is your uptime?")
117+
118+
assert shape.question_parts == 1
119+
assert not _unanswered_question_parts("About two hours.", _Contract(shape))
120+
121+
122+
@pytest.mark.parametrize("filler", ["why?", "really?", "and you?", "how so?"])
123+
def test_contentless_questions_can_never_be_flagged(filler):
124+
""""why?" shares no content word with any answer to it."""
125+
shape = analyze_prompt_shape(f"tell me about the deployment pipeline. {filler}")
126+
127+
missed = _unanswered_question_parts("The pipeline runs on three stages.", _Contract(shape))
128+
129+
assert filler not in missed
130+
131+
132+
def test_the_violation_reaches_the_validator():
133+
"""Wiring: the check is worthless if validate_dialogue_response ignores it."""
134+
shape = analyze_prompt_shape(LIVE_MESSAGE)
135+
136+
result = validate_dialogue_response(
137+
LIVE_REPLY_THAT_DROPPED_HALF, _Contract(shape)
138+
)
139+
140+
assert "unanswered_question_part" in result.violations
141+
assert not result.ok
142+
143+
144+
def test_the_repair_block_quotes_the_dropped_question():
145+
""""Answer every part" was already in the prompt when this happened.
146+
147+
Naming the specific question that went unanswered is what is new.
148+
"""
149+
from core.phases.dialogue_policy import (
150+
DialogueValidation,
151+
build_dialogue_repair_block,
152+
)
153+
154+
shape = analyze_prompt_shape(LIVE_MESSAGE)
155+
block = build_dialogue_repair_block(
156+
_Contract(shape),
157+
DialogueValidation(ok=False, violations=["unanswered_question_part"]),
158+
LIVE_REPLY_THAT_DROPPED_HALF,
159+
)
160+
161+
assert "enjoy" in block
162+
163+
164+
def test_a_turn_without_the_coverage_flag_is_left_alone():
165+
"""No contract requirement, no check — this must not fire everywhere."""
166+
167+
class _Unflagged:
168+
requires_single_reply_coverage = False
169+
question_segments = ("do you enjoy it?", "and what about the room?")
170+
171+
assert not _unanswered_question_parts("Something else entirely.", _Unflagged())

0 commit comments

Comments
 (0)