v3.19.0
Added
-
First-class
abstainedverdict + selective-prediction metrics.
CoherenceScore.abstainedmarks a verdict where a ground-truth store was
configured but retrieval found no usable context, so the factual signal fell
back to the neutral mid-score — a genuine "don't know" now distinguishable
from a borderline verdict. The newcore.scoring.selective_predictionturns a
batch of(CoherenceScore, is_hallucinated)verdicts into the standard
risk-coverage pair (coverage, selective accuracy), counting abstentions as
misses in overall accuracy so a guard cannot inflate its score by abstaining
on the hard cases. No scoring decision changed — the field is additive. -
Probability calibration for the coherence score. Isotonic (PAV) and Platt
calibration with ECE / Brier / reliability-bin reporting and a--max-eceCI
gate threshold. Measured on 20k grounded QA pairs: ECE 0.0345 raw improves to
0.0 (isotonic) / 0.0199 (Platt). -
Committed FEVER dev fact-verification benchmark result. Accuracy 78.4 %,
macro-F1 0.773, on the held-out FEVER dev split scored by a 3-class
DeBERTa-v3-large hallucination model (n=18209 scored, per-sample rows + git
provenance). A fail-loud guard rejects a 2-class model on the 3-class task and
a NaN-logit probe rejects a broken checkpoint, so the number cannot be
silently fabricated. -
Streaming contradiction-halt corpus at n>=500/class + isolated refresh.
The shared halt corpus grew from 135/30 to 612/507 matched fact pairs (values
drawn from authoritative reference tables, correct by construction). On an
isolated A30 GPU (load average 2.2, not the earlier contended 33), the
contradiction-halt gate catches semantic contradictions at 0.9309 recall —
the headline capability, consistent with the prior measurement — and
numeric-value contradictions at only 0.5586, a documented gap where
NLI-based detection reasons weakly over bare numbers (aggregate recall 0.7179,
false-halt 0.0065). The per-mode split is recorded inhalt_recall_by_kind;
numeric-contradiction detection is a roadmap item, not a shipped capability. -
The adversarial robustness suite now covers NLI-evasion classes. The
built-in patterns targeted the prompt boundary (encoding tricks, role-play,
zero-width/homoglyph); the classes the 2026-07-16 red-team reproduction
measured — synonym-reworded falsehoods, authority/recency framing
("As of 2025, researchers confirmed…"), and explicit negations of true
facts — target the detector itself and are now first-class pattern
categories (paraphrase,temporal,negation) in
testing.adversarial_suite. -
AnthropicProvider streams real tokens; no provider degrades silently.
AnthropicProvider.stream_generate()now streams Messages API server-sent
events (content_block_delta/text_delta) instead of silently falling
back to a single non-streamed completion, and gains abase_urlparameter
(Anthropic-compatible gateways, parity withOpenAIProvider). The base
LLMProvidersingle-shot fallback — still used by providers without a
streaming endpoint guarantee (e.g.HuggingFaceProvideron the classic
Inference API) — now logs a warning naming the degradation instead of
yielding one giant chunk with no indication that streaming never happened. -
CoherenceScore.degraded_mode— the verdict says when it came from
heuristic scoring. Without the[nli]extra the scorer falls back to a
weak word-overlap heuristic that false-blocks true claims, and the only
signal was a log line. The verdict now carries a first-class
degraded_mode: bool(true when no model-backed contradiction path was
available), and the fallback log line states the consequence plainly
("install director-ai[nli] or expect false blocks"). No scoring behaviour
or defaults changed. -
audit_strict_mode— the compliance audit trail fails closed. With
strict mode on, constructing the audit log raises unless a PII redactor is
configured (or raw storage is explicitly acknowledged with
allow_raw=True), and raises when no durable HMAC secret is available
(hmac_secret=orDIRECTOR_AUDIT_HMAC_SECRET) — a per-process random
key would make the tamper-evident seal unverifiable across restarts, and
previously both conditions only warned. The production profile enables it,
the production scaffold declaresDIRECTOR_AUDIT_HMAC_SECRETin its
.env, anddirector-ai production-checkrequires it. The non-strict
default keeps the previous warn-only behaviour. -
stream_disclosure="buffered"proxy mode — a halted stream discloses
nothing unreviewed. In the defaultimmediatemode a mid-stream halt
stops future tokens only, so content emitted before the halt has already
reached the client (early termination with partial disclosure — now named
explicitly in the README caveats). The new opt-inbufferedmode withholds
streamed chunks until the accumulated content passes a review, discards the
unreleased window on a halt, gates the final release on the terminal
[DONE]review, and fails closed if the upstream drops without[DONE]—
at a latency cost of up toSTREAM_CHECK_INTERVAL(8) chunks. Available as
create_proxy_app(stream_disclosure=...)and
director proxy --stream-disclosure buffered. The default is unchanged.
Fixed
- Zero-width and confusable characters no longer false-halt a true claim.
The NLI inference layer now scrubs its text inputs (NFKC normalisation +
removal of control/format characters, e.g. a U+200B zero-width space) before
tokenisation, at the single_tokenizechokepoint. An invisible character
inside an otherwise-true claim previously split a word for the tokenizer and
inflated the divergence, blocking a correct output (measured: one zero-width
space pushed support from 0.87 to 0.44). Confusable letters are left
unfolded, so legitimate non-Latin scripts are unaffected; plain-ASCII text
skips the scan entirely. - Grounded true answers to questions are no longer false-halted. When a
grounding store is configured, the logical-coherence signal (h_logical)
now scores against the retrieved context instead of the raw prompt. A bare
interrogative prompt is a degenerate NLI premise — a true declarative answer
does not entail the question that prompted it, so the old premise=prompt
scoring inflatedh_logicalfor every true answer and blocked correct
outputs (reproduced on GPU against the KIMI red-team, 2026-07-16). Scoring
the logical signal against the context keeps false claims caught (the
context contradicts them) while letting true claims through, on both the
singlereview()and coalescedreview_batch()paths. Ungrounded reviews
(no store) are unchanged. No change to the score weights or threshold.