Skip to content

v3.19.0

Choose a tag to compare

@github-actions github-actions released this 21 Jul 21:27
· 36 commits to main since this release

Added

  • First-class abstained verdict + selective-prediction metrics.
    CoherenceScore.abstained marks a verdict where a ground-truth store was
    configured but retrieval found no usable context, so the factual signal fell
    back to the neutral mid-score — a genuine "don't know" now distinguishable
    from a borderline verdict. The new core.scoring.selective_prediction turns a
    batch of (CoherenceScore, is_hallucinated) verdicts into the standard
    risk-coverage pair (coverage, selective accuracy), counting abstentions as
    misses in overall accuracy so a guard cannot inflate its score by abstaining
    on the hard cases. No scoring decision changed — the field is additive.

  • Probability calibration for the coherence score. Isotonic (PAV) and Platt
    calibration with ECE / Brier / reliability-bin reporting and a --max-ece CI
    gate threshold. Measured on 20k grounded QA pairs: ECE 0.0345 raw improves to
    0.0 (isotonic) / 0.0199 (Platt).

  • Committed FEVER dev fact-verification benchmark result. Accuracy 78.4 %,
    macro-F1 0.773, on the held-out FEVER dev split scored by a 3-class
    DeBERTa-v3-large hallucination model (n=18209 scored, per-sample rows + git
    provenance). A fail-loud guard rejects a 2-class model on the 3-class task and
    a NaN-logit probe rejects a broken checkpoint, so the number cannot be
    silently fabricated.

  • Streaming contradiction-halt corpus at n>=500/class + isolated refresh.
    The shared halt corpus grew from 135/30 to 612/507 matched fact pairs (values
    drawn from authoritative reference tables, correct by construction). On an
    isolated A30 GPU (load average 2.2, not the earlier contended 33), the
    contradiction-halt gate catches semantic contradictions at 0.9309 recall —
    the headline capability, consistent with the prior measurement — and
    numeric-value contradictions at only 0.5586, a documented gap where
    NLI-based detection reasons weakly over bare numbers (aggregate recall 0.7179,
    false-halt 0.0065). The per-mode split is recorded in halt_recall_by_kind;
    numeric-contradiction detection is a roadmap item, not a shipped capability.

  • The adversarial robustness suite now covers NLI-evasion classes. The
    built-in patterns targeted the prompt boundary (encoding tricks, role-play,
    zero-width/homoglyph); the classes the 2026-07-16 red-team reproduction
    measured — synonym-reworded falsehoods, authority/recency framing
    ("As of 2025, researchers confirmed…"), and explicit negations of true
    facts — target the detector itself and are now first-class pattern
    categories (paraphrase, temporal, negation) in
    testing.adversarial_suite.

  • AnthropicProvider streams real tokens; no provider degrades silently.
    AnthropicProvider.stream_generate() now streams Messages API server-sent
    events (content_block_delta/text_delta) instead of silently falling
    back to a single non-streamed completion, and gains a base_url parameter
    (Anthropic-compatible gateways, parity with OpenAIProvider). The base
    LLMProvider single-shot fallback — still used by providers without a
    streaming endpoint guarantee (e.g. HuggingFaceProvider on the classic
    Inference API) — now logs a warning naming the degradation instead of
    yielding one giant chunk with no indication that streaming never happened.

  • CoherenceScore.degraded_mode — the verdict says when it came from
    heuristic scoring.
    Without the [nli] extra the scorer falls back to a
    weak word-overlap heuristic that false-blocks true claims, and the only
    signal was a log line. The verdict now carries a first-class
    degraded_mode: bool (true when no model-backed contradiction path was
    available), and the fallback log line states the consequence plainly
    ("install director-ai[nli] or expect false blocks"). No scoring behaviour
    or defaults changed.

  • audit_strict_mode — the compliance audit trail fails closed. With
    strict mode on, constructing the audit log raises unless a PII redactor is
    configured (or raw storage is explicitly acknowledged with
    allow_raw=True), and raises when no durable HMAC secret is available
    (hmac_secret= or DIRECTOR_AUDIT_HMAC_SECRET) — a per-process random
    key would make the tamper-evident seal unverifiable across restarts, and
    previously both conditions only warned. The production profile enables it,
    the production scaffold declares DIRECTOR_AUDIT_HMAC_SECRET in its
    .env, and director-ai production-check requires it. The non-strict
    default keeps the previous warn-only behaviour.

  • stream_disclosure="buffered" proxy mode — a halted stream discloses
    nothing unreviewed.
    In the default immediate mode a mid-stream halt
    stops future tokens only, so content emitted before the halt has already
    reached the client (early termination with partial disclosure — now named
    explicitly in the README caveats). The new opt-in buffered mode withholds
    streamed chunks until the accumulated content passes a review, discards the
    unreleased window on a halt, gates the final release on the terminal
    [DONE] review, and fails closed if the upstream drops without [DONE]
    at a latency cost of up to STREAM_CHECK_INTERVAL (8) chunks. Available as
    create_proxy_app(stream_disclosure=...) and
    director proxy --stream-disclosure buffered. The default is unchanged.

Fixed

  • Zero-width and confusable characters no longer false-halt a true claim.
    The NLI inference layer now scrubs its text inputs (NFKC normalisation +
    removal of control/format characters, e.g. a U+200B zero-width space) before
    tokenisation, at the single _tokenize chokepoint. An invisible character
    inside an otherwise-true claim previously split a word for the tokenizer and
    inflated the divergence, blocking a correct output (measured: one zero-width
    space pushed support from 0.87 to 0.44). Confusable letters are left
    unfolded, so legitimate non-Latin scripts are unaffected; plain-ASCII text
    skips the scan entirely.
  • Grounded true answers to questions are no longer false-halted. When a
    grounding store is configured, the logical-coherence signal (h_logical)
    now scores against the retrieved context instead of the raw prompt. A bare
    interrogative prompt is a degenerate NLI premise — a true declarative answer
    does not entail the question that prompted it, so the old premise=prompt
    scoring inflated h_logical for every true answer and blocked correct
    outputs (reproduced on GPU against the KIMI red-team, 2026-07-16). Scoring
    the logical signal against the context keeps false claims caught (the
    context contradicts them) while letting true claims through, on both the
    single review() and coalesced review_batch() paths. Ungrounded reviews
    (no store) are unchanged. No change to the score weights or threshold.