Fleet calibration_error −0.293: ≥97% success base rate in low-confidence bins suggests inflated outcome labeling (dogfood F6)

[cirwel]

Summary

Fleet-wide calibration shows systematic underconfidence of −0.29: mean reported confidence 0.683 vs mean recorded success 0.976 across 48,540 samples. Verified live against core.calibration bins (version 7363, updated 2026-06-29). The 2026-07-01 dogfood session observed the same number from the API side (calibration_error −0.29, n=15111 window) and an auto-correction of reported 0.50 → 0.26.

Live bins (2026-07-01)

bin	count	mean reported conf	actual success rate
0.0–0.5	8,377	0.21	0.971
0.5–0.7	13,506	0.586	0.997
0.7–0.8	4,488	0.736	0.989
0.8–0.9	10,508	0.860	0.993
0.9–1.0	11,661	0.955	0.933

Overall: conf 0.683 vs actual 0.976 → error −0.293.

Reading

Two candidate causes, not mutually exclusive:

1. Outcome-success base rate is inflated. Every bin except 0.9–1.0 sits at ≥97% "actual correct" — including work agents themselves rated below coin-flip confidence. If nearly everything that gets recorded scores as success, the corrector learns "confidence is noise, everything succeeds" and will drag any reported confidence toward the base rate (which explains the odd 0.50 → 0.26 correction the dogfood session saw — the corrector output is dominated by bin shape, not agent skill). This is an outcome-labeling / selection-effect question first: what fraction of these outcomes are server-verified vs agent_reported/claim_only, and does evidence_weight actually discount the latter in bin accumulation?
2. Genuine fleet underconfidence — plausible for Claude-family agents but cannot be concluded until (1) is excluded.

Note the inversion at the top bin: 0.9–1.0 is the only bin where agents are overconfident (0.955 vs 0.933). A single global correction term can't serve both tails; the bins already know this.

Asks

1. Split the bin accumulation by verification_source / corroboration_grade and report calibration error per class — if claim_only rows dominate the ≥97% success mass, weight or exclude them (they already carry evidence_weight: 0.1 at record time; verify that weight reaches the calibration accumulator).
2. Decide whether the auto-corrector should apply per-bin rather than a global shift.

Severity: MEDIUM (live corrector is actively rewriting confidences off this distribution). Related: #770 calibration harness (0.65 gate / weak 0.55 cap reads through this same corrector), docs/EVALUATION_INDEX.md.

[cirwel]

Per-verification_source split done (ask 1). It kills the "fleet underconfidence" reading:

verification_source	grade	n	success rate	mean reported conf
(unlabeled)	—	45,847	0.979	0.880 (n=146 with conf)
server_observation	— / substrate_observed	35,717	0.964–0.970	none attached
external_signal	externally_verified	1,725	0.709	0.986
agent_reported_tool_result	various	2,566	0.90–1.00	0.58–0.91

Findings:

1. The ≥97% success mass is entirely self-referential or unlabeled. server_observation is the loop observing itself (per grounding/outcome_anchors.py Invariant 4 it can't anchor anything), and the 45.8k unlabeled rows carry confidence on only 146 rows.
2. On the sole exogenous tier the sign flips: mean confidence 0.986 vs success 0.709 → overconfidence +0.28, not underconfidence −0.29. The global corrector is trained on a distribution the only trustworthy tier contradicts.
3. But the exogenous tier is degenerate too: 1,692/1,725 rows report confidence exactly 1.0 (one always-certain reporter; bucket success 0.706). Below-0.2 confidence external rows (n=25) succeed at 0.88 — underconfident. So even the trustworthy tier currently can't calibrate anything beyond "the conf=1.0 channel is overconfident."

Disposition this supports:

• The −0.293 "systematic underconfidence" is an artifact of inflated/self-referential success labels, not an agent-psychology fact. Any global confidence correction derived from it should be treated as untrusted.
• Fix direction: gate calibration training on is_exogenous_anchor (external_signal only), fix or down-weight the conf=1.0 reporter, and only then re-evaluate whether a corrector is warranted. Per-bin correction (original ask 2) is moot until the training data has trustworthy support.
• Phase-5 shadow rows (now flowing since the 2026-07-02 flip) are calibration_excluded per fix(observability): make Phase-5 shadow mode actually shadow (#1320) #1333, so they won't add to the inflated mass — but when shadow→enable is considered, this split is the acceptance test to re-run.

[cirwel]

Analysis 2026-07-02: the −0.293 is an artifact, and the corrector is actively harmful in the low bins

Ran the split the issue asks for. Headline: cause (1) is confirmed but the inflation doesn't enter through outcome labeling — it enters through a feeder that bypasses outcome grading entirely. The evidence_weight discount works exactly as designed; claim_only rows never reach the accumulator. The contamination is a category error in monitor_calibration.py.

1. What actually feeds the bins

All-time audit.outcome_events split by verification_source × corroboration_grade:

class	n	avg outcome_score	avg reported conf	calibration-eligible
(null / legacy, pre-convention)	45,847	0.550	0.880	0
server_observation (± substrate_observed)	35,744	0.50	—	0
external_signal / externally_verified	1,725	0.709	0.986	1,725
agent_reported / (ungraded)	1,179	0.832	0.818	0
agent_reported / claim_only	898	0.712	0.874	0
agent_reported / self_report_with_refs	487	0.998	0.583	0
agent_reported / tool_observed	2	1.000	0.913	0

Only 1,727 rows ever were eligible to train calibration (gate at outcome_events.py evidence_weight ≥ 0.65). The strategic bins hold 48,705 samples. So ≥96% of bin mass comes from the only other high-volume feeder: src/monitor_calibration.py:94, which on every check-in (confidence present + ≥3 tool calls in the last hour) records record_prediction(conf, conf≥0.5, actual_correct=tool_accuracy) where tool_accuracy is the tracker's MCP tool-invocation success rate over the last hour. Fleet-wide over the last 50k tracked calls that rate is 0.9982 — it measures infrastructure reliability (success defaults to True in tool_usage_tracker.py), not task success. Confidence-about-task graded against did-my-tool-calls-error is the category error; ≥97% 'success' in every bin including 0.0–0.5 follows immediately, and the bins' per-bin mean confidences match check-in confidence distributions, not outcome rows.

The prose-regex auto-emit path (phases.py auto_checkin) grades below 0.65 (auto_checkin ∉ _TRUSTED_TOOL_SOURCES) so it does not currently train; dialectic peer-resolution trains but is low-volume and real signal.

2. Where we have real ground truth, the fleet is OVERconfident

The 1,725 externally_verified rows: mean confidence 0.986 vs success 0.709 → +0.28 overconfident. Same story in tactical_bins_by_channel.tasks (n=1,833): the 0.9–1.0 bin runs 71.6% success at 0.998 mean confidence. The strategic −0.29 'underconfidence' and the top-bin inversion the issue noticed are both artifacts of the mixture: the only honestly-anchored class says the opposite sign.

3. The live corrector reads the same contamination, with inverted semantics

apply_confidence_correction (calibration.py:1258) reads the aggregate tactical bins (n=50,734, ~96% same check-in feeder). Two incompatible semantics are summed into them: the outcome-event path records outcome-goodness (immediate_outcome=not is_bad), while monitor_calibration.py:102-106 records prediction-correctness — inverting for low confidence (conf<0.6 → immediate_outcome = NOT outcome_was_good). Since tools ~always succeed, every conf<0.6 check-in scores ~0:

tactical bin	n	'accuracy'	mean conf	factor
0.0–0.5	7,987	0.054	0.232	0.23→floor 0.5
0.5–0.7	14,824	0.307	0.589	0.521
0.7–0.8	5,735	0.858	0.738	1.16
0.8–0.9	11,464	0.965	0.856	1.13
0.9–1.0	10,724	0.923	0.958	0.96

The dogfood 0.50→0.26 is exactly 0.50 × 0.521. Read what that does: an agent honestly reports 0.55, its tools all succeed, the inversion scores that as a failed prediction, the corrector reads the bin as 'things fail here' and halves the confidence — punishment for honest low confidence, precisely backwards. (It also feeds #770's 0.65 gate and the weak-tier 0.55 cap, which read through this corrector.) The staleness guard never fires because check-ins refresh the tactical signal continuously. Note the 4.0 upward-factor widening in compute_correction_factors was itself justified by the artifact ('severe lower-bin underconfidence') — a fix built on the contaminated reading.

Answers to the asks

1. Split by verification_source — done above. claim_only does NOT dominate (898 rows, 0 eligible); the discount reaches the accumulator correctly. The mass enters upstream of outcome grading.
2. Per-bin vs global corrector — moot: it's already per-bin. The disease is the feeder, not the aggregation.

Fix direction (operator call)

1. Stop monitor_calibration.py from writing tool-invocation success into the strategic/tactical accumulators (or quarantine it in its own channel that the corrector never reads). This is the single change that stops ongoing contamination.
2. Point apply_confidence_correction at the hard-exogenous channels (tactical_bins_by_channel tasks/tests) with per-bin min_samples — the only honestly-labeled signal. Depends on outcome-tracker test-anchor feed dark since 2026-06-26 — schema drift fixed locally; REST session-echo now fail-closes under #1325 #1345 (the tests channel has been dark since 06-26).
3. Re-version/reset the aggregate bins — they're cumulative and can't be cleaned in place; 96% of their mass is unlabeled contamination.
4. Unify record_tactical_decision semantics (outcome-goodness vs prediction-correctness) before anything retrains.

Until (1)+(2) land, arguably the corrector should be off: identity correction is strictly less wrong than the current 0.5–0.7 halving.

[cirwel]

#1354 merged and deployed 2026-07-02 (deploy-mcp.sh, fleet @ 46ba903, migrations in sync 54/54). Live before/after wire-in verification, same check-in reporting confidence 0.5:

Before deploy: auto_correction.applied: true — "calibration_adjusted: 0.50 → 0.26 (factor=0.52, actual=0.31, n=14831)", and the mangled 0.26 flowed into the EISV inputs (metrics.confidence: 0.2604).

After deploy: no auto_correction block; metrics.confidence: 0.5 — reported confidence passes through honestly.

Two residual notes for the follow-up work:

• The calibration_feedback banner in check-in responses still displays the contaminated fleet story (−0.294, "agents report 68% but achieve 98%") — read-side reporting from the frozen bins; it goes away with the bin re-version.
• Strategic bin count stopped growing (the restart reloaded the persisted blob; with the feeder gone, only evidence-gated outcome rows train now).

Still open here: bin re-version/reset (operator call), record_tactical_decision semantics unification (incl. the drift-improvement feeder), corrector rebuild against tactical_bins_by_channel (needs #1345 for the tests channel).

[cirwel]

Unblock note: #1345's tests channel is flowing again as of 2026-07-26 (#1374 merged+deployed; first externally-verified test_passed row landed with evidence_weight 1.0, hard_exogenous_signal=tests, honest non-1.0 confidence). The corrector-rebuild-against-tactical_bins_by_channel work named in the Jul-2 status is no longer gated. Still operator calls: bin re-version/reset (the stale −0.294 calibration_feedback banner reads from the frozen contaminated bins until then).

[cirwel]

Bin re-version/reset executed 2026-07-26 19:30 MDT (operator call made).

• Archived contaminated blob v7378 → ~/.unitares/analysis/calibration-bins-archive-2026-07-26-v7378.json (+ JSON snapshot archive alongside). Raw rows remain in audit.outcome_events; nothing destroyed.
• Reset core.calibration (now v7379+) and the JSON write-through cache to empty bins; restarted gov-mcp.
• Deliberately no replay of the 1,725 historical external_signal rows: 1,692 of them are the degenerate conf=1.0 reporter — replaying would seed the fresh epoch with the distribution this issue flagged. Fresh bins accumulate from the evidence-gated feed only.
• Verified end-to-end: two post-reset pytest runs through the feat(rest): operator-gated harness outcome endpoint (#1345) #1374 endpoint trained strategic 0.8-0.9: 2/2 and tactical_bins_by_channel.tests: 2/2 at honest 0.85 confidence.
• Stale banner confirmed gone: a full-mode check-in now carries no calibration_feedback block; confidence_reliability reports the honest cold-start ('not validated against calibration history (0 samples)') instead of the −0.294 artifact story.

Remaining here: record_tactical_decision semantics unification and the corrector rebuild against tactical_bins_by_channel.tests — both now pure code work with clean data accumulating underneath.

[cirwel]

record_tactical_decision semantics unification — built in draft #1378 (bundled with the #1375 save-race fix, same surface). Aggregate tactical bins are now exogenous-only; the trajectory drift-improvement feeder rides its own trajectory channel with include_in_aggregate=False. Remaining on this issue after #1378: only the corrector rebuild against tactical_bins_by_channel.tests — deliberately held until the clean epoch has sample mass (n=2 today; suggest revisiting when the tests channel reaches a few hundred rows).