REST strict-identity gate drops ~2,200 check-in observations: fingerprint-mismatch refusals audit as agent_id=NULL/identity_required (3,292 rows since Jun 12)

[cirwel]

Summary

audit.tool_usage holds 3,292 rows with agent_id IS NULL and
error_type='identity_required', spanning 2026-06-12 12:57 → 2026-07-29 16:05
and still accruing during analysis. Roughly 2,206 process_agent_update
observations are plausibly lost, 1,140 of them unrecoverable at the
(session, tool) level — a 10.9% drop rate fleet-wide for that tool, 28.0%
restricted to the 161 affected session keys. Eleven of the top 20 affected agents
lost 100% of their check-ins.

Every one of the 3,292 rows comes from a single code path:
src/services/http_tool_service.py:245 — the REST /v1/tools/call
strict-identity gate, the only site in the repo that writes
error_type='identity_required'. MCP-dispatch and Path-C refusals wrap in
success_response() and therefore audit as success=true, so this drop is
entirely on the REST tool surface. latency_ms is 0 on all 3,292 rows
(successes average 144–337ms) — the refusal precedes any DB work, consistent with
the pre-dispatch gate at :229-247.

Two distinct populations — do not conflate them

Class	Count	Data lost?
Fail → success 188–433ms later, same key	510 (23%)	No
Contiguous outage, never recovers	~1,140	Yes

The fast class is two Stop hooks double-posting, not a retry after a late
cache write — of 1,345 refusals in 30 days, exactly one precedes its key's
first success, so it is not a startup race. Recovery is confined entirely to
process_agent_update; outcome_event, knowledge, observe, sync_state,
and record_result recover 0%.

Failures cluster: 88.3% sit inside 53 windows of ≥10 failures (30-min gap
boundary); only 39 are isolated singletons. Largest window is 15h12m / 591
failures.

The 1,097 rows with session_id IS NULL (Jun 12–28) are the same class
pre-attribution, not a separate bug — session_id is just
arguments.get("client_session_id") (http_tool_service.py:224), and csid
presence fleet-wide went 3.5–5.4% → 96.7–98.8% across 2026-06-28. They carry the
identical sub-second retry signature (235 of 801 June failures recover <500ms vs
~0.3% chance expectation, ~100× above chance).

Mechanism

Dominant cause is the strict-mode fingerprint hijack guard, outnumbering the
no-binding case ~10:1 in the live log (286 reason=fingerprint_mismatch vs 29
PATH2_RESUME_MISS).

Path: http_api.py:337 consult_sticky_binding(redis_recovery=False) is
structurally skipped for any csid-bearing request (identity_step.py:346-352,
cacheable false when client_session_id present) → derive_session_key
returns the csid verbatim (session.py:702-713; agent- keys skip model
scoping) → resolve_session_identity PATH 1 hits Redis session:<csid> →
resolution.py:448-500 compares the binding's stored bind_ip_ua against the
caller's current fingerprint; mismatch under strict returns True →
PATH3_MINT_REFUSED → _resolve_http_bound_agent returns None → the gate at
http_tool_service.py:170-202 refuses.

Live env confirmed: STRICT_IDENTITY_REQUIRED=true and
UNITARES_SESSION_FINGERPRINT_CHECK=strict.

The fingerprint is f"{host}:{md5(UA)[:6]}" (src/http_api.py:56-63).
Reproduced against the server's own preimage log:

Fingerprint	Client
9f1c93	curl/8.7.1 — ~/.claude/hooks/stop-checkin.sh
355f8e	Python-urllib/3.14 — plugin scripts/checkin.py
0af84a / eca485	claude-code/2.1.220 / 2.1.218 (CLI) — the MCP path

bind_ip_ua is stamped from whichever client bound first
(persistence.py:153-170, :349-350, :379-380). So the deciding variable is which
client performed the original bind, and the refusal follows mechanically.

The sub-class that actually loses data gets worse the more correctly an agent binds

An agent that onboards over MCP — the canonical identity source — binds
bind_ip_ua to the claude-code CLI fingerprint. Both REST Stop hooks then
mismatch and are refused for the life of the session.

Reproduced on a live session during this investigation:
session:agent-b7372b63-3ba → bind_ip_ua = 127.0.0.1:0af84a;
audit.tool_usage for that key shows 0 successful process_agent_update, 2
refusals. Its turn observations were dropped because it bound properly.

Contrast session:agent-94c0dd1c-e60 → bind_ip_ua = 127.0.0.1:355f8e (urllib
bound first): the plugin's check-in succeeds, curl's is refused, and a paired
success lands 0.2–0.6s later. The observation survives; the refusal row is a
duplicate.

Caller split — not co-equal

Measured over the live log window (2026-07-26 19:36 → 2026-07-29 16:19; 152 REST
refusals, matching audit.tool_usage 1:1):

Caller	Refusals	Successful check-ins
~/.claude/hooks/stop-checkin.sh:146-168 (curl)	111 (~73%)	1
plugin scripts/checkin.py:155-171 (urllib)	23	150
PATH2_RESUME_MISS, no fingerprint	18	—

stop-checkin.sh is the dominant refusal source and contributes ~0.4
observations/day — retiring it removes ~73% of REST refusals and is close to
density-neutral. The plugin is the working emitter, not a co-equal twin. Note the
plugin runs from
~/.claude/plugins/cache/unitares-governance/unitares-governance/0.4.7/, not the
0.4.9 checkout.

Lumen (agent-69a1a4f7-a30, 378 refusals, 11.5% — the largest single key)

This is the same fingerprint-mismatch mechanism at a second IP, not a
missing-binding case: session:agent-69a1a4f7-a30 exists in Redis with
bind_ip_ua=100.84.100.128:000000 (empty-UA client) while a second Pi client
presents 100.84.100.128:9df06c (Python/3.13 aiohttp/3.13.3) — 151
PATH1_FINGERPRINT_MISMATCH events. Two Pi clients contending for one key.

Lumen's 378 failures are two solid blocks with no interleaved successes; each
block ends exactly one 3-minute cadence tick after the last failure. During the
Jul 9 outage no twin key absorbed its check-ins (busiest other session in that
10h45m window had 4 successes) — the 264 check-ins are genuinely lost, not
re-attributed.

Corrections to two plausible-but-wrong readings

Recorded so they are not re-derived:

1. "Provisioning the durable core.sessions row healed Lumen's Jul 19–23
   outage" (it ended 46s after that row appeared) — co-timing, not
   causality. A strict mismatch sets resume = False (resolution.py:989),
   and both PATH 2 return branches are gated on resume (:1098, :1125) under
   an explicit "when resume=False, don't return PG identity" rule (:1069). A
   PG session row cannot clear a fingerprint mismatch. What ended it was a
   Redis rebind, which co-writes both stores — evidenced by 1:1
   Redis-present/core.sessions-present correspondence across all 18 failing
   keys (6 present, 12 absent, zero splits). The Jul 9 burst settles it
   independently: it recovered at 14:09:52 with no core.sessions row in
   existence.
2. Therefore a durable-binding / PATH-2 fallback would NOT have prevented the
   Lumen outage. A durable binding carrying the same stale bind_ip_ua
   reproduces the refusal. The Lumen class needs rebind/fingerprint stability, or
   a fingerprint-independent credential — not durability.

For reference: core.session_bindings (migration 51's dual-write target) has 0
rows, and its reader get_session_binding (src/db/mixins/session.py:226) has
zero callers. Real, but not the cause here.

Proposed direction

Tier 1 — surface the failure (safe now). The refusal is currently invisible
at the caller: stop-checkin.sh:146-168 posts with
curl -sf --max-time 3 ... >/dev/null 2>&1 &, so -sf plus discarded output
plus backgrounding means a refused check-in is indistinguishable from a delivered
one. Log the typed refusal locally. This is the only change with no identity-model
risk.

Tier 2 — retire stop-checkin.sh as a per-turn emitter. 111 refusals for 1
landed observation, and it duplicates the plugin on every turn where both fire.

Tier 3 — persist a per-slot continuity_token. Verified server-side at
session.py:324-330, resolved via PATH 2.8. Correct because it is
fingerprint-independent and the slot file is per-session, therefore
per-agent.

Do not fix it with the operator token

Adding -H X-Unitares-Operator to the check-in hooks (mirroring
outcome-tracker.sh:110-113) would fabricate attribution and collapse every
agent on the machine onto one identity:

• With no agent_uuid in the body, http_api.py:299-317 runs
  resolve_operator_identity and returns before derive_session_key (:337) —
  the operator identity wins over the caller's client_session_id. The
  header becomes the attribution rather than authorizing it.
• One token → one identity: operator.py:142-147 keys on sha256(token)[:16],
  :212-231 memoizes one agent_uuid per fingerprint for 300s, :233-250 mints
  with force_new=True on miss. operator.py:39-40 states the rule directly:
  "Use one token per client identity, not a shared one."
• The gate cannot catch it: http_tool_service.py:180-183 returns None the
  moment get_context_agent_id() is truthy, with no proof_origin check.
  Refusals become wrongly-attributed successes that write state.
• core.agent_state.synthetic defaults false NOT NULL
  (src/db/mixins/state.py:36-49), so blended rows satisfy
  synthetic IS NOT TRUE at scripts/analysis/eisv_individuality_v2.py:132 and
  enter the pre-registered population permanently. 20 distinct agents /
  4,456 successful check-ins in 7 days would fold onto one UUID, crossing
  MIN_STATES=100 within hours as an "agent" whose trajectory is a mixture of
  20.

The precedent argues against it: outcome-tracker.sh:69-73 hard-skips unless
the full UUID is present and sends it in the body (:95-107), hitting
http_api.py:288 before resolve_operator_identity is reached. There the token
authorizes explicit attribution; in the check-in hook there would be nothing to
authorize.

Also do not pass the slot cache's agent_id field as attribution — it is a
label (Claude_20260729); the UUID lives under uuid. The gate's check is bare
truthiness (http_tool_service.py:185) but http_api.py:288 requires len==36
with 4 hyphens, so a label would bypass the gate and then resolve to nothing —
strictly worse than the current drop.

Related

• Fingerprint-mismatch resume rejection silently mints ephemeral identity; durable write proceeds success:true under phantom UUID (dogfood F3) #1319 — fingerprint-mismatch resume rejection (the PATH3_MINT_REFUSED path)
• outcome-tracker test-anchor feed dark since 2026-06-26 — schema drift fixed locally; REST session-echo now fail-closes under #1325 #1345 — REST session-echo fail-closes under fix(identity): fail closed when a hijack guard rejects a resume (#1319) #1325
• Tokenless prefix-echo over-claims strong/caller_asserted across an unverifiable process boundary (tier-honesty debt) #807 — tokenless prefix-echo tier-honesty debt
• Identity-bootstrap contract: refuse-pre-onboard with typed response (close the five non-allowlisted mint paths) #425 — identity-bootstrap contract: refuse-pre-onboard with typed response

Separate latent gap

stop-checkin.sh:65-67 exits 0 on an empty cache, and the plugin's lazy
onboard_helper.py:229 writes only $PWD/.unitares while stop-checkin.sh:40
reads only $HOME/.unitares — that yields no row at all, not a NULL row, so
it is invisible in this dataset.

If a $HOME/$PWD mirror is added, it must be gated on a slot being present —
mirroring session_cache.py:261 if args.kind == "session" and slot:. Without
and slot, onboard_helper's slot=None path (_slot_filename →
session.json, reachable because hooks/post-stop:113 passes RAW_SLOT which
_session_lookup.py:190 can emit empty) writes $HOME/.unitares/session.json —
the shared cache removed 2026-04-18.

[cirwel]

Correction — this issue's framing was wrong in two ways, and its numbers undercount

An independent diagnosis run the same evening (~/.unitares/analysis/identity-required-checkin-refusals-2026-07-29.md, 2026-07-29 20:46) reached the mechanism faster and corrected two claims above. Recording that here rather than leaving the issue as the authoritative version.

1. Root cause is a config flip, not a code defect. UNITARES_SESSION_FINGERPRINT_CHECK=strict was set on the live process at 2026-06-22 20:23 (dated by the sibling …plist.bak-fingerprint-strict-20260622). The code default is log (config/governance_config.py:1241), and the code's own comment at :1262-1265 says the global flag cannot safely flip to strict fleet-wide because IP:UA is legitimately shared by co-resident localhost clients — "strict would false-reject real twins." That is exactly what happened: 0 of 75 post-restart mismatches is a plausible hijack (52 × 127.0.0.1, 23 × Lumen's own Tailscale peer, 1 genuine IP renumber). The fix is a one-line env flip plus restart, not the tiered client-side work proposed above.

2. The fast-recovery class is NOT benign, and calling it "the #425 contract working" was wrong. Those pairs are not a retry after a refusal. They are two different Stop hooks — stop-checkin.sh (curl) and stop-milestone-audit.sh, plus the gov-plugin's urllib leg — of which only the one holding the matching fingerprint succeeds. ~36% of failures are accidentally masked because a sibling leg landed 100–950 ms later. Net loss is ≈12–24 Claude Code check-in turns/day, not zero.

⚠️ Consequence for any fix: do not repair both legs and leave both registered. There is no idempotency guard, so the currently-masked ~36% would become duplicate core.agent_state rows, doubling check-in cadence and perturbing calibration bins and the adoption series. Remove one leg; don't fix both.

3. The counts above undercount, and "entirely on the REST tool surface" is an artifact of the instrument. MCP-dispatch strict refusals are not written to audit.tool_usage at all. Lumen reads 493 ok / 0 fail across a window in which it was refused 23 times. Every number in this issue derives from tool_usage, so it is blind to every resident on the MCP transport. Residents are affected: Lumen eats ~23 false hijack broadcasts and forced re-onboards per 25h (it does not lose states — PATH 0 passthrough re-onboards it immediately).

4. Observability is the worst part, and there are two holes. /v1/tools/call returns HTTP 200 with the refusal in the body, so curl -sf succeeds and checkin.py logs status=sent; grep -c identity_required ~/.unitares/checkins.log returns 0. Plus the MCP-dispatch gap above. This supersedes the "Tier 1 — surface the failure" framing: the REST client cannot see a refusal even in principle from the status code.

5. outcome_event 11/11 loss is a separate one-word bug — outcome-tracker.sh:99,106 and stop-milestone-audit.sh:155 send agent_uuid, which _resolve_http_bound_agent never inspects.

6. One session is in the terminal stage. agent-0b551710-90d (identity 12397, still active) has had both legs fail every turn since 2026-07-28 23:16 — ungoverned 21+ hours. Refused calls never renew the TTL (update_session_activity runs only in the resolved branch), so Redis expires at 24h and the sweep then deletes the core.sessions row, after which even a correctly-matching client is refused.

Retained from the original report: the mechanism trace (sticky-binding skip → derive_session_key → PATH 1 → resolution.py mismatch → refusal at http_tool_service.py:245), the fingerprint reproduction, and the standing prohibitions — no operator-token attribution, no slot-agent_id label (it is a label; the UUID is under uuid), no bind_ip_ua rewriting.

Gating note: the fix was deliberately held for the pre-registered EISV read at 2026-07-30 09:00, which has now fired (verdict FAIL, recorded). That gate is clear.

[cirwel]

Status 2026-07-31 (end of day) — the drop fell ~70%, it did not stop

An earlier draft of this comment claimed the drop had stopped. That was true when measured and is now false — it was written ~6 h after the UNITARES_SESSION_FINGERPRINT_CHECK=log flip, during a genuinely quiet window. Re-measured at end of day, refusals resumed. Correcting rather than posting the stale version.

1. The real magnitude: large reduction, not elimination

day	agent_id IS NULL AND error_type='identity_required'
2026-07-25	14
2026-07-26	53
2026-07-27	47
2026-07-28	71
2026-07-29	63
2026-07-30	50
2026-07-31	17

All-time total is now 3,405 (3,292 when this issue was filed). 18 rows postdate the flip at 2026-07-30 20:56:55 −06; the most recent is 2026-07-31 15:16:57 −06.

So the mitigation is real and large — roughly a 70% reduction against the 47–71/day baseline — but the class is not closed. Two of today's refusals land immediately after a server restart (13:47), which is expected; the rest are not restart-adjacent.

2. What the 18 actually are

process_agent_update | 16 | 20:56:55 → 15:16:57 | 11 distinct sessions
knowledge            |  1 | 13:47:21            | session_id NULL
observe              |  1 | 13:47:21            | session_id NULL

Still overwhelmingly the check-in path, still spread across many sessions rather than one stuck client.

3. T1 shipped, and it is what made the residual legible

~/.claude/hooks/stop-checkin.sh now logs the typed refusal locally (backup at stop-checkin.sh.bak). Rationale is independent of the fingerprint mode and worth restating, because it is the part that generalises: strict_identity_refusal_payload is documented in-source as "A structured success-shape, not an error" — HTTP 200 with {"status": "identity_required", …}. curl -sf … >/dev/null 2>&1 could never have surfaced it in any mode. -f was dropped deliberately (it discards the body the refusal rides in); still backgrounded, still exit 0, no attribution change, no operator token.

29 lines so far:

shape	n	meaning
skip reason=no_session_cache	18	Stop hook posts nothing at all — invisible in audit.tool_usage by construction
refused code=200	6	the typed refusal, body captured
refused code=000	5	server unreachable (restart windows)

Caller-side and server-side agree exactly. The last DB refusal at 2026-07-31 15:16:57.801953 −06 has its counterpart in the hook log at 2026-07-31T21:16:57Z carrying the full typed body. That correspondence is the thing this issue lacked: the refusal used to be indistinguishable from a delivered check-in at the caller.

The 18 no_session_cache skips are the larger and previously unmeasurable class — turns that never post, so they generate no row to be counted. They do not appear in the 3,405 and never did.

4. Standing corrections to the original analysis

• The "111 refusals : 1 landed observation" arithmetic is dead. Under mode=log the curl hook lands. The case against stop-checkin.sh as a per-turn emitter is now duplication, not drop — sub-2 s duplicate check-ins ran ~17.8% in the first post-flip window. Before deleting it: it is the only emitter that reads continuity_token, so retiring it forecloses the T3 caller side. Prefer server-side idempotency on (session_id, turn).
• The "separate latent gap" yields a misattributed row, not a missing one. read_slotted()'s fallback calls session_cache.py get session --workspace "$PWD" with no --slot, and cmd_get has no slotless guard (only cmd_set does), so it reads the shared $PWD/.unitares/session.json. Harmless while that key has no Redis binding — but mode=log removes the guard that would catch a foreign client if it ever acquires one.
• The "do not fix it with the operator token" section stands unchallenged.

Recommendation

Keep open, re-scoped. The framing should change from "the REST gate drops check-ins" to "a residual ~17/day, now observable from both ends." Three successors:

1. The no_session_cache class — orchestrated/dispatch worktree threads whose Stop hook posts nothing. Larger than the refusal class and structurally absent from audit.tool_usage. This is new information from T1 and probably deserves its own issue.
2. Duplicate check-ins — server-side idempotency vs retiring the curl emitter.
3. Is mode=log permanent? If yes, Fingerprint-mismatch resume rejection silently mints ephemeral identity; durable write proceeds success:true under phantom UUID (dogfood F3) #1319's hijack guard is advisory fleet-wide and should be documented as such. If no, T3 is the actual fix — and note its caller side already exists; the gap is that no writer populates continuity_token. T3 is a writer change, not a hook change.

(Read-only apart from the user-level hook edit. No DB writes.)