fix+feat+test: HangBuster review follow-ups + capture-path bugs + raw NDJSON mode (closes #79 #80 #81 #82 #83 #84 #85)

[conorluddy]

Combined PR for the four HangBuster review follow-ups plus three live-test discoveries (#83/#84/#85). Each was originally a separate PR (#86, #87, #88, #89); merged into one branch on request. The seven logical commits remain atomic — easy to revert individually if needed.

Closes #79, #80, #81, #82, #83, #84, #85.

Commits in this PR

Commit	Closes	Diff
fix: append-only JSONL for auto_samples to remove read-modify-write race	#81	+109/-13
fix: handle zero-duration clusters in diff_sessions()	#79	+100/-4
feat: hash compute_fingerprint with sha256, bump FINGERPRINT_VERSION to 2	#80	+39/-11
test: harden HangBuster coverage — crashed-worker, fixture, verdicts	#82	+209/-64
chore: doc sweep — bump script count to 29, add Device State + Simulator Discovery sections	—	+44/-16
fix: --bundle-id no longer narrows hang predicate	#83	+107/-16
fix+feat: HangBuster worker EOF detect + bounded auto-restart + capture duration	#84	+498/-74
feat: --raw-capture NDJSON mode + size cap + gzip + aggregate prune	#85	+451/-14

#81 — auto_samples JSONL (race fix)

HangBuster._stash_auto_sample did a read-modify-write on auto_samples.json with no locking; concurrent stashes from a busy worker could drop entries. Switch to append-only auto_samples.jsonl mirroring the existing events.jsonl discipline (line-buffered + fsync).

New SessionStore.stash_auto_sample() / read_auto_samples() (last-write-wins per fp). Worker delegates — no local I/O.

#79 — zero-duration drift handling

diff_sessions() had a divide-by-zero guard that did a bare continue, silently dropping zero-duration clusters from the diff. Worst case: a 0 → 800 ms regression vanished.

Replaced with explicit 4-way branching:

Case	Result
A == 0, B == 0	stable++
A == 0, B > 0	drift, delta_pct = inf (rendered as new)
A > 0, B == 0	drift, delta_pct = -100.0
A > 0, B > 0	existing percentage logic

format_diff special-cases the inf delta so it prints as new rather than +inf%.

#80 — sha256 fingerprint, FINGERPRINT_VERSION=2

compute_fingerprint() returned the raw sym:<symbol> / msg:<message_prefix> string. Upstream normalisation collapses tokens to <addr> / <pid> / <n> placeholders, so distinct messages with overlapping normalised prefixes collided. Now hashed with sha256[:16] (fp:<16hex>).

Cluster.symbol_or_prefix already serves as the human-readable label — fingerprint is purely identity. No new display_prefix field needed.

FINGERPRINT_VERSION bumps 1 → 2. diff_sessions() already warns + skips on version mismatch (existing test coverage), so old v1 summaries degrade cleanly.

#82 — test coverage hardening

Gap 1: mark_crashed + load_summary→None (4 new tests) — the crashed-worker recovery path was entirely untested.

Gap 2: fixture too soft (5 new tests + fixture rewrite) — the soft fixture only produced CRITICAL events. L1 hit ~36% of budget, L2 only ~8%. New fixture spans all 4 severities, 12 clusters, 2 processes, populates bursts + quiet periods + process_distribution. All four L2 aggregate branches now actually render.

Gap 3: verdict + format_diff render coverage (6 new tests) — covers the 3 remaining verdict strings (new minor / improvement: resolved / drift: mixed signs) and the previously-unrendered new / resolved / drift output blocks.

Combined branch unified two near-identical helper functions (_zero_dur_summary from #79 and _summary_with_clusters from #82) into a single shared _summary_with_clusters((fp, dur, sev) tuple) — kept the more flexible form.

#83 — --bundle-id no longer narrows the predicate

Live-test surfaced an unfiled bug: _resolve_predicate(predicate, bundle_id) appended AND (process == <app>) to the os_log predicate. The default hang predicate matches RunningBoard / SpringBoard / watchdog events that originate outside the target app's process, so the AND silently dropped the bulk of useful hang signal.

Verified on Grapla: a 4-minute session with --bundle-id com.conorluddy.grapla captured 0 events while the simulator was actively logging hangs. After removing the AND, an identical workload captures hundreds of RunningBoard events per minute.

Fix: predicate stays simulator-global; bundle filtering applied post-parse against the event's process field. Promoted the existing _matches_bundle helper to a module-level matches_bundle() so the HangBuster worker reuses it. _resolve_predicate signature simplified — dropping bundle_id arg entirely and locking down via inspect.signature regression test so the bug cannot return.

#84 — Worker EOF detection + bounded auto-restart + capture duration

Live-test surfaced a second bug: log stream EOF (or 30s of silence under the old false-positive bail) broke the read loop, wrote a stream_ended event, and exited. SessionStore.persist_worker_counters then overwrote the status back to running — sessions appeared live indefinitely even though the worker had been dead for hours.

Three-part fix:

1. Crash detection (run_worker rewrite). Removed the 30s silence bail (a quiet log stream is not a dead one). EOF or proc.poll() != None now emits a stream_died event with exit_code. try/finally wrapping ensures mark_crashed runs whenever the worker exits involuntarily. Inner read loop extracted into _read_stream_into_events for direct testability.

2. Bounded auto-restart. New env var IOS_SIM_HANG_MAX_RESTARTS (default 3). Each restart logs a stream_restart event with attempt number and a 2s backoff prevents tight respawn loops. counters tracks stream_restarts; surfaces in summary + --list-sessions. SIGTERM during backoff or read loop exits cleanly without restart.

3. Capture-duration tracking. mark_crashed records stopped_at/stopped_at_ms. persist_worker_counters preserves both CRASHED and STOPPED status. build_summary prefers meta.stopped_at_ms over now() so summaries reflect the capture window. --list-sessions surfaces capture=Xs and restarts=N.

5 new worker-loop tests + 3 new session-store tests use a scripted fake subprocess.Popen to drive EOF / mid-stream death / SIGTERM / xcrun-missing scenarios deterministically.

#85 — --raw-capture NDJSON mode + size cap + gzip + aggregate prune

The summarised pipeline is the right shape when you trust the bucketing, but sometimes you want full-fidelity log data and ad-hoc jq exploration. --raw-capture spawns log stream --style ndjson and dumps lines verbatim to raw.ndjson, bypassing clustering.

Garbage collection is automatic and layered:

• Per-session size cap (--max-size-mb, default 10) — worker stops cleanly on cap hit, writes size_cap_hit marker, sets extras.truncated.
• Gzip on stop (skippable with --no-gzip) — NDJSON compresses ~15-19× in practice.
• TTL prune (existing IOS_SIM_HANG_SESSION_TTL_HOURS=24) runs on every --start. New aggregate cap IOS_SIM_HANG_TOTAL_CAP_MB (default 100) runs alongside, evicting oldest sessions first when total skill state exceeds.

Verified end-to-end on Grapla: a 12-second session captured 737 raw NDJSON lines, 0.96 MB → 0.05 MB gzipped (~19×). jq queries work directly on the gzipped file:

$ zcat raw.ndjson.gz | jq -s 'group_by(.processImagePath) | map({proc: (.[0].processImagePath | split("/") | last), n: length}) | sort_by(-.n) | .[:5]'
[
  {"proc": "runningboardd", "n": 544},
  {"proc": "SpringBoard",   "n":  83},
  {"proc": "mediaremoted",  "n":  39},
  {"proc": "locationd",     "n":  39},
  {"proc": "grapla",        "n":  28}
]

That process breakdown is the smoking gun for #83: only 4% of useful hang signal comes from the app process itself.

Implementation notes:

• New CLI: --raw-capture, --max-size-mb, --no-gzip. New env: IOS_SIM_HANG_TOTAL_CAP_MB.
• run_worker grew a raw-capture branch that reuses the bug: HangBuster worker silently dies after log stream EOF, session left in stale 'running' state #84 restart/crash machinery. Spawns log stream --style ndjson, feeds to _read_stream_into_raw, which dumps lines verbatim and bails on size cap.
• Non-JSON banners (log stream's "Filtering the log data ..." first line, pwuid_r warnings) are skipped so jq consumers get strict NDJSON.
• _stop_raw_session gzips via stdlib gzip + shutil.copyfileobj, removes uncompressed file, records bytes in meta.extras. No summary.json for raw sessions; --get-details points to the file with a zcat | jq hint.
• SessionStore.session_total_bytes + prune_to_aggregate_cap — sum sizes per session, evict oldest first until total ≤ cap. Called from HangBuster.start() alongside prune_expired().
• --list-sessions surfaces raw size (gzipped if present), truncated flag, capture duration.

7 new tests in tests/test_worker_loop.py — raw line writing, size-cap trigger, gzip round-trip + content preservation, --no-gzip preserves uncompressed, aggregate-cap eviction order, session_total_bytes.

Test count

Main baseline: 88. After all seven: 138 (+50).

Verification

ruff check: All checks passed
black --check (scripts): clean
python -m pytest tests/: 138 passed in 0.47s

Live end-to-end on Grapla simulator:

• --bundle-id no longer narrows predicate (5060+ matched lines vs pre-fix 0).
• --raw-capture --max-size-mb 1 produced 737 lines / 0.05 MB gzipped, jq-queryable.
• Bounded auto-restart confirmed by fake-subprocess unit tests.

Compat notes

• FINGERPRINT_VERSION = 2 writes into new summary.json files. Old v1 summaries on disk still load; --diff between v1 and v2 fires the existing version-mismatch warning.
• auto_samples.json → auto_samples.jsonl filename change. No external readers of the old file existed. Old files become inert and TTL-prune naturally.
• _resolve_predicate(predicate) signature change (dropped bundle_id arg) — internal API only. --bundle-id flag still accepted; now filters post-parse.

Out of scope (deferred from original review)

• Promoting common/hang_pipeline.py to a generic common/log_filters.py — AHA, wait for a second consumer.
• Temporal-burst overlapping-windows fix (m1 from original review) — defer until someone trips it.
• _BARE_INT redaction narrowing (n4) — partially mitigated by enhancement: hash compute_fingerprint() and bump fingerprint_version to 2 #80's hashing absorbing remaining collision risk.
• Streaming summary while session is live — current model: summary only on --stop.
• Multi-simulator capture — one session = one UDID.