feat(ingest): cursor_sqlite format — ingest Cursor IDE chat history from state.vscdb (RFC #361)

[eric-tramel]

Implements RFC #361: ingest Cursor IDE chat history directly from state.vscdb (SQLite) as a new fully supported cursor_sqlite source format. Cursor agent-transcript JSONL support (#363) covers cursor-agent CLI sessions; this adds the IDE-side composer/bubble history that only exists in SQLite, completing Cursor harness coverage.

What changed

New SQLite polling engine (crates/moraine-ingest-core/src/sqlite_poll.rs)

• Hash-cursor change detection per RFC RFC: add efficient SQLite polling support for database-backed harnesses #361: cursorDiskKV has no watermark column, so the checkpoint carries a per-key content-hash map (cursor_json) plus a db/wal/shm stat fingerprint (nanosecond mtimes) for cheap no-change short-circuits.
• Read-only opens, never checkpoints another app's WAL (query_only=ON, 500ms busy timeout). Cleanly-closed WAL databases on read-only media (e.g. the dev sandbox's :ro bind mount) are retried with immutable=1 — safe exactly because the WAL sidecars are absent. The refusal surfaces as SQLITE_CANTOPEN on Linux and SQLITE_READONLY_DIRECTORY on macOS; both retry.
• Values are read by storage class (TEXT/BLOB/NULL all accepted): real Cursor writes JSON as TEXT despite the BLOB-declared column. Caught by integration testing against real host databases — synthetic fixtures originally stored BLOBs and masked it; fixtures now match reality.
• Paged prefix scans (512 rows / 32 MiB per page, whichever first) with per-page record synthesis, so raw value bytes (Cursor stores ~1.2 MB base64 screenshots inside tool results, doubled by toolCallBinary) never accumulate beyond one page.
• 10k relevant-key ceiling (RFC RFC: add efficient SQLite polling support for database-backed harnesses #361) enforced both as a pre-count and against the actual scan, with a clear "use JSONL transcripts instead" error.
• Failure modes (sqlite_open_error, sqlite_schema_mismatch, sqlite_cursor_too_large, sqlite_scan_error) emit one ingest_errors row per state change — repeated identical failures send nothing (no checkpoint churn), and error checkpoints never advance last_offset, so a failed poll can never out-rank a pending success checkpoint in the sink's flush window.

Stable logical event identity (sources/cursor.rs)

• Mutable DB rows can't hash raw_json into UIDs (RFC RFC: add efficient SQLite polling support for database-backed harnesses #361 §7). Bubble/composer events mint UIDs from logical identity (cursor_sqlite:bubble:{composer}:{bubble} etc.) and always stamp creation time, so a mutated row re-emits byte-identical sort keys and ReplacingMergeTree collapses versions. Verified live in e2e: rewriting a composer + appending a bubble leaves exactly one session_meta row.
• Rows without a parseable createdAt are deferred (no event, no error spam) until Cursor writes one — placeholder timestamps would strand permanent epoch-dated duplicates in the sort key.
• Tri-modal assistant bubbles (text / capabilityType 30 thinking / 15 toolFormerData) → message / reasoning / tool_call+tool_result events; toolCallBinary dropped; embedded-JSON params/results parsed; >64 KiB strings elided; tool_call_id takes the first line of Cursor's newline-joined id pair; degenerate mcp-- tool names repaired from rawArgs.
• Ghost composers (Cursor pre-creates a composerData shell per window: no headers, no name) are skipped — 12 of 15 composers in the real host DB are shells.

Watcher/dispatch integration

• state.vscdb-wal/-shm sidecar events map to the canonical DB path and coalesce (WAL-mode writes often touch only sidecars).
• Symlink canonicalization for cursor_sqlite paths only (macOS FSEvents reports /private/var/... for /var/... watch roots, which split one DB into two checkpoint identities). Deliberately not applied to existing file-backed formats: their checkpoint keys and event UIDs embed source_file, and re-keying long-standing symlinked sources (dotfiles-managed ~/.codex etc.) would orphan checkpoints and duplicate history. Regression-tested both ways.
• cursor_sqlite requires harness = "cursor" (validated at config load; .vscdb format inference is harness-gated the same way the hermes session_json inference is).

Schema (sql/015_sqlite_checkpoint_cursor.sql)

• ingest_checkpoints gains cursor_json String, source_fingerprint UInt64, schema_fingerprint UInt64 (additive, IF NOT EXISTS, default-valued). Pre-migration ingestors degrade gracefully: a startup warning explains cursors won't persist until migrate and restart.
• Transcript fetch dedups by event_uid (LIMIT 1 BY, newest event_version wins). events is partitioned by toYYYYMM(ingested_at) and ReplacingMergeTree never collapses across partitions; cursor_sqlite makes mutation-re-emit a steady-state path (a session resumed next month re-emits its composer row), so the read side now renders each event exactly once. See "Known limitations".

Operational impact

• Migration 015 (additive ALTERs) applies on moraine up / moraine db migrate.
• Default on: the cursor-sqlite source ships enabled in both default_sources() (platform-aware: ~/Library/Application Support/Cursor/User on macOS, ~/.config/Cursor/User elsewhere) and the config/moraine.toml template. Machines without Cursor simply match nothing. Configs with their own explicit [[ingest.sources]] are unaffected. Cursor has no stable local-database contract — when the schema drifts, moraine reports rate-limited sqlite_* errors and skips the database rather than ingesting bad rows, and the normalizer gets updated to follow the format; tracking that drift is moraine's job. Two template-guard tests pin the default-on state. Docs: format table/matrix + a new "SQLite-Polled Sources" section in docs/development/ingest-sources.md; README/docs harness tables now list Cursor (pre-existing gap).
• Dev sandbox mounts the host's Cursor state dir read-only via --mount-host-sessions (SANDBOX_CURSOR_STATE_DIR, default ~/Library/Application Support/Cursor/User).

Scope deviations from RFC #361 (flagged for owner veto)

• agentKv:blob:* (provider-wire duplicates, 66 MB observed), checkpointId:*, and ItemTable (holds live auth tokens — must never be ingested) are excluded from v1. Composer + bubble keys carry the full conversation content.

Known limitations

• Cross-month duplicate rows in events/search_documents storage remain possible for sessions mutated across a calendar-month boundary (partition key is toYYYYMM(ingested_at)). The primary transcript fetch dedups at read time (above); v_conversation_trace and search-side dedup need a follow-up (view rebuild migration). This is pre-existing behavior for hermes session_json rewrites; cursor_sqlite makes it more likely, hence the read-side mitigation now.
• Cursor histories beyond 10k composer/bubble keys are rejected by design (RFC RFC: add efficient SQLite polling support for database-backed harnesses #361 ceiling) with a pointer to JSONL transcripts.

Validation

• cargo test --workspace --locked — all green (incl. 13 new sqlite_poll unit tests: first-poll emission, no-op short-circuit, stable-UID re-emit on mutation, NULL/BLOB/TEXT storage classes, ghost gating, read-only-directory immutable fallback, schema-mismatch once-only + checkpoint-quiescence, 10k ceiling, createdAt deferral, URI escaping; golden fixture coverage for all bubble modes).
• make clippy (strict baseline), cargo fmt --all -- --check — clean.
• bash scripts/ci/e2e-stack.sh — full stack pass, including new cursor-sqlite assertions: 4 raw rows / 5 events / 5 distinct kinds / title extraction / 2 tool_io on first ingest, then a live-update phase (INSERT bubble + rewrite composer against the running stack) asserting 6 events and session_meta still collapsed to 1; MCP smoke passes for all 8 sources.
• Real-data integration (dev sandbox, host Cursor state mounted read-only): both real composer sessions ingested — 204 and 19 events, titles ("Program functionality explanation", "Cooking ideas inspiration"), 83 tool_call/tool_result pairs with real tool names (read_file_v2, edit_file_v2, MCP browser tools), 16 reasoning events, max payload 10.6 KB (screenshot elision working), zero ingest_errors, 12 ghost composers correctly absent, workspace-storage DBs (no relevant keys) polled cleanly via the immutable fallback.
• Adversarial multi-agent review over the full diff (5 dimensions, per-finding adversarial verification): 18 confirmed findings — all fixed in this branch except the cross-month partition item, which is mitigated at read time and documented above.

Closes #361.

🤖 Generated with Claude Code

[eric-tramel]

Flipped cursor_sqlite to default on per review (6d06d64): the source now ships enabled in default_sources() (platform-aware Cursor state dir) and in the config/moraine.toml template, with guard tests pinning both. Schema-drift posture stays the same — rate-limited sqlite_* errors + skip, then patch the normalizer to follow the format. PR description updated to match.

🤖 Generated with Claude Code