Commit 86b1de3
committed
feat(sync): schema v3 nested Y.Map metadata — lazy migration
Introduces schema v3 metadata model: file metadata entries are written
as nested Y.Maps instead of opaque JSON objects, giving field-level
CRDT resolution and eliminating whole-object tombstones on mtime updates.
## Core changes
### src/sync/fileMeta.ts (new)
Unified dual-shape helper module for v2 (flat) and v3 (nested Y.Map)
metadata. Provides type-safe decoders, read helpers, write helpers,
lazy conversion, incremental diff, and semantic change types.
Single authoritative interface — no call site accesses metadata directly.
### src/sync/schema.ts (new)
Pure, Obsidian-free SCHEMA_VERSION = 3 constant. Importable in tests
without dragging in the obsidian dependency.
### Metadata model
- All writes produce nested Y.Maps via ensureNestedMetaEntry + create helpers
- Reads dual-decode both flat (v2) and nested (v3) shapes everywhere
- Lazy on-write conversion: untouched flat entries remain flat indefinitely
- No eager migration, no distributed migration storm
- sys.schemaVersion bumped to 3 on first v3 client connect (markSchemaV3)
- migrateSchemaToV2 reverted to write flat v2 objects (was incorrectly
writing nested maps, creating v3 shapes under v2 schema marker)
### Semantic observer (observeMetaChanges)
Single shared meta.observeDeep handler on VaultSync. Uses event paths
for O(k) incremental diff instead of O(N) full snapshot on every change.
Dispatches MetaChangeBatch with origin + isLocal to all subscribers.
DiskMirror and witness tracker consume semantic changes directly.
### DiskMirror
- Replaced shallow meta.observe with observeMetaChanges subscription
- Correctly handles nested field mutations (deletedAt, path, mtime)
- Skips local-origin batches (isLocal=true) to prevent local writes
feeding back as remote file operations
- Suppresses disk.rename.observed remoteOrigin flag via
_pendingRemoteRenameNewPaths when handleRemoteRename runs
- Normalizes all paths from semantic change events before disk ops
### Server
- SERVER_MIN/MAX_SCHEMA_VERSION bumped to 3
- countActivePathsInDoc / computeDocStats dual-read v2 and v3 metadata
- documentSummary debug response includes flatMetaEntries, nestedMetaEntries,
invalidMetaEntries shape counters
- readMetaPath / isMetaDeleted helper methods for dual-shape reads
### Analyzer (orphan-after-rename rule)
New remoteOrigin exemption: disk.rename.observed events with
remoteOrigin:true (set by main.ts when DiskMirror's pending rename set
contains the new path) are not flagged as orphans. Passive receiver
devices correctly produce disk renames without CRDT rename events —
the CRDT rename was initiated by the other device.
## Tests
### New test suites (430 assertions across 10 suites):
- tests/file-meta-decode.ts — 112: decoder, type guards, helpers
- tests/file-meta-lazy-write.ts — 34: no-storm proof, concurrent convergence
- tests/meta-observer-integration.ts — 40: nested mutations fire semantic
changes; origin filtering proven local vs remote; incremental diff correct
- tests/meta-v3-schema-gate-and-stats.ts — 47: schema gate imports real
server constants; mixed metadata stats; realistic vault round-trip
- tests/meta-diskmirror-integration.ts — 52: real DiskMirror integration
with spied handlers; proves remote nested delete/rename/revive trigger
correct disk ops; proves local changes are ignored
### Updated:
- tests/disk-mirror-observer.ts: added observeMetaChanges to fakeVaultSync
- tests/v2-offline-rename-regressions.mjs: use getMetaPath/getMetaDeletedAt
helpers (restore now writes nested Y.Maps, flat property access broke)
- tests/run-regressions.mjs: added meta-diskmirror-integration to suite
## QA scenario (S15)
Two-vault CDP scenario on ~/temenos + ~/temenos-b against the deployed
kavin-yaos.ripplor.workers.dev server (SQL storage, schema v3):
Phase 1 create: hash match ✓ (19037d3bbde3)
Phase 2 rename: hash match ✓ (dc952595d28f), old path gone ✓
Phase 3 delete: file gone on B ✓
Phase 4 revive: hash match ✓ (7593fdbb0b82)
Phase 5 mtime-only: B disk hash unchanged (f8f98eccff4a = f8f98eccff4a) ✓
Phase 6 schema: schemaVersion A=3 B=3 ✓
Both analyzer passes: 0 hard failures. Exit 0.
Server post-run: flatMeta=1403, nestedMeta=5 (only touched entries converted).
Full CI: npm ci + npm ci --prefix server + npm run build +
npm run test:ci + npm run test:regressions (73 suites) +
npm --prefix server run typecheck — all clean.1 parent d7998d9 commit 86b1de3
20 files changed
Lines changed: 4516 additions & 229 deletions
File tree
- qa
- analyzers/rules
- controllers
- obsidian-harness/scenarios
- scripts
- server/src
- src
- sync
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
0 commit comments