Skip to content

Commit 86b1de3

Browse files
committed
feat(sync): schema v3 nested Y.Map metadata — lazy migration
Introduces schema v3 metadata model: file metadata entries are written as nested Y.Maps instead of opaque JSON objects, giving field-level CRDT resolution and eliminating whole-object tombstones on mtime updates. ## Core changes ### src/sync/fileMeta.ts (new) Unified dual-shape helper module for v2 (flat) and v3 (nested Y.Map) metadata. Provides type-safe decoders, read helpers, write helpers, lazy conversion, incremental diff, and semantic change types. Single authoritative interface — no call site accesses metadata directly. ### src/sync/schema.ts (new) Pure, Obsidian-free SCHEMA_VERSION = 3 constant. Importable in tests without dragging in the obsidian dependency. ### Metadata model - All writes produce nested Y.Maps via ensureNestedMetaEntry + create helpers - Reads dual-decode both flat (v2) and nested (v3) shapes everywhere - Lazy on-write conversion: untouched flat entries remain flat indefinitely - No eager migration, no distributed migration storm - sys.schemaVersion bumped to 3 on first v3 client connect (markSchemaV3) - migrateSchemaToV2 reverted to write flat v2 objects (was incorrectly writing nested maps, creating v3 shapes under v2 schema marker) ### Semantic observer (observeMetaChanges) Single shared meta.observeDeep handler on VaultSync. Uses event paths for O(k) incremental diff instead of O(N) full snapshot on every change. Dispatches MetaChangeBatch with origin + isLocal to all subscribers. DiskMirror and witness tracker consume semantic changes directly. ### DiskMirror - Replaced shallow meta.observe with observeMetaChanges subscription - Correctly handles nested field mutations (deletedAt, path, mtime) - Skips local-origin batches (isLocal=true) to prevent local writes feeding back as remote file operations - Suppresses disk.rename.observed remoteOrigin flag via _pendingRemoteRenameNewPaths when handleRemoteRename runs - Normalizes all paths from semantic change events before disk ops ### Server - SERVER_MIN/MAX_SCHEMA_VERSION bumped to 3 - countActivePathsInDoc / computeDocStats dual-read v2 and v3 metadata - documentSummary debug response includes flatMetaEntries, nestedMetaEntries, invalidMetaEntries shape counters - readMetaPath / isMetaDeleted helper methods for dual-shape reads ### Analyzer (orphan-after-rename rule) New remoteOrigin exemption: disk.rename.observed events with remoteOrigin:true (set by main.ts when DiskMirror's pending rename set contains the new path) are not flagged as orphans. Passive receiver devices correctly produce disk renames without CRDT rename events — the CRDT rename was initiated by the other device. ## Tests ### New test suites (430 assertions across 10 suites): - tests/file-meta-decode.ts — 112: decoder, type guards, helpers - tests/file-meta-lazy-write.ts — 34: no-storm proof, concurrent convergence - tests/meta-observer-integration.ts — 40: nested mutations fire semantic changes; origin filtering proven local vs remote; incremental diff correct - tests/meta-v3-schema-gate-and-stats.ts — 47: schema gate imports real server constants; mixed metadata stats; realistic vault round-trip - tests/meta-diskmirror-integration.ts — 52: real DiskMirror integration with spied handlers; proves remote nested delete/rename/revive trigger correct disk ops; proves local changes are ignored ### Updated: - tests/disk-mirror-observer.ts: added observeMetaChanges to fakeVaultSync - tests/v2-offline-rename-regressions.mjs: use getMetaPath/getMetaDeletedAt helpers (restore now writes nested Y.Maps, flat property access broke) - tests/run-regressions.mjs: added meta-diskmirror-integration to suite ## QA scenario (S15) Two-vault CDP scenario on ~/temenos + ~/temenos-b against the deployed kavin-yaos.ripplor.workers.dev server (SQL storage, schema v3): Phase 1 create: hash match ✓ (19037d3bbde3) Phase 2 rename: hash match ✓ (dc952595d28f), old path gone ✓ Phase 3 delete: file gone on B ✓ Phase 4 revive: hash match ✓ (7593fdbb0b82) Phase 5 mtime-only: B disk hash unchanged (f8f98eccff4a = f8f98eccff4a) ✓ Phase 6 schema: schemaVersion A=3 B=3 ✓ Both analyzer passes: 0 hard failures. Exit 0. Server post-run: flatMeta=1403, nestedMeta=5 (only touched entries converted). Full CI: npm ci + npm ci --prefix server + npm run build + npm run test:ci + npm run test:regressions (73 suites) + npm --prefix server run typecheck — all clean.
1 parent d7998d9 commit 86b1de3

20 files changed

Lines changed: 4516 additions & 229 deletions
Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
import type { FlightEvent } from "../flight-event";
2+
import type { AnalyzerFinding } from "../report";
3+
4+
/**
5+
* Rule: orphan-after-rename
6+
*
7+
* disk.rename.observed is now emitted as TWO events sharing the same opId:
8+
* - renameRole: "source" (oldPath → oldPathId)
9+
* - renameRole: "target" (newPath → newPathId)
10+
*
11+
* After both fire, the trace should show:
12+
* (a) crdt.file.renamed with newPathId
13+
* (b) NO crdt.file.created for newPathId after the rename (identity lost)
14+
* (c) NO crdt.file.tombstoned for newPathId before the cleanup phase
15+
*
16+
* Phase-marker logic:
17+
* Tombstones that happen during the cleanup phase (after qa.phase{phase:"cleanup"})
18+
* are expected scenario teardown and are not flagged.
19+
*
20+
* Backward compatibility:
21+
* Traces without qa.phase events fall back to using disk.delete.observed
22+
* as a proxy for "intentional user delete" (the old heuristic).
23+
*
24+
* Also flags:
25+
* - target-role disk.rename.observed with no matching crdt.file.renamed
26+
*/
27+
const WINDOW_MS = 15_000;
28+
29+
export function checkOrphanAfterRename(events: FlightEvent[]): AnalyzerFinding[] {
30+
const findings: AnalyzerFinding[] = [];
31+
32+
// Find all target-role rename events (the new path side).
33+
// Pre-dual-event traces had no renameRole; fall back to any disk.rename.observed.
34+
const targetRenames = events.filter((e) => {
35+
if (e.kind !== "disk.rename.observed") return false;
36+
const role = (e.data as Record<string, unknown> | undefined)?.renameRole;
37+
return role === "target" || role === undefined;
38+
});
39+
40+
if (targetRenames.length === 0) return findings;
41+
42+
// Build set of pathIds that appear as a SOURCE in a rename event within this trace.
43+
// A pathId that is both a rename target AND a subsequent rename source is an
44+
// intermediate hop in a rename chain (e.g. A→B→C: B is target of A→B and source
45+
// of B→C). YAOS collapses chains into a single CRDT rename (A→C), so B never gets
46+
// a crdt.file.renamed event. These intermediate hops are exempt from the hard failure.
47+
const sourcePathIds = new Set(
48+
events
49+
.filter((e) =>
50+
e.kind === "disk.rename.observed"
51+
&& (e.data as Record<string, unknown> | undefined)?.renameRole === "source",
52+
)
53+
.map((e) => e.pathId)
54+
.filter((id): id is string => !!id && id !== "p:unavailable"),
55+
);
56+
57+
// Determine cleanup phase start time from qa.phase markers (taxonomy v5+).
58+
const cleanupPhaseTs = events
59+
.filter((e) => e.kind === "qa.phase" && (e.data as Record<string, unknown> | undefined)?.phase === "cleanup")
60+
.reduce((min, e) => Math.min(min, e.ts), Infinity);
61+
const hasCleanupPhaseMarker = cleanupPhaseTs < Infinity;
62+
63+
for (const renameEvent of targetRenames) {
64+
const newPathId = renameEvent.pathId;
65+
if (!newPathId || newPathId === "p:unavailable") continue;
66+
67+
// Intermediate chain hop: this path was immediately renamed again.
68+
// YAOS collapses the chain in the CRDT batch, so no crdt.file.renamed
69+
// will appear for this intermediate path. This is correct behavior.
70+
if (sourcePathIds.has(newPathId)) continue;
71+
72+
// Remote-origin rename: DiskMirror applied this rename in response to a remote
73+
// metadata path change (schema v3 nested Y.Map path field mutation). The passive
74+
// receiver device performs the disk rename directly via handleRemoteRename without
75+
// emitting crdt.file.renamed — the CRDT rename was already applied on the active
76+
// device. remoteOrigin:true is set in the event data by the vault rename handler
77+
// when DiskMirror's _pendingRemoteRenameNewPaths set contains the new path.
78+
const isRemoteOrigin = (renameEvent.data as Record<string, unknown> | undefined)?.remoteOrigin === true;
79+
if (isRemoteOrigin) continue;
80+
81+
// Find the source-role event with the same opId to get oldPathId.
82+
const sourceEvent = events.find((e) =>
83+
e.kind === "disk.rename.observed"
84+
&& e.opId === renameEvent.opId
85+
&& (e.data as Record<string, unknown> | undefined)?.renameRole === "source",
86+
);
87+
const oldPathId = sourceEvent?.pathId ?? "unknown";
88+
89+
const renameTs = renameEvent.ts;
90+
91+
// (a) crdt.file.renamed must appear within window.
92+
// If not, check whether this is a pre-CRDT race recovery or a true silent drop.
93+
const crdtRenamed = events.find((e) =>
94+
e.kind === "crdt.file.renamed"
95+
&& e.pathId === newPathId
96+
&& e.ts >= renameTs
97+
&& e.ts - renameTs <= WINDOW_MS,
98+
);
99+
100+
if (!crdtRenamed) {
101+
// Determine whether the source file ever had a CRDT identity before the rename.
102+
// If oldPath had a crdt.file.created event before the rename fired, YAOS should
103+
// have had a fileId and the rename should have produced crdt.file.renamed.
104+
// If it did NOT, this is the pre-CRDT race: rename fired before ensureFile ran.
105+
const sourceHadCrdtIdentityBeforeRename = events.some((e) =>
106+
e.kind === "crdt.file.created"
107+
&& e.pathId === oldPathId
108+
&& e.ts < renameTs,
109+
);
110+
111+
// In the race case, the valid recovery outcome is crdt.file.created at newPath
112+
// (content redirected, new fileId assigned). Only accept this downgrade when:
113+
// 1. Source had NO prior CRDT identity (true race — no fileId to rename from)
114+
// 2. crdt.file.created at newPath appeared within window
115+
const crdtCreatedAsRecovery = !sourceHadCrdtIdentityBeforeRename
116+
? events.find((e) =>
117+
e.kind === "crdt.file.created"
118+
&& e.pathId === newPathId
119+
&& e.ts >= renameTs
120+
&& e.ts - renameTs <= WINDOW_MS,
121+
)
122+
: undefined;
123+
124+
if (crdtCreatedAsRecovery) {
125+
// Pre-CRDT race recovery: content preserved at newPath, but via a new
126+
// fileId rather than an identity-preserving rename. Warning, not hard failure.
127+
findings.push({
128+
rule: "orphan-after-rename",
129+
severity: "warning",
130+
pathId: newPathId,
131+
eventSeqs: [renameEvent.seq, crdtCreatedAsRecovery.seq],
132+
description:
133+
`disk.rename.observed for pathId=${newPathId} was handled via race recovery ` +
134+
`(crdt.file.created instead of crdt.file.renamed) — rename fired before CRDT ` +
135+
`had a fileId for oldPath=${oldPathId}. Content preserved; fileId is new.`,
136+
});
137+
} else {
138+
// Either source had a CRDT identity (real identity-loss bug) or no
139+
// crdt.file.created appeared at all (content lost entirely).
140+
const reason = sourceHadCrdtIdentityBeforeRename
141+
? `source pathId=${oldPathId} had a prior CRDT identity — this is identity loss, not race recovery`
142+
: `no crdt.file.created or crdt.file.renamed for pathId=${newPathId} within ${WINDOW_MS}ms`;
143+
findings.push({
144+
rule: "orphan-after-rename",
145+
severity: "hard",
146+
pathId: newPathId,
147+
eventSeqs: [renameEvent.seq],
148+
description:
149+
`disk.rename.observed (seq=${renameEvent.seq}, opId=${renameEvent.opId ?? "?"}) ` +
150+
`was not followed by crdt.file.renamed for pathId=${newPathId} within ${WINDOW_MS}ms ` +
151+
`— ${reason}. oldPathId=${oldPathId}`,
152+
});
153+
}
154+
continue;
155+
}
156+
157+
// (b) crdt.file.created AFTER rename = identity lost.
158+
const crdtCreatedAfterRename = events.find((e) =>
159+
e.kind === "crdt.file.created"
160+
&& e.pathId === newPathId
161+
&& e.ts > crdtRenamed.ts,
162+
);
163+
if (crdtCreatedAfterRename) {
164+
findings.push({
165+
rule: "orphan-after-rename",
166+
severity: "hard",
167+
pathId: newPathId,
168+
eventSeqs: [renameEvent.seq, crdtRenamed.seq, crdtCreatedAfterRename.seq],
169+
description:
170+
`crdt.file.created appeared after crdt.file.renamed for pathId=${newPathId} — ` +
171+
`file identity was lost (rename + create instead of identity-preserving rename). ` +
172+
`oldPathId=${oldPathId}`,
173+
});
174+
}
175+
176+
// (c) crdt.file.tombstoned without revive = spurious tombstone.
177+
// A tombstone is intentional if:
178+
// - Phase markers: it happened at or after the cleanup phase start, OR
179+
// - Fallback (no cleanup marker in trace): disk.delete.observed preceded it.
180+
const tombstonedAfterRename = events.find((e) =>
181+
e.kind === "crdt.file.tombstoned"
182+
&& e.pathId === newPathId
183+
&& e.ts > crdtRenamed.ts
184+
&& e.ts - crdtRenamed.ts <= WINDOW_MS,
185+
);
186+
if (tombstonedAfterRename) {
187+
const isIntentional =
188+
// Phase-marker check (cleanup phase event in trace = teardown)
189+
(hasCleanupPhaseMarker && tombstonedAfterRename.ts >= cleanupPhaseTs)
190+
// Fallback heuristic: disk.delete.observed preceded the tombstone
191+
|| events.some((e) =>
192+
e.kind === "disk.delete.observed"
193+
&& e.pathId === newPathId
194+
&& e.ts <= tombstonedAfterRename.ts,
195+
);
196+
197+
if (!isIntentional) {
198+
const revivedAfter = events.find((e) =>
199+
e.kind === "crdt.file.revived"
200+
&& e.pathId === newPathId
201+
&& e.ts > tombstonedAfterRename.ts,
202+
);
203+
if (!revivedAfter) {
204+
const markerNote = hasCleanupPhaseMarker
205+
? `cleanup phase starts at ${cleanupPhaseTs}`
206+
: "no cleanup phase marker (fallback: disk.delete.observed)";
207+
findings.push({
208+
rule: "orphan-after-rename",
209+
severity: "hard",
210+
pathId: newPathId,
211+
eventSeqs: [crdtRenamed.seq, tombstonedAfterRename.seq],
212+
description:
213+
`crdt.file.tombstoned appeared for pathId=${newPathId} after rename ` +
214+
`without an intentional delete signal (${markerNote}) and no revive — ` +
215+
`renamed file may have been tombstoned by the system. oldPathId=${oldPathId}`,
216+
});
217+
}
218+
}
219+
}
220+
}
221+
222+
return findings;
223+
}

0 commit comments

Comments
 (0)