feat: headless CLI for vault-to-CRDT sync by enieuwy · Pull Request #16 · kavinsood/yaos

enieuwy · 2026-04-13T17:26:40Z

Summary

Adds a headless Node.js CLI that mirrors a Markdown vault directory to a YAOS CRDT room without Obsidian. The supported alpha target is a Linux server using a local filesystem.

New: `packages/cli/`

yaos-cli daemon — startup reconciliation, filesystem watcher, graceful shutdown.
yaos-cli sync — one reconciliation pass, then exit.
yaos-cli status — connection/cache status as JSON.
Layered config: CLI flags > environment > ~/.config/yaos/cli.json.
Chokidar + Node fs disk mirror for Markdown files.
WebSocket client using the existing YAOS VaultSync CRDT engine with a Node ws polyfill.

Maintainer-feedback hardening

Atomic Node writes: temp file next to target, datasync, rename, parent-directory sync.
Local Yjs state persistence: .yaos-state.bin plus .yaos-state.json, loaded on startup for delta sync across sync/cron runs.
Cache identity checks: state metadata must match host + vault ID; corrupt or cross-room caches fail loudly.
yaos-cli sync warns when no local state cache exists and the run must fetch room state before delta-syncing.
Shared vault-path normalization now enforces Unicode NFC.
Node disk scan rejects non-NFC filesystem entries loudly; watcher logs and skips them.
Server update_required exits with status 2 so systemd can use RestartPreventExitStatus=2, including fatal auth received after daemon startup.
README documents Linux/local-filesystem-only, no NFS/SMB/FUSE/cloud-drive mounts, and one writer per vault directory.

Review follow-up hardening

Shutdown drains pending debounced disk-to-CRDT and CRDT-to-disk work before clearing queues.
CRDT paths with . / .. segments are rejected before filesystem operations.
Writes/deletes/renames reject symlink parent traversal outside the vault root.
Oversized files are skipped by byte size before reading them into memory.
Atomic replacement explicitly reapplies preserved file mode so umask cannot silently tighten permissions.
Remote delete/rename paths sync affected parent directories after directory-entry changes.
First-provider-sync races after a startup timeout now force the deferred authoritative reconciliation.

Core/plugin changes

src/utils/normalizeVaultPath.ts is shared by plugin and CLI and now enforces NFC.
src/sync/vaultSync.ts accepts a persistence factory for non-browser runtimes, detects an already-synced provider before waiting for a future sync event, and exposes fatal-auth notification for headless daemon shutdown.
Existing plugin sync schema and migration logic remain shared.

Explicit constraints

Unsupported in this PR:

Attachment/blob sync.
.obsidian settings/plugin sync.
NFS/SMB/FUSE/cloud-drive mounts.
Multiple writers against the same vault directory, including daemon + Obsidian writing through a shared drive.
Native binary packaging.

Follow-up architecture

The VaultFS/core extraction is not bundled into this PR. I agree with the constraint that YAOS should not maintain two parallel monoliths; I can open a focused prerequisite PR for the VaultFS adapter boundary next, then rebase this CLI onto it if you prefer extraction-first sequencing.

Test plan

Two bugs prevented the filesystem watcher from working: 1. shouldIgnoreNormalizedPath treated null-stats paths as files. When chokidar calls _isIgnored for the root directory without stats, the path was checked against isMarkdownSyncable (returns false for directory names not ending in .md), causing the entire tree to be pruned. Fix: when stats are null, only ignore paths that are definitively non-markdown files (have an extension but not .md). 2. Chokidar's internal _isIgnored calls .map() on the ignored option. When ignored is a bare function, .map() throws TypeError (functions don't have .map), which is silently caught — the watcher starts but watches nothing. Wrapping in an array preserves the function through normalizeIgnored's type check. Also upgraded chokidar from 4.0.3 to 5.0.0.

P1: Add trailing slash removal to normalizeVaultPath to match Obsidian normalizePath() semantics, preventing CRDT key mismatches between plugin and headless clients. P1: Add requireRuntimeConfig() to status command so host/token/vaultId are validated before creating a sync client. P1: Move startMapObservers() before reconcileFromDisk() so remote edits arriving during startup disk scan are immediately mirrored. Safe because observers filter local origins (ORIGIN_SEED). P2: Resolve relative vault dir to absolute path in resolveCliConfig so YAOS_DIR/config values don't depend on process working directory. P2: Reject invalid externalEditPolicy values instead of silently defaulting to the most permissive policy (always). P2: Change createNodeVaultSync to accept RuntimeCliConfig instead of ResolvedCliConfig, making the type system enforce that connection settings are validated before sync client construction. Also includes prior uncommitted fixes: Promise.allSettled for batch writes, fs.rm cleanup on rename fallback, ENOENT handling during vault walk, early reconnection handler installation, reconcileInFlight flag, commander exitOverride, and prepare/dev scripts.

P1 fixes: - nodeDiskMirror: reject path traversal via .. segments in toAbsolutePath() - nodeDiskMirror: don't prune directories when chokidar has no stats - exclude: preserve trailing slash in user exclude patterns P2 fixes: - config: expand ~ in vault directory before resolve - cli: use strict regex for positive integer parsing - config: filter empty strings from CLI overrides in pickDefined - nodeVaultSync: gate reconnect callback until startup state initialized - nodeDiskMirror: write new file before deleting old in rename fallback

…resh # Conflicts: # package-lock.json

enieuwy · 2026-05-02T06:56:14Z

@kavinsood I refreshed this PR on top of current main and pushed the hardening pass in 1ad86ee. Follow-up review hardening is now included through fc81e9b.

Mapping the #11 requirements / follow-up feedback to the current head:

Atomic Node writes — done (writeFileAtomic: temp file, datasync, rename, parent-directory sync; follow-up also reapplies preserved file modes and syncs delete/rename parent directories).
Stateless sync warning — done (yaos-cli sync warns when no .yaos-state.bin is loaded).
Real Yjs state persistence for delta startup — done (.yaos-state.bin + .yaos-state.json).
Identity-checked state cache — done as an additional safety check; host/vault mismatches and corrupt caches fail loudly.
Unicode NFC vault-path normalization — done.
Non-NFC filesystem detection — done; scan fails loudly, watcher logs and skips.
update_required exit code — done; CLI exits 2 for systemd RestartPreventExitStatus=2, including fatal auth received after daemon startup.
Linux/local-fs/single-writer constraints — documented.
Additional review hardening — done: shutdown drains pending work, dot-segment paths are rejected, symlink-parent traversal is rejected, oversized files are checked by bytes before read, and first-provider-sync timeout races trigger authoritative reconciliation.
VaultFS/core extraction — agreed this should come first as a focused prerequisite PR before this CLI PR is mergeable. I’ll take that on next rather than asking to land this alpha first.

Verification on the current head:

npm run @yaos/cli/typecheck
npm run @yaos/cli/test (17/17 passing)
npm run @yaos/cli/build
npm run build
git diff --check

One remaining scope question for the VaultFS extraction PR: should path normalization, exclude semantics, and frontmatter-guard decisions live in the shared layer, or should those remain adapter/runtime responsibilities? My default will be conservative: extract filesystem primitives first and keep normalization/excludes/frontmatter behavior where it already lives unless you prefer otherwise.

enieuwy · 2026-05-02T07:51:57Z

Review follow-up pushed in 40da1bf.

This addresses the review findings from the hardening pass:

drains pending markdown/write debounce queues on daemon shutdown before clearing state;
rejects . / .. CRDT path segments before filesystem operations;
rejects symlink-parent traversal outside the vault root for write/delete/rename paths;
uses byte-size guards before reading oversized Markdown files;
reapplies preserved file mode on atomic temp files so umask cannot change replacement permissions;
fsyncs parent directories after delete/rename directory-entry changes;
races daemon shutdown against fatal auth, so post-start update_required exits through code 2;
fixes the first-provider-sync timeout race so a deferred first sync gets an authoritative reconciliation;
corrects README wording for the no-cache + provider-timeout conservative-mode behavior.

Verification rerun on 40da1bf:

npm run @yaos/cli/typecheck
npm run @yaos/cli/test (17/17 passing)
npm run @yaos/cli/build
npm run build
git diff --check

enieuwy · 2026-05-02T08:00:37Z

Second-pass review found two remaining symlink-safety gaps in the rename fallback path; fixed in fc81e9b.

Changes:

validate the old rename parent before entering fallback, so a symlink-traversal failure cannot fall through to fallback cleanup;
validate the nearest existing ancestor before creating missing directories, so mkdir cannot create through a symlinked path and only fail afterward;
make rename fallback call flushWrite(newPath, true) directly, so the old path is deleted only after the replacement write succeeds.

Verification rerun on fc81e9b:

npm run @yaos/cli/typecheck
npm run @yaos/cli/test (17/17 passing)
npm run @yaos/cli/build
npm run build
git diff --check

enieuwy added 5 commits April 12, 2026 11:03

Add headless YAOS CLI

a6683e1

Add WebSocket polyfill (ws) for Node.js runtime

b2f1471

enieuwy mentioned this pull request Apr 13, 2026

Add headless YAOS CLI (packages/cli) #13

Closed

enieuwy marked this pull request as draft April 13, 2026 17:27

enieuwy marked this pull request as ready for review April 13, 2026 17:27

enieuwy added 6 commits May 2, 2026 12:17

Merge remote-tracking branch 'upstream/main' into feat/headless-cli-f…

b3fa079

…resh # Conflicts: # package-lock.json

fix(cli): atomic writes for headless filesystem output

eb4215e

fix(sync): normalize vault paths to unicode nfc

d018c78

feat(cli): persist identity-checked yjs state

fb6519a

fix(cli): use distinct exit code for update_required

a15740a

docs(cli): document headless runtime constraints

1ad86ee

fix(cli): address headless hardening review findings

40da1bf

fix(cli): close symlink gaps in rename fallback

fc81e9b

enieuwy mentioned this pull request May 2, 2026

refactor: introduce client-side VaultFs boundary #37

Draft

kavinsood force-pushed the main branch 4 times, most recently from fc810fe to 3837c05 Compare May 18, 2026 15:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: headless CLI for vault-to-CRDT sync#16

feat: headless CLI for vault-to-CRDT sync#16
enieuwy wants to merge 13 commits into
kavinsood:mainfrom
enieuwy:feat/headless-cli-fresh

enieuwy commented Apr 13, 2026 •

edited

Loading

Uh oh!

enieuwy commented May 2, 2026 •

edited

Loading

Uh oh!

enieuwy commented May 2, 2026

Uh oh!

enieuwy commented May 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

enieuwy commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

New: packages/cli/

Maintainer-feedback hardening

Review follow-up hardening

Core/plugin changes

Explicit constraints

Follow-up architecture

Test plan

Uh oh!

enieuwy commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

enieuwy commented May 2, 2026

Uh oh!

enieuwy commented May 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

enieuwy commented Apr 13, 2026 •

edited

Loading

New: `packages/cli/`

enieuwy commented May 2, 2026 •

edited

Loading