Skip to content

feat(workspace): replace markdownMaterializer with createMaterializer factory#1650

Merged
braden-w merged 13 commits intomainfrom
pr/2-materializer-factory
Apr 12, 2026
Merged

feat(workspace): replace markdownMaterializer with createMaterializer factory#1650
braden-w merged 13 commits intomainfrom
pr/2-materializer-factory

Conversation

@braden-w
Copy link
Copy Markdown
Member

@braden-w braden-w commented Apr 12, 2026

The markdown materializer started life as a single monolithic function that owned serialization, file writing, and Yjs observation all at once. That was fine while the API was still being figured out, but the spec kept moving—closure-based document access, opt-in materialization, factory composition—so the implementation had to stop pretending the shape was settled.

This PR follows that path and lands on a workspace-bound createMaterializer(...) factory. The materializer now plugs into the extension chain, waits for whenReady before reading state, and lets each table or KV namespace opt in explicitly instead of forcing everything through one big markdown path.

// before: one function did everything
markdownMaterializer({ directory: './data', tables })

// after: compose the materializer at the workspace boundary
.withWorkspaceExtension('materializer', (ctx) =>
  createMaterializer(ctx, { dir: './data' })
    .table('posts', { serialize: slugFilename('title') })
    .kv(),
)

While getting there, the repeated “compute once inside an observer” pattern got pulled into a reusable lazy() helper so those caches stop being hand-rolled in three different places.

const getAllEntries = lazy(() => yarray.toArray())

const entries = getAllEntries()

That cleanup also exposed a real scaling bug in YKeyValueLww: bulk updates were re-scanning the array on every write, which made the hot path quadratic. The fix batches the conflict cleanup so the observer resolves the batch once instead of repeatedly walking the same data.

There’s also a short article on the lazy initializer pattern, plus doc updates to keep the workspace docs aligned with the current API instead of the older chaining examples.

Stacks on #1649. 13 commits, 14 files changed, +1322/-438.

braden-w added 13 commits April 12, 2026 15:26
…est instead of delete

Breddit is a local-first Reddit data browser built on the existing
ingest pipeline. The workspace cleanup spec now moves ingest/ to
apps/breddit/ instead of deleting it.
…ment access

Planning doc for redesigning the markdown materializer API to support
document content through the extension closure, eliminating app-specific
materializers in fuji and opensidian. Explores typed table helpers,
SerializeFn callbacks, and composable serialize presets.
… args pattern

The .version() builder API was replaced by variadic args months ago.
All 163 call sites in TypeScript already use the correct pattern, but
the workspace-api skill doc and 4 articles still showed the old form.
…y pattern

Evolves the spec from markdown-specific to format-agnostic:
- createMaterializer(tables, config) factory with typed .table() chain
- General { filename, content } serialize contract
- markdown() helper for the common frontmatter + body case
- KV materialization to single JSON file
- Default-materialize-all with per-table overrides
- First arg is now ctx ({ tables, kv }) instead of just tables
- Structurally typed so it receives extension context directly
- Added open questions section from abstraction boundary audit:
  preset naming, default-materialize-all assumptions, KV observation,
  markdown() link conversion opt-out, whenReady timing, MaybePromise
- .kv({ skip: true }) → .skipKv() for consistency with .skip()
- MaybePromise<T> → import from lifecycle.ts (already exists)
- KV observation confirmed: kv.observeAll() exists for live updates
- markdown() opt-out: toMarkdown() is the documented escape hatch
- Serialize presets: JSDoc @remarks documents markdown output
- whenReady: JSDoc documents lazy start after .table()/.skip() calls
- Default-materialize-all: JSDoc documents .skip() for non-content tables
…dd kv serialize

Three design refinements:
- Opt-in over default-materialize-all: .table() and .kv() explicitly
  opt in. No .skip()/.skipKv() needed—API is purely additive.
- dir everywhere: consistent short name for base path and subfolder.
- .kv({ serialize }) receives typed KV snapshot, returns SerializeResult.
  No global default serialize—would lose row type inference.
Without awaiting ctx.whenReady, the materializer races persistence/sync
and reads empty tables on first materialization. Both existing app-specific
materializers already do this—the generic one must too.
Skip deleteEntryByKey in set() when inside an outer transaction (batch
case). The observer handles conflict resolution in one pass using a lazy
entryIndexMap instead of repeated indexOf scans. Individual (non-batched)
set() calls keep the eager delete to avoid double observer invocation.

- Completed spec items 1.1–1.7
- Deviation: conditional skip vs unconditional (see spec note)
The perf commit (728043f) introduced inline let+null-check caches in the
YKeyValueLww observer. This extracts that pattern into a reusable lazy()
helper with detailed JSDoc, removing the manual cache bookkeeping.
Covers the sync lazy() helper discovered while optimizing YKeyValueLww,
contrasted with the async lazy singleton pattern already documented.
… factory

Implement the new typed createMaterializer(ctx, { dir }) factory with
.table() and .kv() opt-in builder chains. General serialize contract
({ filename, content }) replaces the markdown-specific return shape.

- createMaterializer with TTables/TKv generics for type-safe table names and row inference
- markdown() helper wraps toMarkdown + wikilink conversion
- slugFilename/bodyField presets return SerializeResult via markdown()
- toSlugFilename/toIdFilename standalone utilities
- KV materialization via kv.observeAll() with JSON default
- Completed spec items 1.1-1.11
Base automatically changed from pr/1-foundation to main April 12, 2026 08:54
@braden-w braden-w merged commit 8ed083b into main Apr 12, 2026
2 of 9 checks passed
@braden-w braden-w deleted the pr/2-materializer-factory branch April 12, 2026 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant