feat(workspace): add persistence compaction, scaling benchmarks, and trim API surface#1649
Merged
feat(workspace): add persistence compaction, scaling benchmarks, and trim API surface#1649
Conversation
Long desktop sessions with frequent large-value autosaves (e.g. 30 KB notes saved every 30s) accumulated unbounded update logs—28 MB for a 147 KB compact doc over 8 hours. Compaction previously only ran at startup and dispose, leaving the log to grow indefinitely mid-session. Add mid-session compaction that fires when accumulated bytes since last compaction exceed 2 MB, debounced by 5s to avoid interrupting bulk writes. compactUpdateLog now returns boolean so the byte counter only resets when compaction actually ran (prevents infinite retry when the compacted doc itself exceeds the 2 MB BLOB limit). Also: inline single-caller initPersistenceDb, add missing clearLocalData lifecycle hook (was present on IndexedDB persistence but absent here— desktop users couldn't wipe local data on sign-out).
…), and storage edge cases Nine new tests across three describe blocks covering gaps the existing suite missed: - Insert cliff detection (1K→50K progression, cliff at 25K) - Single-row update time vs table size (proves autosave safe at 50K) - Bulk update vs bulk insert asymmetry (updates 3-7x slower) - Editing intensity at 10K rows (1x/5x/20x edits, <4% overhead) - Full lifecycle at 10K (add→edit 3x→delete all, 36 bytes residual) - Mixed permanent + churning rows (10 cycles, +22 bytes total) - Single-row high-frequency edits (500x, 0.0 bytes growth) - Multi-client storage overhead (~22 bytes/client) - Cold load parse time (10K rows in 10ms)
The vault CLI config imports createFujiWorkspace and createFujiMaterializer from @epicenter/fuji subpath exports, but neither existed after the workspace.ts → workspace/ directory refactor. This adds both, matching tab-manager's factory pattern. - workspace/workspace.ts: createFujiWorkspace() factory wrapping createWorkspace - workspace/index.ts: re-export factory + fujiWorkspace definition - materializer.ts: one-way markdown materializer that reads document content per entry (the generic markdownMaterializer only handles table row data) - package.json: add ./materializer export + slugify/filenamify deps
…and Drizzle re-exports Remove 4 internal-only symbols from the root barrel that had zero external callers: createUnionSchema, standardSchemaToJsonSchema, createTimeline, generateInitialOrders. These remain available at their source paths for internal use. Remove 17 Drizzle ORM operator re-exports (eq, gt, and, or, sql, etc.) — no consumer imports these from @epicenter/workspace; they import directly from drizzle-orm. Reduces root barrel from 218 to 185 lines.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Foundation stack starts here because the pieces underneath the rest of the series needed to stop fighting the data model. SQLite persistence was happily appending Yjs updates forever, which meant long-lived desktop sessions could turn a healthy document into an oversized update log. At the same time,
@epicenter/workspacewas exporting Drizzle helpers, schema utilities, and other internals as if they were part of the public contract, which made the API look broader—and fuzzier—than it really is.This PR fixes both ends of that problem. Persistence now compacts itself once the accumulated update log crosses a byte threshold, so the database stays bounded without any manual cleanup. The compaction is transparent to consumers—they still wire it up the same way:
The workspace package also stops re-exporting internal Drizzle and validation helpers, so consumers import those from the packages that actually own them instead of getting them for free through a grab bag entrypoint:
Fuji gets a proper workspace factory export, which keeps its app-specific setup in one place:
The new scaling benchmarks are there to make sure this doesn't regress quietly. They pin down insert cliffs, O(n) update behavior, and storage growth under heavy edit patterns before the rest of the stack builds on top of it.
5 commits, 7 files changed, +599/-73. First in a stack of 5—merge this before pr/2-materializer-factory.