What's the story for database versioning and data migrations?

[BastiDood]

Awesome project here. As I've been scouring the ecosystem for similar solutions for an app that I want to build, SignalDB has been the most promising so far. However, one question that's been nagging at me after digging through the documentation and the reference is:

What do production-grade database versioning and data migration patterns look like?

Setting up collections is easy and even quite type-safe, but how do we migrate collections when schema additions/changes/deletions (inevitably) arise? There are no examples and documentation pages that mention it, so I thought that I'd ask here.

For what it's worth, Dexie.js has an elegant API for this exact problem.

When applied to the SignalDB case, though, I'd imagine it wouldn't be as flexible and straightforward because collections are referenced by strings in Dexie.js whereas they are referenced by Collection class instances in SignalDB. If such an API exists in SignalDB, the old collection classes will have to be persisted in the codebase (for a longggggggg time) until all consumers of the old schema have migrated.

// Here is how I imagine it might look like for a to-do app.

// We would have to keep the old collection in the codebase. :(
// We also have to somehow maintain a versioning scheme. Maybe like this?
interface OldNotes { ... }
const OldNotes = new Collection<OldNotes>('notes', { version: 1 });

// The new schema would have to co-exist with the old version. :(
// Observe that we've updated the version number.
interface NewNotes { ... }
const NewNotes = new Collection<NewNotes>('notes', { version: 2 });

// We would then have some kinda "migration manager" that reconciles
// migrations between versions *atomically*. Correctness of the migration
// is the responsibility of the developer. SignalDB only cares about the interface
// (much like how the `SyncManager` is currently implemented).
const manager = new MigrationManager({
  // A reference to the "canonical" collection (i.e., the most up-to-date version).
  canonical: NewNotes,
  // An array of previously canonical versions of the collection.
  stale: [OldNotes, ...],
  // The migrate helper that migrates the data from `stale` collections to the `canonical`
  // representation. This migration should be atomic. This will be invoked until no stale
  // collections remain in the queue.
  migrate({ canonical, stale }) {
    canonical.batch(() => {
      switch (stale.version) {
        case 0: {
          const data = stale.find({ ... });
          canonical.updateMany(...);
          // fall through?
        }
        case 1: {
          const data = stale.find({ ... });
          canonical.replaceMany(...);
          // fall through?
        }
      }
    });
  },
});

// To run the migration, we may provide options to the `migrate` helper.
// One option that I'm imagining would be useful (but probably best to leave off
// by default as it is in this example) is the `truncate` flag, which determines
// whether old collections should be preserved, truncated, or disposed.
const result = await migration.migrate({
  // `preserve` keeps the collection untouched after the migration. This is the safest option.
  // `truncate` clears all the rows, but keeps the collection in the persistence adapter.
  // `dispose` deletes the collection entirely. Old code paths may still accidentally invoke `OldNotes`, though.
  cleanup: 'preserve' | 'truncate' | 'dispose',
});

// Alternatively, the cleanup phase can also be done in userland if migration succeeds.
if (result.success) {
  // Best practice: `OldNotes` should not be exported to the UI code so that nobody uses it anymore.
  OldNotes.dispose();
}

Now, technically, this could all be implemented in userland via ad hoc naming conventions for collections (e.g., notes-v1 and notes-v2). But, it would be nice to have first-class versioning support by the library.

Whichever path we take—whether first-class versioning support or userland migrations—I'd like to see this more explicitly endorsed in the documentation to set the correct expectations on the scope and limitations of SignalDB.

Note

I personally wouldn't mind if data migration is beyond the scope of the project, but a nudge towards community solutions or an endorsed/blessed design pattern in the docs would be enough.

[michaelkrog]

One way to tackle it if you have a server to sync with and its API is versioned, is to only do migration on the server side and then on the client side persist collections with a name that includes the version of the api.

fx. todos-v2025-05-12 for the v2025-05-12 version of the API.

Then you always know that the entities from the server will always match what you the model on the client side. And when you decide to upgrade to a newer version of the API, you retrieve all entities again from your API and persist your collections with the new version in the name. You are then free to trash the old collections.

It is more data traffic, but I feel it is a more simple approach to ensure all devices agrees on the data model in use.

[BastiDood]

It is more data traffic, but I feel it is a more simple approach to ensure all devices agrees on the data model in use.

That is true and quite unfortunate. I suppose the underlying assumption here is that the local-first data is not a lot when scoped at the user level. I'm willing to conjecture that that is indeed the case for many single-user apps.