Add SQLite session support

[rodydavis]

• FFI implementation.
• Check resource leaks on FFI
• WASM implementation
• Allow accessing old/new/conflicting values in conflict callback

[simolus3]

Thanks! I'll take a look at WASM support tomorrow. I've changed some things to make the API feel a bit more native to Dart (e.g. it makes no sense to me to have enable() and disable() methods, we can just have a bool enabled). I've also added APIs to iterate over a changeset.

Regarding the implementation, I think we should not add bindings for stuff we don't need (yet). This includes the conflict handling / rebasing API that is still marked as experimental. Another concern is avoiding resource leaks, I've added native finalizers around the session / changeset classes to make sure they get deallocated natively when they're no longer referenced in Dart. I definitely need to check this again more carefully though.
Something we definitely need to check before merging this is that package:sqlite3 must continue to work against sqlite3 builds without the session extension (as long as the session methods are not actually used of course).

I've also started writing tests for this, but I couldn't get the extension to work. This test fails due to the session still being empty after making a change. Do you know what could be up with that? It's entirely possible I broke something refactoring the native implementation, but the _create and _attach calls look good to me so I don't understand what's going wrong here.

test('isEmpty', () {
  final database = sqlite3.openInMemory();
  final session = Session(database);
  expect(session.isEmpty, isTrue);
  expect(session.isNotEmpty, isFalse);

  // Change without attaching session
  database.execute('INSERT INTO entries DEFAULT VALUES;');
  expect(session.isEmpty, isTrue);

  session.attach();
  database.execute('INSERT INTO entries VALUES (?);', ['my first entry']);

  expect(session.isEmpty, isFalse); // fails!
});

[rodydavis]

Thanks! I'll take a look at WASM support tomorrow. I've changed some things to make the API feel a bit more native to Dart (e.g. it makes no sense to me to have enable() and disable() methods, we can just have a bool enabled). I've also added APIs to iterate over a changeset.

I was trying to expose the Node.js Session abstraction but totally fine with those changes. I would prefer as close to the C api as possible.

Regarding the implementation, I think we should not add bindings for stuff we don't need (yet). This includes the conflict handling / rebasing API that is still marked as experimental.

I was debating including them but totally fine not having them until they are stable!

Another concern is avoiding resource leaks, I've added native finalizers around the session / changeset classes to make sure they get deallocated natively when they're no longer referenced in Dart.

Good call, I wasn't sure how to implement that with what I was seeing in the code base.

I definitely need to check this again more carefully though.
Something we definitely need to check before merging this is that package:sqlite3 must continue to work against sqlite3 builds without the session extension (as long as the session methods are not actually used of course).

As far as my testing, there should be no reason it would not work with any version of sqlite that does not have the proper compile flags. You for sure need them to call the methods though. Probably could check the config object for the two flags if needed too.

I've also started writing tests for this, but I couldn't get the extension to work. This test fails due to the session still being empty after making a change. Do you know what could be up with that? It's entirely possible I broke something refactoring the native implementation, but the _create and _attach calls look good to me so I don't understand what's going wrong here.

This tripped me up when first working with it and the issue is the table must have a primary key to be tracked! It cannot be a temp or virtual table. The primary key must also be deterministic (like INTEGER or TEXT).

CREATE TABLE IF NOT EXISTS my_table (id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT);
INSERT INTO my_table (name) VALUES ('1'), ('2'), ('3');

"The session extension only works with tables that have a declared PRIMARY KEY. The PRIMARY KEY of a table may be an INTEGER PRIMARY KEY (rowid alias) or an external PRIMARY KEY."
https://sqlite.org/sessionintro.html

Also these 2 flags must be set when compiling:

-DSQLITE_ENABLE_SESSION -DSQLITE_ENABLE_PREUPDATE_HOOK

The official WASM build does have these on by default and works pretty well. Having them in the flutter_sqlite_libs would be amazing.

[rodydavis]

Also for the isEmpty test:

"By default, this function always returns 0. For it to return a useful result, the sqlite3_session object must have been configured to enable this API using sqlite3session_object_config() with the SQLITE_SESSION_OBJCONFIG_SIZE verb."
https://sqlite.org/session/sqlite3session_changeset_size.html

That got me too and I thought it was a bug till I looked at the docs!

[simolus3]

Hi, sorry for the delay on this. I think the memory leaks on FFI and wasm support are pretty much done now, but I'm not quite sure why this test is failing (it reports an empty diff whereas I would expect an update for the content column). Do you have any idea?

[rodydavis]

Ok looking at the branch again!

My initial guess is that the table needs to be created before attaching:

 test('diff', () {
    var session = Session(database);
    database.execute('INSERT INTO entries (content) VALUES (?);', ['a']);

    database
      ..execute(
          'CREATE TABLE another.entries (id INTEGER PRIMARY KEY, content TEXT);')
      ..execute("ATTACH ':memory:' AS another;")
      ..execute('INSERT INTO another.entries (content) VALUES (?);', ['b']);

    session = Session(database)..diff('another', 'entries');
    final changeset = session.changeset();
    expect(changeset, [
      isOp(
          operation: SqliteUpdateKind.update,
          oldValues: [1, 'b'],
          newValues: [1, 'a'])
    ]);
  }, skip: "TODO: Figure out what I'm dong wrong");

Another thing from the docs, is that the primary key must be present:

If zTbl does not exist, or if it does not have a primary key, this function is a no-op (but does not return an error).

So an explicit or auto incrementing key must be used.

---

I am having trouble running the tests: 00:20 +426 ~10 -10: Some tests failed.

On my Mac I compiled the SQLite wasm from source and also ran the tests with this: dart test -P full.

Is there something missing I need to do to run a local test suite for wasm with the session extension?

[simolus3]

On my Mac I compiled the SQLite wasm from source and also ran the tests with this: dart test -P full.

Is there something missing I need to do to run a local test suite for wasm with the session extension?

Is it only the web tests that are failing for you? -P full runs them as well but it's probably fine to test native only. For wasm you would have to follow these steps to compile a sqlite3.wasm with session support used in tests.

[simolus3]

I've manually merged this into main and will release it soon. I'll also take a look at ensuring we support this from sqlite3_flutter_libs.

[simolus3]

Released in sqlite3: 2.8.0, sqlite3_native_assets: 0.0.8 and sqlite3_flutter_libs: 0.5.38. Thanks for your contribution!

[rodydavis]

Thank you! 🚀