-
Notifications
You must be signed in to change notification settings - Fork 5
More efficient storage storage format #27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Merged
860ea04
to
32005ca
Compare
stevensJourney
approved these changes
Sep 19, 2024
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changes
ps_oplog
to only store PUT operations, roughly in the format described here.Specific changes on ps_oplog:
superseded
column.op
column.bucket
column from TEXT (bucket name) to INTEGER (bucket id).Instead of storing pending REMOVE operations in
ps_oplog
, it is now stored in a dedicatedps_updated_rows
table. Only the row type and id are relevant here - it indicates rows which may need to be re-synced from the local bucket data.This has some advantages:
The operations
delete_pending_buckets
andclear_remove_ops
are now no-ops, and may be removed from the respective SDKs.Benchmarks
Flutter desktop on linux, 500k PUT operations in a single
global[]
bucket, time initial sync and database size.Before: 56s total, of which 18s is spent in
sync_local
. 401MB database.After: 57s total, of which 19s is spent in
sync_local
. 390MB database.Conclusion: Does not significantly affect initial sync time. Does slightly reduces database size. May have a bigger effect when syncing many REMOVE operations, when deleting buckets, or when there are long bucket names.
Tests
This now adds dart-based tests to test the extension - specifically the schema migrations. This could be any high-level language - just used Dart because the tests were already written in Dart.
Other tests may be moved from the powersync.dart repo to here in the future.