Persist Checksums #26
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Incrementally maintain bucket checksums to improve performance when syncing a large number of rows. The effect can be quite noticeable with 100k+ synced rows, and to a smaller extent from 10k+ rows already.
Additionally, reduce the cases where a "compact" (removing REMOVE operations) is performed.
This does not change any of the consistency properties. Any change in server-side checksums, such as after sync rule changes, will still be detected.
Manually deleting a row from the local ps_oplog table will not be detected after this change. We could add a full checksum validation to test for those cases.
This includes a migration for two new columns on
ps_buckets. The down migration can be used to revert back to v0.2.0 without issue.Benchmarks
Flutter app, Linux desktop.
Start with 500k PUT operations, then time the latency of syncing 1 PUT or REMOVE operation.
Latency is timed from the client receiving the new checkpoint, to the changes being fully applied locally.
Before: ~860ms
After: ~3ms
After, REMOVE operation: ~300ms
REMOVE operations are still expensive due to the compact performed afterwards. The compact can be performed only if there are a high number of REMOVE operations, which will amortize the cost (added as a new option to the
clear_remove_opsoperation). To completely remove the need for the compact operation, see #27.Release note
This will likely be released as part of #27, rather than as a standalone release.