Package review PR #1

KarthikSubbarao · 2024-04-29T19:40:11Z

This is a PR which merges current changes in unstable into an empty branch in order to help with a review code of the entire codebase.

src/commands/bloom.rs

hpatro · 2024-05-01T22:03:26Z

src/commands/bloom.rs

+    let mut result = Vec::new();
+    match value {
+        Some(bf) => {
+            for item in input_args.iter().take(argc).skip(idx) {
+                result.push(RedisValue::Integer(bf.add_item(item.as_slice())));
+            }
+            Ok(RedisValue::Array(result))
+        }
+        None => {
+            if nocreate {
+                return Err(RedisError::Str("ERR not found"));
+            }
+            let mut bf = BloomFilterType::new_reserved(fp_rate, capacity, expansion);
+            for item in input_args.iter().take(argc).skip(idx) {
+                result.push(RedisValue::Integer(bf.add_item(item.as_slice())));
+            }
+            match filter_key.set_value(&BLOOM_FILTER_TYPE, bf) {
+                Ok(_) => Ok(RedisValue::Array(result)),
+                Err(_) => Err(RedisError::Str(ERROR)),
+            }
+        }


Could we check the bloom filter exists or not first and then insert the data in a single flow. Don't like the code duplication around insertion.

I can create a separate function for multi adds and call it from both flows. I wanted to handle both through the same flow, however it will be out of reference scope and result in a moved error

hpatro · 2024-05-01T22:08:06Z

src/commands/bloom_util.rs

+pub struct BloomFilterType {
+    pub expansion: u32,
+    pub fp_rate: f32,
+    pub filters: Vec<BloomFilter>,


Does the underlying support scalable bloom filters or do we need to support this mechanism?

This particular library does not have auto scaling

https://docs.rs/bloomfilter/1.0.13/bloomfilter/struct.Bloom.html

src/commands/bloom_util.rs

hpatro · 2024-05-01T22:12:18Z

src/commands/bloom_util.rs

+                let new_capacity = filter.capacity * self.expansion;
+                let mut new_filter = BloomFilter::new(self.fp_rate, new_capacity);
+                // Add item.
+                new_filter.bloom.set(item);
+                new_filter.num_items += 1;
+                self.filters.push(new_filter);


Shouldn't we fail the request if the expansion is much higher than we can support?

expansion really means expansion_rate. I can rename this variable if that would help clarify this.

I don't think there is a limit on the number of sub filters an object can have. But (if we want) we can define a config for this to either silently fail expansion (by allowing a set) beyond a limit of X sub filters per object OR explicitly return an error (without a set).

One other aspect we should handle this checking / rejecting based on memory overhead of every operation that creates a new BloomFilter object (BF.ADD, BF.MADD, BD.RESERVE, BF.INSERT, RDB Load). Before any of these operations, we should probably check the memory usage and reject the operations if there is not sufficient space. We have a mechanism to compute the estimated additional memory overhead per creation.

src/wrapper/bloom_callback.rs

src/commands/bloom_data_type.rs

Signed-off-by: Viktor Söderqvist <[email protected]> Co-authored-by: Madelyn Olson <[email protected]>

Signed-off-by: Karthik Subbarao <[email protected]>

… objects. Update RDB load/save Signed-off-by: Karthik Subbarao <[email protected]>

Signed-off-by: Karthik Subbarao <[email protected]>

…ters and memory_bytes. Additionally updated drop and created tests around this new info handler (#19) Signed-off-by: zackcam <[email protected]>

* add bf.load and bf.dump to support bloom filter aofrewrite. Signed-off-by: wuranxx <[email protected]> add version in BloomFilterType. change bf.load behavior: bf.load can't override exists keys. Signed-off-by: wuranxx <[email protected]> * remove bf.dump. add bloom filter encode decode comment. add unit test for decode error case. Signed-off-by: wuranxx <[email protected]> * 1. Currently, the version field is added to the array only encode. 2. Add metrics when decode bloomFilterType. 3. Add metrics test when load bloom filter from aof. Signed-off-by: wuranxx <[email protected]> * 1. Add filter bytes check and unit test. 2. Fixed some comments and returned error messages. Signed-off-by: wuranxx <[email protected]> * fix `test_bf_decode_when_bytes_is_exceed_limit_should_failed` unit test. Signed-off-by: wuranxx <[email protected]> * Add expansion maximum value check to filter decode. Signed-off-by: wuranxx <[email protected]> --------- Signed-off-by: wuranxx <[email protected]>

Signed-off-by: Vanessa Tang <[email protected]>

* Support for DEBUG DIGEST module data type callback Signed-off-by: Nihal Mehta <[email protected]> * Update test cases Signed-off-by: Nihal Mehta <[email protected]> * Move digest to wrapper Signed-off-by: Nihal Mehta <[email protected]> * Update tests Signed-off-by: Nihal Mehta <[email protected]> * Add more scenarios for debug test Signed-off-by: Nihal Mehta <[email protected]> * Clean code and add scenario for debug test Signed-off-by: Nihal Mehta <[email protected]> --------- Signed-off-by: Nihal Mehta <[email protected]>

…filter dependency to version 3.0 (#23) * Updating how we create BloomFilter from rdb loads. BloomFilter vec now has capacity of filter we are loading from Signed-off-by: zackcam <[email protected]> * Updating bloomfilter dependency to version 3, fixing breaking changes as well Signed-off-by: zackcam <[email protected]> * Updating the digest changes to follow updated version of bloom. As well as removing unnecesary fields saved in rdb Signed-off-by: zackcam <[email protected]> * Update log in src/bloom/data_type.rs Signed-off-by: KarthikSubbarao <[email protected]> * Update comment in src/bloom/utils.rs Signed-off-by: KarthikSubbarao <[email protected]> * Clippy error in src/bloom/data_type.rs Signed-off-by: KarthikSubbarao <[email protected]> --------- Signed-off-by: zackcam <[email protected]> Signed-off-by: KarthikSubbarao <[email protected]> Co-authored-by: KarthikSubbarao <[email protected]>

Signed-off-by: Karthik Subbarao <[email protected]>

Signed-off-by: Nihal Mehta <[email protected]>

…n a non bloom key (#30) Signed-off-by: zackcam <[email protected]>

…r structs (#24) * Updating defragmentation to defrag both bloomfiltertype and bloomfiler structs Signed-off-by: zackcam <[email protected]> * Draft: Extra debugging and more factors being defragged Signed-off-by: zackcam <[email protected]> * Updating defrag and tests to use cursors and make test more robust by getting hits Signed-off-by: zackcam <[email protected]> * Fixing merge conflicts Signed-off-by: zackcam <[email protected]> * Fixing merge conflicts with random seed. Updating defrag test to use mexists and add_items_till_capacity. Refactored metric tracking as well to reduce code repetition Signed-off-by: zackcam <[email protected]> --------- Signed-off-by: zackcam <[email protected]>

…commands contain the correct acl categories (#29) Signed-off-by: zackcam <[email protected]>

Signed-off-by: VanessaTang <[email protected]>

…e properties on replica nodes to match the object on the primary (#32) Signed-off-by: Karthik Subbarao <[email protected]>

…s get incremented in defrag test (#33) Signed-off-by: zackcam <[email protected]>

…fail (#34) * Deflaking tests to remove chance of false positives causing tests to fail Signed-off-by: zackcam <[email protected]> * Update tests/test_bloom_command.py Co-authored-by: KarthikSubbarao <[email protected]> Signed-off-by: zackcam <[email protected]> --------- Signed-off-by: zackcam <[email protected]> Co-authored-by: KarthikSubbarao <[email protected]>

Signed-off-by: Nihal Mehta <[email protected]>

Signed-off-by: Karthik Subbarao <[email protected]>

Signed-off-by: VanessaTang <[email protected]>

#37) Signed-off-by: Karthik Subbarao <[email protected]>

Signed-off-by: Karthik Subbarao <[email protected]>

Signed-off-by: KarthikSubbarao <[email protected]>

Signed-off-by: zackcam <[email protected]>

KarthikSubbarao force-pushed the unstable branch 3 times, most recently from 61aa1fb to 0e4d4d1 Compare April 30, 2024 18:14

KarthikSubbarao changed the title ~~Package review~~ Package review PR May 1, 2024

KarthikSubbarao self-assigned this May 1, 2024

hpatro reviewed May 1, 2024

View reviewed changes

KarthikSubbarao force-pushed the unstable branch from 055728d to 7027032 Compare July 16, 2024 03:25

KarthikSubbarao force-pushed the unstable branch from c87fd97 to 869a254 Compare August 23, 2024 17:20

KarthikSubbarao pushed a commit that referenced this pull request Aug 23, 2024

Add information about RFC process (#1)

657f83c

Signed-off-by: Viktor Söderqvist <[email protected]> Co-authored-by: Madelyn Olson <[email protected]>

KarthikSubbarao added 21 commits September 10, 2024 23:06

Initial commit

5237a10

Signed-off-by: Karthik Subbarao <[email protected]>

Add support for BF.ADD, BF.EXISTS, BF.CARD

a96c08a

Signed-off-by: Karthik Subbarao <[email protected]>

Update README

be97a25

Signed-off-by: Karthik Subbarao <[email protected]>

Optimize bloom operations to use a reference to already created Bloom…

5ebc1cb

… objects. Update RDB load/save Signed-off-by: Karthik Subbarao <[email protected]>

Remove older Bloom APIs using prev serialization apporach

8557b55

Signed-off-by: Karthik Subbarao <[email protected]>

Add Dev Profile in Cargo.toml for debugging

1729520

Signed-off-by: Karthik Subbarao <[email protected]>

Create Module Datatype and support RDB load, save, free

be80d9d

Signed-off-by: Karthik Subbarao <[email protected]>

Update README.md

ed4d0f9

Signed-off-by: Karthik Subbarao <[email protected]>

Add support for BF.RESERVE and expansion config

2db0e4b

Signed-off-by: Karthik Subbarao <[email protected]>

Add support for BF.INFO

778c219

Signed-off-by: Karthik Subbarao <[email protected]>

Add support for BF.MEXISTS

b11970e

Signed-off-by: Karthik Subbarao <[email protected]>

Add support for BF.MADD

bdf5f0d

Signed-off-by: Karthik Subbarao <[email protected]>

Update error handling / messages and update expansion logic

8f5e3fc

Signed-off-by: Karthik Subbarao <[email protected]>

Add auto scaling support for bloom filters

5545680

Signed-off-by: Karthik Subbarao <[email protected]>

Fix RDB Save for scaled filters

42d015c

Signed-off-by: Karthik Subbarao <[email protected]>

Refactoring

481ecac

Signed-off-by: Karthik Subbarao <[email protected]>

Update TODOs

ff60817

Signed-off-by: Karthik Subbarao <[email protected]>

Fix mem_usage calculation

1c1b6ab

Signed-off-by: Karthik Subbarao <[email protected]>

Update Cargo.toml

d00fa99

Signed-off-by: Karthik Subbarao <[email protected]>

minor refactoring

053d5e9

Signed-off-by: Karthik Subbarao <[email protected]>

Add support for BF.INSERT and fix multi add logic

fbdfb5b

Signed-off-by: Karthik Subbarao <[email protected]>

zackcam and others added 2 commits November 5, 2024 12:01

Adding info handler, that contains three fields, num_objects, num_fil…

31d21c0

…ters and memory_bytes. Additionally updated drop and created tests around this new info handler (#19) Signed-off-by: zackcam <[email protected]>

KarthikSubbarao force-pushed the unstable branch 2 times, most recently from e74b825 to 188835b Compare November 21, 2024 21:50

Add new metrics to show capacity and items across objects (#20)

a33e0e3

Signed-off-by: Vanessa Tang <[email protected]>

KarthikSubbarao force-pushed the unstable branch from 188835b to a33e0e3 Compare November 21, 2024 21:52

KarthikSubbarao force-pushed the unstable branch from 3f24e34 to 7c25468 Compare November 30, 2024 07:25

zackcam and others added 2 commits December 3, 2024 22:02

Support random seed per bloom object by default (configurable) (#27)

1b0d819

Signed-off-by: Karthik Subbarao <[email protected]>

KarthikSubbarao force-pushed the unstable branch from 99c6f58 to 1b0d819 Compare December 5, 2024 08:06

nnmehta and others added 6 commits December 6, 2024 13:29

Add tightening_ratio support in BloomFilterType structure (#26)

f10a4e5

Signed-off-by: Nihal Mehta <[email protected]>

Adding a wrong type error message when performing a bloom operation o…

eaa218f

…n a non bloom key (#30) Signed-off-by: zackcam <[email protected]>

Adding bloom acl category to commands. Added tests to check that the …

a5ec8de

…commands contain the correct acl categories (#29) Signed-off-by: zackcam <[email protected]>

Update default configuration and related test (#28)

4a8af80

Signed-off-by: VanessaTang <[email protected]>

Deterministic replication - bloom object creation should have the sam…

8b94c17

…e properties on replica nodes to match the object on the primary (#32) Signed-off-by: Karthik Subbarao <[email protected]>

KarthikSubbarao force-pushed the unstable branch from 391d947 to 8b94c17 Compare December 12, 2024 21:35

zackcam and others added 4 commits December 12, 2024 18:30

Adding bloom defrag hits and misses to info, added check these metric…

3589d30

…s get incremented in defrag test (#33) Signed-off-by: zackcam <[email protected]>

Add TIGHTENING Ratio and FP_Rate configs support (#31)

9baf6fc

Signed-off-by: Nihal Mehta <[email protected]>

Fix size check and update build script to fail on warnings

6a514bf

Signed-off-by: Karthik Subbarao <[email protected]>

KarthikSubbarao force-pushed the unstable branch from fcaad77 to 6a514bf Compare December 19, 2024 18:27

YueTang-Vanessa and others added 2 commits January 2, 2025 19:22

Add test for loading rdb onto non-bloom cluster (#36)

1c949fb

Signed-off-by: VanessaTang <[email protected]>

Large bloom object handling + Rename to BloomFilterType to BloomObject (

ab158bb

#37) Signed-off-by: Karthik Subbarao <[email protected]>

KarthikSubbarao force-pushed the unstable branch from 9859dd6 to ab158bb Compare January 11, 2025 00:04

ACL Updates, BF.INSERT updates, other misc updates (#38)

a9b578e

Signed-off-by: Karthik Subbarao <[email protected]>

KarthikSubbarao force-pushed the unstable branch from d3d7522 to a9b578e Compare January 12, 2025 02:55

KarthikSubbarao and others added 2 commits January 11, 2025 20:01

Update README.md (#39)

1bb7d25

Signed-off-by: KarthikSubbarao <[email protected]>

Adding a bloom command keyspace test (#40)

591ab10

Signed-off-by: zackcam <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package review PR #1

Package review PR #1

KarthikSubbarao commented Apr 29, 2024 •

edited

Loading

hpatro May 1, 2024

KarthikSubbarao May 1, 2024

hpatro May 1, 2024

KarthikSubbarao May 1, 2024

hpatro May 1, 2024

KarthikSubbarao May 1, 2024

KarthikSubbarao May 2, 2024

Package review PR #1

Are you sure you want to change the base?

Package review PR #1

Conversation

KarthikSubbarao commented Apr 29, 2024 • edited Loading

hpatro May 1, 2024

Choose a reason for hiding this comment

KarthikSubbarao May 1, 2024

Choose a reason for hiding this comment

hpatro May 1, 2024

Choose a reason for hiding this comment

KarthikSubbarao May 1, 2024

Choose a reason for hiding this comment

hpatro May 1, 2024

Choose a reason for hiding this comment

KarthikSubbarao May 1, 2024

Choose a reason for hiding this comment

KarthikSubbarao May 2, 2024

Choose a reason for hiding this comment

KarthikSubbarao commented Apr 29, 2024 •

edited

Loading