Skip to content

Compress Parsed DB Matrix#12

Merged
itzmeanjan merged 34 commits intomainfrom
compress-parsed-db-mat
May 5, 2025
Merged

Compress Parsed DB Matrix#12
itzmeanjan merged 34 commits intomainfrom
compress-parsed-db-mat

Conversation

@itzmeanjan
Copy link
Owner

@itzmeanjan itzmeanjan commented Apr 29, 2025

  • Keep DB matrix in compressed representation
    - Reduce memory usage server-side
    - Increase PIR throughput
  • Add more tests, ensuring these optimizations don't introduce any undesired "feature" 🤣
  • Update benchmark results.

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
It uses the fact that the number of the significant bits in each matrix element
is much lesser than 32 i.e. the bit-width of each matrix element. Meaning we can compress
multiple smaller elements into single matrix element.

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
…f compression works correctly

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
It is not possible to correctly compute decompressed matrix's number of
columns just from compressed matrix's number of columns and compression-factor
i.e. how many elements are compressed into a single one.

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
… as expected

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
… transposed database matrix

Decompression is performed on the fly. This change should address the memory bandwidth bottleneck issue
and open path to better performance for large databases.

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
…essed form, after it is transposed

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
…-len > 16

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
…ngle one

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
…k with new fixed compression factor of 2

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Based on whether uncompressed number of columns is even or odd,
it separates codepath, which results in better performance.

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
…e `mat-elem-bit-len` as input anymore

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
…to make test run faster

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
…ve), based on cipher-text-bit-length

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
…arity 4

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
… work with all the compression factors

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
…t-bit-length

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
… database using binary fuse filter

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
…ed-matrix multiplication works

Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
itzmeanjan added 4 commits May 5, 2025 07:15
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
Signed-off-by: Anjan Roy <hello@itzmeanjan.in>
@itzmeanjan itzmeanjan merged commit 7f469fa into main May 5, 2025
5 checks passed
@itzmeanjan itzmeanjan deleted the compress-parsed-db-mat branch May 5, 2025 17:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant