Skip to content

Releases: Intsights/PyDeduplines

v0.6.1

14 Feb 16:00
0c3a214
Compare
Choose a tag to compare
  • added support for apple m1 users

V0.6.0 python.311

17 Jan 11:02
e0d66e7
Compare
Choose a tag to compare

V0.6.0 python.311

v0.5.0

02 Feb 14:37
Compare
Choose a tag to compare
  • Added Python3.10 support
  • Upgrade dependencies versions

v0.4.0

23 Sep 16:02
f8e38b2
Compare
Choose a tag to compare

This branch introduces a refactoring overhaul to improve performance dramatically.
There have been a number of changes in this branch that improve the performance and accuracy.
Using a low-level approach, we are now able to iterate through lines without transferring data from binary format to utf8 or allocating memory.
As opposed to the default Rust implementation, ahash was used as the hash algorithm.

v0.3.1

07 Jul 12:12
Compare
Choose a tag to compare
  • The library now supports being interrupted by signals such an SigInt (Ctrl + C)
  • The library now propagates errors back to the caller

v0.3.0

04 Jul 11:15
Compare
Choose a tag to compare

Library overhauled from top to bottom. The library has been rewritten in Rust to fix memory and runtime bugs.
There was a substantial improvement in performance compared to the previous version. Rather than shipping source wheels to compile, it now ships binary wheels. The API has been changed and the class has been removed in favor of free functions. To the functions should now be passed the working directory and thread count.

v0.2.0

24 Aug 14:45
Compare
Choose a tag to compare
  • reimplemented the whole library to use an on-disk algorithm with parallel computation
  • removed mimalloc usage
  • updated all the other dependencies
  • test runner is now pytest

Special thanks to @yonatan-intsights for the unique awesome algorithm contribution.