Suggestion: Add Threaded read-ahead decompression

[kcgen]

Threaded read-ahead decompression

• Threaded, so latency isn't added to the current requested read block.
• The Read-ahead thread should wait until the requested block is returned to avoid competing for IO simultaneously versus the current request.
• If a requested location already exists in the decoded read-ahead blocks, then it's returned immediately and any new delta read-ahead block are decoded to the cache.
• Read-ahead distance in KiB is configurable via API (which gets converted to N compressed blocks internally).
• If -1 is passed in for the read-ahead distance, then the read-ahead thread will keep going until it decodes the last block (only pausing to yield to current requests to avoid contention), likewise the read-ahead memory cache will expand to hold all decoded blocks.

[rtissera]

Two first bullet points are doable and would be really an improvement Le sam. 24 oct. 2020 à 01:16, kcgen <[email protected]> a écrit :

…

- *Threaded read-ahead decompression* Configurable via API in KiB (which gets converted to N compressed blocks internally). Threaded, so latency isn't added to the requested read. If a requested location already exists in the decoded set of read-ahead blocks, then it's returned immediately and any new read-ahead block are decoded to the cache. - *LRU cache* Configurable via API in KiB. So requests for already decompressed data can be returned immediately instead of burning another decompression round. - *ZStandard Codec Support* (via LibZStandard); with pass-through of API settings for common ZStandard features - like number of cores, block sizes, compression settings, and so on. The API will allow emulator-specific and platform-specific settings to optimize the performance of of the library. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#34>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAW4WBJXHYIH4ACLQSJA3K3SMIFGBANCNFSM4S5EKCEQ> .

[rtissera]

A straightforward implementation of this using pthread implementation is being considered.

[Zero3K]

Any news regarding it?