Skip to content

Performance Improvements in Compression logic for brotli/Gzip and add zstd - #6592

Closed
ResuBaka wants to merge 6 commits into
clockworklabs:masterfrom
ResuBaka:compression/brotli-and-zstd
Closed

ResuBaka wants to merge 6 commits into
clockworklabs:masterfrom
ResuBaka:compression/brotli-and-zstd

Conversation

@ResuBaka

Copy link
Copy Markdown
Contributor

Description of Changes

Improves WebSocket compression for small updates and large subscription responses:

  • Adds opt-in Zstd fast-2 support to the server and Rust SDK. Which is a faster then both Brotli and Gzip and also produces most of the time smaller messages.
  • Tunes Brotli to quality 2 and explicitly selects the pure-Rust zlib-rs gzip backend.
  • Lowers compression attempts from >1 KiB to ≥64 bytes, retaining compressed output only when smaller. This is also testes on the client rust sdk with captured traffic from bitcraft tool to test with real traffic.
  • Adds size-gated multithreading with a shared half-CPU budget and synchronous fallback. Which help with faster responses for initial messages.
  • Includes reproducible synthetic benchmarks and documented tradeoffs.

Related to #6319’s compression-threshold concern and #5686’s Zstd request.

API and ABI breaking changes

Adds Compression::Zstd, requiring updates to downstream exhaustive Rust matches. Zstd is explicitly requested; existing codecs, defaults, and message schemas remain compatible. Older servers reject the new option.

Rollback safety impact

n/a — no persisted-format changes. Clients requesting Zstd must select an older codec when connecting to a rolled-back server.

Expected complexity level and risk

3/5. Changes compression scheduling, buffer ownership, and codec negotiation. Concurrent MT jobs can increase transient memory. MT remains disabled with core pinning; performance findings are synthetic, not production guarantees.

Testing

  • Core, client API, protocol-message, and Rust SDK suites passed.
  • Added roundtrip/corruption, cutoff, size-fallback, buffer-reclamation, and concurrent-budget coverage.
  • Verified actual server frames decode through the SDK parser; browser build and one-CPU compression tests passed.
  • Isolated release benchmarks compare gzip backends and levels. Browser runtime and loaded-server performance remain untested.

@ResuBaka ResuBaka changed the title Performance Improvements in Compression logic for brotli and add zstd Performance Improvements in Compression logic for brotli/Gzip and add zstd Nov 16, 2026
@bfops

bfops commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Our AI suggests that this PR covers several substantial changes that could be split into smaller PRs for easier review:

  1. Compression threshold and size fallback — the 64-byte threshold and keeping compressed output only when smaller.
  2. Brotli/gzip tuning — Brotli quality 2, zlib-rs, and associated benchmarks.
  3. Opt-in Zstd — negotiation, synchronous server compression, and native/browser Rust SDK decoding together.
  4. Parallel compression — worker threads, shared CPU budget, size thresholds, buffer ownership, and fallback behavior.

@ResuBaka

Copy link
Copy Markdown
Contributor Author

I have done some more testing. I will not do the change for Brotli quality as that is a lot depended on the schema/data which one is better at size and/or speed. So not a general good change to go for level 2. If we would look at decode speed that would change it. There it could make sense as with quality we could save the client 15% to 50% on decode time. So I'm going to check in the multi thread change if maybe moving to a different quality level is a good change when we can use multi lanes/threads in Brotli.

But that will then come as a followup branch/pr to the multi thread compression.

For the rest I will move them to there own PR's.

@ResuBaka ResuBaka closed this Nov 18, 2026
@bfops

bfops commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Awesome, thank you for following up!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants