Experiment with compressing GPBFT messages and certs

[masih]

Modify observer to report space gain by compressing GPBFT messages using zstd.

Add a compression test that measures space gain in compressing generated finality certificates.

The results run on filecoin/46 passive testing show that compressing GPBFT messages gain ~60% in space saving.

The latest captured so far:

messages observed so far: 37648
Min compression gain: 7.4 %
Max compression gain: 69.0 %
Avg compression gain: 57.3 %

Relates to: #757

[masih]

You can run this experiment yourself while a passive testing is unpaused using:

$ go run ./cmd/f3 observe \
  --bootstrapAddr /dns/bootstrap-venus.mainnet.filincubator.com/tcp/8888/p2p/QmQu8C6deXwKvJP2D8B6QGyhngc3ZiDnFzEHBDx8yeBXST \
  --bootstrapAddr /dns/bootstrap-mainnet-0.chainsafe-fil.io/tcp/34000/p2p/12D3KooWKKkCZbcigsWTEu1cgNetNbZJqeNtysRtFpq7DTqw3eqH \
  --bootstrapAddr /dns/bootstrap-mainnet-1.chainsafe-fil.io/tcp/34000/p2p/12D3KooWGnkd9GQKo3apkShQDaq1d6cKJJmsVe6KiQkacUk1T8oZ \
  --bootstrapAddr /dns/bootstrap-mainnet-2.chainsafe-fil.io/tcp/34000/p2p/12D3KooWHQRSDFv4FvAjtU32shQ7znz7oRbLBryXzZ9NMK2feyyH \
  --manifestServerID 12D3KooWENMwUF9YxvQxar7uBWJtZkA6amvK4xWmKXfSiHUo2Qq7 \
  --rotateInterval 10m \
  --retention 868h

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 36 lines in your changes missing coverage. Please review.

Project coverage is 69.19%. Comparing base (be650d0) to head (c56941f).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
observer/observer.go	0.00%	36 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #787      +/-   ##
==========================================
- Coverage   69.55%   69.19%   -0.36%     
==========================================
  Files          77       77              
  Lines        7896     7931      +35     
==========================================
- Hits         5492     5488       -4     
- Misses       1954     1991      +37     
- Partials      450      452       +2     

Files with missing lines	Coverage Δ
observer/observer.go	0.00% <0.00%> (ø)

... and 4 files with indirect coverage changes

[rvagg]

Any ballpark idea of what this is going to add to CPU usage of F3? These are core messages constantly being sent and received, right, so the overhead is not going to be trivial?

[masih]

Any ballpark idea of what this is going to add to CPU usage of F3?

I think it would be a single percent increase? But have not run an experiment to specifically measure that. Theoretically speaking zstd is very resource efficient, specially for the size of messages we are taking about here.

[masih]

In terms of at scale measurement, the plan is to make compression into a flag settable in manifest that we can experiment with.

[masih]

Closing now that compression is implemented in #793