Integrate zstd compression into chain exchange

[masih]

The GPBFT message exchange over pubsub already uses zstd compression on top of CBOR encoded messages. The work here integrates the same style of compression for chain exchange messages, with additional unification of the encoding mechanism across the two.

The work refactors the root level encoding implementation into a generic encoder decoder that both chain exchange and gpbft used. Tests and benchmarks are updated to reflect this.

The benchmarking of partial gmessage encoding is also adjusted to fix a few redundant statements and bugs in testing.

Fixes #819

[codecov]

Codecov Report

Attention: Patch coverage is 60.37736% with 21 lines in your changes missing coverage. Please review.

Project coverage is 67.45%. Comparing base (9c11ba3) to head (7e3205a).

Files with missing lines	Patch %	Lines
internal/encoding/encoding.go	60.52%	10 Missing and 5 partials ⚠️
chainexchange/pubsub.go	44.44%	2 Missing and 3 partials ⚠️
host.go	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #842      +/-   ##
==========================================
+ Coverage   67.39%   67.45%   +0.06%     
==========================================
  Files          84       84              
  Lines        8915     8918       +3     
==========================================
+ Hits         6008     6016       +8     
+ Misses       2380     2376       -4     
+ Partials      527      526       -1     

Files with missing lines	Coverage Δ
host.go	64.92% <83.33%> (+0.67%)	⬆️
chainexchange/pubsub.go	77.15% <44.44%> (-0.92%)	⬇️
internal/encoding/encoding.go	60.52% <60.52%> (ø)

... and 3 files with indirect coverage changes

[Stebalien]

Can we add a compression bomb test? I think we're fine, but it would be nice to have a test.

• The fact that we're stream decoding into the CBOR decoder coupled with the fact that the CBOR decoder has a bunch of limits should protect us against memory issues.
• The fact that our CBOR decoder won't read through unlimited input (e.g., doesn't have logic that skips unknown fields, etc.) means we can't get "stuck" when decoding. But it would be nice if we could wrap the decompressed reader in a limited reader before passing it off to the CBOR decoder, just in case. That way we can guarantee that we'll process at most N bytes of CBOR before giving up.

[Kubuxu]

Are we going with ZSTD by default?

[masih]

For chain exchange yes. For GPBFT it's configurable via manifest.

Happy to make it configurable for chain exchange too if you think it's worth doing.

[masih]

Can we add a compression bomb test?

Sure. Captured #843