Code

A0-4318: rate-limiter for all#1857

Code

Merged

fixxxedpoint merged 49 commits intomainCardinal-Cryptography/aleph-node:mainfrom A0-4318_rate_limited_substrate_networkCardinal-Cryptography/aleph-node:A0-4318_rate_limited_substrate_networkCopy head branch name to clipboardNov 27, 2024

+1,965-628

Commits

Commits on Nov 13, 2024

• revamped TokenBucket for rate-limiter:

  Show description for 9e967ae
  fixxxedpoint

  committed
  9e967ae

  Copy full SHA for 9e967ae

• rate_limiter:

  Show description for f896b4c
  fixxxedpoint

  committed
  f896b4c

  Copy full SHA for f896b4c

• RateLimitedAsyncRead and Write moved to rate_limiter crate

  fixxxedpoint

  committed
  ab74c45

  Copy full SHA for ab74c45

• custom implementation of Transport with rate-limiting (only read) for substrate's [sc_client::network::NetworkWorker]

  fixxxedpoint

  committed
  0b17e45

  Copy full SHA for 0b17e45

• intergrated rate-limiting for substrate [sc_client::network::NetworkWorker]

  fixxxedpoint

  committed
  03142e2

  Copy full SHA for 03142e2

• increased limits for the rate-limiter, both for alephbft and substrate

  fixxxedpoint

  committed
  2eb7b72

  Copy full SHA for 2eb7b72

• - s/substrate_bit_rate/substrate_network_bit_rate

  Show description for fc806fc
  fixxxedpoint

  committed
  fc806fc

  Copy full SHA for fc806fc

• renamed local variable in build/transport.rs - it no longer references TokenBucket

  fixxxedpoint

  committed
  0a41d22

  Copy full SHA for 0a41d22

• removed unecessary pub-use to [TokenBucket] in rate_limiter/lib.rs

  fixxxedpoint

  committed
  eb6dd52

  Copy full SHA for eb6dd52

• removed unecessary pub in pub mod transport; in [finality-aleph/src/network/build/mod.rs]

  fixxxedpoint

  committed
  b5937b8

  Copy full SHA for b5937b8

• removed newline in [finality-aleph/src/network/build/mod.rs]

  fixxxedpoint

  committed
  d0b0ce5

  Copy full SHA for d0b0ce5

• added impl Default for DefaultTimeProvider in [token_bucket.rs]

  fixxxedpoint

  committed
  d9fdce1

  Copy full SHA for d9fdce1

• removed unecessary .into() conversion of [SleepingRateLimiter] in [finality-aleph/src/network/build/transport.rs]

  fixxxedpoint

  committed
  6ce1663

  Copy full SHA for 6ce1663

• - build_transport moved to [finality-aleph/src/network/build/transport.rs]

  Show description for 019f9a9
  fixxxedpoint

  committed
  019f9a9

  Copy full SHA for 019f9a9

• Default macro for DefaultTimeProvider - review changes

  fixxxedpoint

  committed
  0525c17

  Copy full SHA for 0525c17

• using parking_lot::Mutex in rate_limiter instead of std::sync::Mutex

  fixxxedpoint

  committed
  2dd96c2

  Copy full SHA for 2dd96c2

• simplified TokenBucket algorithm: using only requested field instead of requeste and available. It should be also easier now to transform it into wait-free version.

  fixxxedpoint

  committed
  bbe8e8a

  Copy full SHA for bbe8e8a

• TokenBucket per connection instead of a shared one

  fixxxedpoint

  committed
  e69894b

  Copy full SHA for e69894b

• using new API of substrate::NetworkWorker::new_with_custom_transport

  fixxxedpoint

  committed
  3fe9404

  Copy full SHA for 3fe9404

• TokenBucket return None when rate_limit is 0 instead of some arbitrary big value

  fixxxedpoint

  committed
  7e680da

  Copy full SHA for 7e680da

• default rate-limit for the sync network is now 1 MiB per connection - so far only tested locally using 7 instances of aleph-node.

  fixxxedpoint

  committed
  563de02

  Copy full SHA for 563de02

• fmt for rate_limiter.rs

  fixxxedpoint

  committed
  b049419

  Copy full SHA for b049419

• fixed unit-tests for token-bucket after recent changes of its api

  fixxxedpoint

  committed
  f88d6ac

  Copy full SHA for f88d6ac

• rust-fmt after changes in rate-limiter

  fixxxedpoint

  committed
  4fa4dc0

  Copy full SHA for 4fa4dc0

• added type alias for Box<dyn sc_network::config::NotificationService> in network/substrate.rs

  fixxxedpoint

  committed
  33eb851

  Copy full SHA for 33eb851

• s/substrate_network_bit_rate_per_connection/bit_rate_per_connection in finality-aleph/src/network/build/mod.rs

  fixxxedpoint

  committed
  f791005

  Copy full SHA for f791005

• better readability for TokenBucket.rs

  fixxxedpoint

  committed
  e9cf1de

  Copy full SHA for e9cf1de

• better readability in TokenBucket.rs ++

  fixxxedpoint

  committed
  a2f1eff

  Copy full SHA for a2f1eff

• rust-fmt in bin/node/src/service.rs

  fixxxedpoint

  committed
  6fdecaa

  Copy full SHA for 6fdecaa

• using enum Deadline {Never, Instant} instead of Option<Option<Instant>> in TokenBucket

  fixxxedpoint

  committed
  a379641

  Copy full SHA for a379641

• rust-fmt in tests for TokenBucket

  fixxxedpoint

  committed
  dbeeca2

  Copy full SHA for dbeeca2

• - simpler version of the TokenBucket (rate-limiter)

  Show description for f0dc89c
  fixxxedpoint

  committed
  f0dc89c

  Copy full SHA for f0dc89c

• added bunch of unit-tests for TokenBucket and SharedTokenBucket

  fixxxedpoint

  committed
  374a45f

  Copy full SHA for 374a45f

• - integration of the new SharedTokenBucket with both Substrate-based and Socket-based networks (sync + alephbft)

  Show description for ba07522
  fixxxedpoint

  committed
  ba07522

  Copy full SHA for ba07522

• added ALEPHBFT_NETWORK_BIT_RATE SUBSTRATE_NETWORK_BIT_RATE params to docker/docker_entrypoint.sh to allow configuration of rate-limiting for both sync and alephbft networks

  fixxxedpoint

  committed
  ba90a89

  Copy full SHA for ba90a89

• slightly refactored token_bucket.rs

  fixxxedpoint

  committed
  6c7404f

  Copy full SHA for 6c7404f

• fixed and updated unit tests in token_bucket.rs

  fixxxedpoint

  committed
  1f5b787

  Copy full SHA for 1f5b787

• rust-fmt for token_bucket and related

  fixxxedpoint

  committed
  b11b80c

  Copy full SHA for b11b80c

• added more docs in token_bucket.rs

  fixxxedpoint

  committed
  fc27ed5

  Copy full SHA for fc27ed5

• more docs for token_bucket.rs

  fixxxedpoint

  committed
  fa131da

  Copy full SHA for fa131da

• bumped version of rust-toolchain in nix/versions.nix

  fixxxedpoint

  committed
  c8db058

  Copy full SHA for c8db058

• improved docs in token_bucket.rs

  fixxxedpoint

  committed
  e2e0bb3

  Copy full SHA for e2e0bb3

• removed no longer necessary types in rate-limiter crate - helpers related with impls of AsyncRead, etc.

  fixxxedpoint

  committed
  0f26764

  Copy full SHA for 0f26764

Commits on Nov 14, 2024

• Added a verbose share() method to SharedRateLimiter (and to other types in its hierarchy). It is more readable this way - previous version was using Clone, which could be confusing regarding …

  Show description for 1fbf91b
  fixxxedpoint

  committed
  1fbf91b

  Copy full SHA for 1fbf91b

Commits on Nov 26, 2024

• using Ordering::SeqCst for all atomic operations in token_bucket.rs

  fixxxedpoint

  committed
  82cbe9f

  Copy full SHA for 82cbe9f

• fixed flaky test in token_bucket.rs - mutex

  fixxxedpoint

  committed
  c207292

  Copy full SHA for c207292

• added a comment about Send constraints on the return type of [network::build::transport::build_transport]

  fixxxedpoint

  committed
  657248d

  Copy full SHA for 657248d

• Merge remote-tracking branch 'origin/main' into A0-4318_rate_limited_substrate_network

  fixxxedpoint

  committed
  ec7e914

  Copy full SHA for ec7e914

• FIX: added --locked flag to cargo install subxt-cli --version 0.30.1 in .github/workflows/_check-runtime-metadata.yml. Without it, subxt-cli was unable to intall due to some dependency requiryi…

  Show description for 4989fa2
  fixxxedpoint

  committed
  4989fa2

  Copy full SHA for 4989fa2