feat(repo): Merge nats s3 hybrid publishing into main #364
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* feat(publisher): Publish streams to S3 * feat(publisher): add Localstack for testing s3 locally * feat(publisher): Ignore localstack data * feat(publisher): Fix build errors from integrating S3Client * feat(publisher): Ensure published objects can be streamed Also update tests * feat(publisher): Consistently hash static payload for s3 path * feat(publisher): Use testnet or mainnet buckets based on the env * feat(publisher): Distinct env vars for S3
* feat(repo): Added draft ws version * feat(repo): Fixes after AJs merge * feat(repo): Added docker-compose script for creating S3 bucket * fix(repo): Added better deserialization * fix(publisher): re-allow ctrl c shutdown publisher process * refactor(repo): Disallow nested Cargo.lock * refactor(publisher): Include default S3 bucket for local development * feat(repo): Improved serializations * feat(repo): Removed binary for ws-streamer * feat(repo): Pass any wildcard to stream filtering * feat(repo): Added deliver policy to ws pattern for historical data * feat(repo): Switched to tokio-tungestenite * feat(repo): Small change to websocket subscriptions --------- Co-authored-by: AJ <[email protected]>
* fix(repo): Solving publisher slowness (#352) * feat(repo): add sv-emitter package * build(repo): adjust run-publisher * feat(repo): first version of consumer * feat(repo): add the rest of payloads * refactor(repo): reuse executors * fix(repo): tests * build(repo): configure devops to emitter/consumer * build(repo): update chart version * fix(repo): consumer cli argument * build(repo): docker action * build(repo): fix consumer chart * build(repo): fix consumer chart * build(repo): fix consumer chart * build(repo): chart * fix(repo): general stuff * build(repo): docker action * ci(repo): fix * fix(repo): chain config * fix(repo): docker images * feat(repo): improve docker build * refactor(repo): a lot of improvements * fix(repo): fix helm chart nats config * feat(repo): worked using mirror * fix(repo): adjust nats cluster config * build(repo): adjust github action * build(repo): update chart version * build(repo): fix docker images * build(repo): lint warnings * ci(repo): adjust docker action * build(repo): fix docker file * fix(repo): stream creation * build(repo): fix tls for nats * build(repo): fix chart * build(repo): fix chart * build(repo): fix chart * build(repo): fix chart * fix(repo): use sts instead of deployment for consumer * fix(repo): fix chart * fix(repo): remove initcontainer * fix(repo): update chart * fix(repo): fix resources on consumer * fix(repo): bump chart * fix(repo): bump chart * fix(repo): bump chart * build(repo): fix makefile * build(repo): add publisher into one single Dockerfile * ci(repo): fix docker publish action * build(repo): fix build docker * ci(repo): fix docker action * ci(repo): fix tests * ci(repo): fix ci.yaml * build(repo): adjust fuel dependencies features * build(repo): Add DNS/TLS configuration for Nats (#357) * build(repo): Add DNS/TLS configuration for Nats * build(repo): bump chart version * build(repo): fix chart * build(repo): fix chart * build(repo): fix secretName on nats client websocket config * build(repo): Update NATS dependency on chart (#358) * build(repo): Update NATS dependency on chart * build(repo): Bump chart version * refactor(repo): remove FuelNetwork from nats * refactor(repo): remove unused benches * refactor(repo): general improvements * fix(repo): general fixes * build(repo): adjust chart for new architecture * build(repo): adjust chart * build(repo): fix chart * build(repo): fix chart * build(repo): fix chart * build(repo): fix chart * build(repo): docker image * build(repo): bump chart * build(repo): fix emitter Dockerimage * build(repo): adjust workflow dispatch for docker * build(repo): docker action * build(repo): fix publisher config * build(repo): bump chart * build(repo): general fixes and adjustments * build(repo): bump chart * fix(repo): s3 client workaround * refactor(repo): rename crates to be consistent * build(repo): bump chart * build(repo): some adjusts * fix(consumer): remove max_messages * refactor(repo): change s3 path * build(repo): bump chart * refactor(repo): perf improvements * refactor(webserver): send decoded payload direct on websocket
0xterminator
requested review from
luizstacio,
pedronauck and
Jurshsmith
as code owners
0xterminator
changed the title
pedronauck
approved these changes
Dec 23, 2024
0xterminator
force-pushed
the
nats-s3-hybrid-publishing-backup
branch
4 times, most recently
from
c857a09 to
7ca2d91
Compare
0xterminator
force-pushed
the
nats-s3-hybrid-publishing-backup
branch
from
7ca2d91 to
22eda74
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.