feat(repo): Merge nats s3 hybrid publishing into main (#364)

* feat(publisher): Extend NATS publishing by storing payloads in S3 (#348) * feat(publisher): Publish streams to S3 * feat(publisher): add Localstack for testing s3 locally * feat(publisher): Ignore localstack data * feat(publisher): Fix build errors from integrating S3Client * feat(publisher): Ensure published objects can be streamed Also update tests * feat(publisher): Consistently hash static payload for s3 path * feat(publisher): Use testnet or mainnet buckets based on the env * feat(publisher): Distinct env vars for S3 * fix(publisher): Remove unused cli arg (#353) * fix(publisher): Remove trailing hyphen in bucket name (#354) * fix(publisher): Don't defer error logging (#355) * build(repo): Update Cargo.lock * feat(repo): Added ws streamer service (#345) * feat(repo): Added draft ws version * feat(repo): Fixes after AJs merge * feat(repo): Added docker-compose script for creating S3 bucket * fix(repo): Added better deserialization * fix(publisher): re-allow ctrl c shutdown publisher process * refactor(repo): Disallow nested Cargo.lock * refactor(publisher): Include default S3 bucket for local development * feat(repo): Improved serializations * feat(repo): Removed binary for ws-streamer * feat(repo): Pass any wildcard to stream filtering * feat(repo): Added deliver policy to ws pattern for historical data * feat(repo): Switched to tokio-tungestenite * feat(repo): Small change to websocket subscriptions --------- Co-authored-by: AJ <[email protected]> * feat(repo): Added webserver service and added docker build step * fix(repo): use cli instead of config.toml * fix(ws): remove duplicated nats clients * fix(ws): use nats_url instead of FuelNetwork * feat(repo): Removed unused args by infra for ws streamer * feat: Adjusts to release S3 + WebServer (#361) * fix(repo): Solving publisher slowness (#352) * feat(repo): add sv-emitter package * build(repo): adjust run-publisher * feat(repo): first version of consumer * feat(repo): add the rest of payloads * refactor(repo): reuse executors * fix(repo): tests * build(repo): configure devops to emitter/consumer * build(repo): update chart version * fix(repo): consumer cli argument * build(repo): docker action * build(repo): fix consumer chart * build(repo): fix consumer chart * build(repo): fix consumer chart * build(repo): chart * fix(repo): general stuff * build(repo): docker action * ci(repo): fix * fix(repo): chain config * fix(repo): docker images * feat(repo): improve docker build * refactor(repo): a lot of improvements * fix(repo): fix helm chart nats config * feat(repo): worked using mirror * fix(repo): adjust nats cluster config * build(repo): adjust github action * build(repo): update chart version * build(repo): fix docker images * build(repo): lint warnings * ci(repo): adjust docker action * build(repo): fix docker file * fix(repo): stream creation * build(repo): fix tls for nats * build(repo): fix chart * build(repo): fix chart * build(repo): fix chart * build(repo): fix chart * fix(repo): use sts instead of deployment for consumer * fix(repo): fix chart * fix(repo): remove initcontainer * fix(repo): update chart * fix(repo): fix resources on consumer * fix(repo): bump chart * fix(repo): bump chart * fix(repo): bump chart * build(repo): fix makefile * build(repo): add publisher into one single Dockerfile * ci(repo): fix docker publish action * build(repo): fix build docker * ci(repo): fix docker action * ci(repo): fix tests * ci(repo): fix ci.yaml * build(repo): adjust fuel dependencies features * build(repo): Add DNS/TLS configuration for Nats (#357) * build(repo): Add DNS/TLS configuration for Nats * build(repo): bump chart version * build(repo): fix chart * build(repo): fix chart * build(repo): fix secretName on nats client websocket config * build(repo): Update NATS dependency on chart (#358) * build(repo): Update NATS dependency on chart * build(repo): Bump chart version * refactor(repo): remove FuelNetwork from nats * refactor(repo): remove unused benches * refactor(repo): general improvements * fix(repo): general fixes * build(repo): adjust chart for new architecture * build(repo): adjust chart * build(repo): fix chart * build(repo): fix chart * build(repo): fix chart * build(repo): fix chart * build(repo): docker image * build(repo): bump chart * build(repo): fix emitter Dockerimage * build(repo): adjust workflow dispatch for docker * build(repo): docker action * build(repo): fix publisher config * build(repo): bump chart * build(repo): general fixes and adjustments * build(repo): bump chart * fix(repo): s3 client workaround * refactor(repo): rename crates to be consistent * build(repo): bump chart * build(repo): some adjusts * fix(consumer): remove max_messages * refactor(repo): change s3 path * build(repo): bump chart * refactor(repo): perf improvements * refactor(webserver): send decoded payload direct on websocket * feat(repo): integrate rust sdk with websockets * build(repo): fix tests * docs(repo): adjust main readme --------- Co-authored-by: AJ <[email protected]> Co-authored-by: Pedro Nauck <[email protected]> Co-authored-by: AJ <[email protected]>
FuelLabs · Dec 23, 2024 · baad7e8 · baad7e8
1 parent ad5a3af
commit baad7e8
Show file tree

Hide file tree

Showing 189 changed files with 7,015 additions and 14,793 deletions.
diff --git a/.env.sample b/.env.sample
@@ -1,11 +1,31 @@
-# Common Configuration
+# Authentication & Security
 KEYPAIR=generated-p2p-secret
+JWT_AUTH_SECRET=generated-secret
+
+# AWS S3 Configuration
+AWS_ACCESS_KEY_ID=test
+AWS_SECRET_ACCESS_KEY=test
+AWS_ENDPOINT_URL=http://localhost:4566
+AWS_REGION=us-east-1
+AWS_S3_ENABLED=false
+AWS_S3_BUCKET_NAME=fuel-streams-local
+
+# NATS Configuration
 NATS_URL=nats://localhost:4222
-NATS_ADMIN_PASS=generated-secret
-NATS_SYSTEM_PASS=generated-secret
+NATS_PUBLISHER_URL=nats://localhost:4333
+NATS_SYSTEM_USER=sys
+NATS_SYSTEM_PASS=sys
+NATS_ADMIN_USER=admin
+NATS_ADMIN_PASS=admin
+NATS_PUBLIC_USER=default_user
+NATS_PUBLIC_PASS=""
+
+# Monitoring & Logging
 USE_ELASTIC_LOGGING=false
-USE_PUBLISHER_METRICS=true
+USE_METRICS=true
 PUBLISHER_MAX_THREADS=16
+
+# Elasticsearch Configuration
 ELASTICSEARCH_URL=http://127.0.0.1:9200
 ELASTICSEARCH_USERNAME=elastic
 ELASTICSEARCH_PASSWORD=generated-secret

diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -1 +1 @@
-* @luizstacio @pedronauck @jurshsmith @0xterminator
+* @luizstacio @pedronauck @jurshsmith @0xterminator @cue8it
diff --git a/.github/actions/setup-rust/action.yaml b/.github/actions/setup-rust/action.yaml
@@ -21,8 +21,16 @@ runs:
     - name: Create .env file with NATS environment variables
       shell: bash
       run: |
-        echo "NATS_ADMIN_PASS=${NATS_ADMIN_PASS:-default_pass}" >> .env
-        echo "NATS_PUBLIC_PASS=${NATS_PUBLIC_PASS:-temp-public-pass}" >> .env
+        set_env_var() {
+          echo "$1=${!1:-$2}" >> $GITHUB_ENV
+          echo "$1=${!1:-$2}" >> .env
+        }
+        set_env_var "NATS_SYSTEM_USER" "sys"
+        set_env_var "NATS_SYSTEM_PASS" "sys"
+        set_env_var "NATS_ADMIN_USER" "admin"
+        set_env_var "NATS_ADMIN_PASS" "admin"
+        set_env_var "NATS_PUBLIC_USER" "default_user"
+        set_env_var "NATS_PUBLIC_PASS" ""
 
     - name: Install Rust
       uses: dtolnay/rust-toolchain@master

diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -43,10 +43,13 @@ jobs:
             deps
             release
             core
-            publisher
             data-parser
+            networks
             fuel-streams
             macros
+            publisher
+            storage
+            ws
 
   lockfile:
     name: Validate Lockfile
@@ -206,18 +209,28 @@ jobs:
     runs-on: ubuntu-latest
     env:
       NATS_URL: nats://127.0.0.1:4222
-      NATS_ADMIN_PASS: secret
-      NATS_PUBLIC_PASS: secret
+      NATS_PUBLISHER_URL: nats://127.0.0.1:4333
+      NATS_SYSTEM_USER: sys
+      NATS_SYSTEM_PASSWORD: sys
+      NATS_ADMIN_USER: admin
+      NATS_ADMIN_PASS: admin
+      NATS_PUBLIC_USER: default_user
+      NATS_PUBLIC_PASS: ""
+      AWS_ACCESS_KEY_ID: test
+      AWS_SECRET_ACCESS_KEY: test
+      AWS_REGION: us-east-1
+      AWS_ENDPOINT_URL: http://localhost:4566
+      AWS_S3_BUCKET_NAME: fuel-streams-test
     strategy:
       fail-fast: false
       matrix:
         package:
-          # - fuel-data-parser
+          - fuel-data-parser
           - fuel-streams
           - fuel-streams-core
           - fuel-streams-macros
-          - fuel-streams-publisher
-
+          - sv-webserver
+          - sv-publisher
     steps:
       - uses: actions/checkout@v4
 
@@ -251,7 +264,9 @@ jobs:
       fail-fast: false
       matrix:
         package:
-          - fuel-streams-publisher
+          - sv-consumer
+          - sv-publisher
+          - sv-webserver
         is_release:
           - ${{ github.ref == 'refs/heads/main' || github.event_name == 'workflow_dispatch' }}
         platform:
@@ -320,34 +335,34 @@ jobs:
           rustup target add ${{ matrix.platform.target }}
           cargo build --release --locked --target ${{ matrix.platform.target }} --package ${{ matrix.package }}
 
-      - name: Strip binaries
-        run: ./scripts/strip-binary.sh "${{ matrix.platform.target }}"
-
-      - name: Set Artifact Name
-        id: artifact-name
-        shell: bash
-        run: |
-          echo "value=${{ matrix.package }}-${{ matrix.platform.os_name }}" >> $GITHUB_OUTPUT
-
-      - name: Package as archive
-        shell: bash
-        run: |
-          cd target/${{ matrix.platform.target }}/release
-          tar czvf ../../../${{ steps.artifact-name.outputs.value }}.tar.gz ${{ matrix.package }}
-          cd -
-
-      - name: Publish release artifacts
-        uses: actions/upload-artifact@v4
-        if: >-
-          (github.event_name == 'push' &&
-          github.ref == 'refs/heads/main' &&
-          contains(github.event.head_commit.message, 'ci(release): Preparing')) ||
-          github.event_name == 'workflow_dispatch'
-        with:
-          name: ${{ steps.artifact-name.outputs.value }}
-          path: ${{ matrix.package }}-*
-          if-no-files-found: error
-          retention-days: 30
+      # - name: Strip binaries
+      #   run: ./scripts/strip-binary.sh "${{ matrix.platform.target }}"
+
+      # - name: Set Artifact Name
+      #   id: artifact-name
+      #   shell: bash
+      #   run: |
+      #     echo "value=${{ matrix.package }}-${{ matrix.platform.os_name }}" >> $GITHUB_OUTPUT
+
+      # - name: Package as archive
+      #   shell: bash
+      #   run: |
+      #     cd target/${{ matrix.platform.target }}/release
+      #     tar czvf ../../../${{ steps.artifact-name.outputs.value }}.tar.gz ${{ matrix.package }}
+      #     cd -
+
+      # - name: Publish release artifacts
+      #   uses: actions/upload-artifact@v4
+      #   if: >-
+      #     (github.event_name == 'push' &&
+      #     github.ref == 'refs/heads/main' &&
+      #     contains(github.event.head_commit.message, 'ci(release): Preparing')) ||
+      #     github.event_name == 'workflow_dispatch'
+      #   with:
+      #     name: ${{ steps.artifact-name.outputs.value }}
+      #     path: ${{ matrix.package }}-*
+      #     if-no-files-found: error
+      #     retention-days: 30
 
   release:
     name: Create Release with Knope
@@ -368,14 +383,14 @@ jobs:
       - name: Checkout Repository
         uses: actions/checkout@v4
 
-      - name: Download Artifacts
-        uses: actions/download-artifact@v4
-        with:
-          path: artifacts
-          merge-multiple: true
+      # - name: Download Artifacts
+      #   uses: actions/download-artifact@v4
+      #   with:
+      #     path: artifacts
+      #     merge-multiple: true
 
-      - name: List Artifacts
-        run: ls -R artifacts
+      # - name: List Artifacts
+      #   run: ls -R artifacts
 
       - name: Run Knope Action
         uses: knope-dev/[email protected]

diff --git a/.github/workflows/docker_publish.yaml b/.github/workflows/docker_publish.yaml
@@ -2,6 +2,17 @@ name: Build and publish Docker image
 
 on:
   workflow_dispatch:
+    inputs:
+      package:
+        type: choice
+        description: "Package to build and publish"
+        default: "all"
+        required: true
+        options:
+          - all
+          - sv-publisher
+          - sv-webserver
+          - sv-consumer
   push:
     branches:
       - main
@@ -23,10 +34,10 @@ jobs:
     strategy:
       matrix:
         package:
-          - name: fuel-streams-publisher
-            image: cluster/docker/fuel-core.Dockerfile
-          - name: sv-emitter
-            image: cluster/docker/fuel-core.Dockerfile
+          - name: sv-webserver
+            image: cluster/docker/sv-webserver.Dockerfile
+          - name: sv-publisher
+            image: cluster/docker/sv-publisher.Dockerfile
           - name: sv-consumer
             image: cluster/docker/sv-consumer.Dockerfile
     steps:
@@ -36,13 +47,14 @@ jobs:
         id: sha
         run: echo "short_sha=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
 
-      - name: Build and push Docker (${{ steps.sha.outputs.short_sha }})
+      - name: Build and push Docker for ${matrix.package.name} (${{ steps.sha.outputs.short_sha }})
+        if: |
+          (github.event_name == 'workflow_dispatch' && (github.event.inputs.package == 'all' || github.event.inputs.package == matrix.package.name)) ||
+          github.event_name != 'workflow_dispatch'
         uses: ./.github/actions/docker-publish
         id: publish
         with:
           username: ${{ github.repository_owner }}
           password: ${{ secrets.GITHUB_TOKEN }}
           image: ghcr.io/fuellabs/${{ matrix.package.name }}
           dockerfile: ${{ matrix.package.image }}
-          build-args: |-
-            PACKAGE_NAME=${{ matrix.package.name }}
diff --git a/.github/workflows/helm_publish.yaml b/.github/workflows/helm_publish.yaml
@@ -2,108 +2,53 @@ name: Build and Publish Helm Chart
 
 on:
   workflow_dispatch:
-    inputs:
-      chart:
-        description: "Select the Helm chart to deploy"
-        required: true
-        type: choice
-        options:
-          - fuel-streams-publisher
-          - fuel-streams
   push:
     branches:
       - main
-    paths:
-      - cluster/charts/fuel-streams-publisher/Chart.yaml
-      - cluster/charts/fuel-streams/Chart.yaml
+  release:
+    types:
+      - published
+
+env:
+  CHART_NAME: fuel-streams
+  CHART_PATH: cluster/charts/fuel-streams
 
 permissions:
   contents: read
+  packages: write
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
 
 jobs:
   helm-release:
-    name: Build Helm Charts
     runs-on: ubuntu-latest
-    if: |
-      github.event_name == 'workflow_dispatch' ||
-      (github.event_name == 'release' && github.event.action == 'published') ||
-      github.ref == 'refs/heads/main' ||
-      github.event_name == 'pull_request'
-    permissions:
-      contents: read
-      packages: write
     steps:
-      - name: Check out code
-        uses: actions/checkout@v4
-
-      - name: Determine charts to process
-        id: charts
-        run: |
-          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
-            echo "charts=${{ github.event.inputs.chart }}" >> $GITHUB_OUTPUT
-          else
-            echo "charts=fuel-streams-publisher fuel-streams" >> $GITHUB_OUTPUT
-          fi
-
-      - name: Validate Chart Names
-        run: |
-          for chart in ${{ steps.charts.outputs.charts }}; do
-            if [ ! -d "cluster/charts/$chart" ]; then
-              echo "Error: Chart '$chart' does not exist."
-              exit 1
-            fi
-          done
-
+      - uses: actions/checkout@v4
       - name: Helm Dependencies Update
         run: |
-          set -e
-          for chart in ${{ steps.charts.outputs.charts }}; do
-            echo "Updating dependencies for $chart"
-            helm dependency update cluster/charts/$chart
-          done
+          cd ${{ env.CHART_PATH }} && helm dependency update
 
-      - name: Get chart versions
-        id: versions
+      - name: Get chart version
+        id: version
         run: |
-          publisher_version=$(awk '/^version:/ {print $2}' cluster/charts/fuel-streams-publisher/Chart.yaml)
-          streams_version=$(awk '/^version:/ {print $2}' cluster/charts/fuel-streams/Chart.yaml)
-          echo "publisher_version=$publisher_version" >> $GITHUB_OUTPUT
-          echo "streams_version=$streams_version" >> $GITHUB_OUTPUT
-
-      - name: "Build chart: [fuel-streams-publisher v${{ steps.versions.outputs.publisher_version }}]"
-        if: contains(steps.charts.outputs.charts, 'fuel-streams-publisher')
-        uses: bsord/[email protected]
-        with:
-          useOCIRegistry: true
-          registry-url: oci://ghcr.io/fuellabs/helmcharts
-          username: ${{ github.repository_owner }}
-          access-token: ${{ secrets.GITHUB_TOKEN }}
-          force: true
-          chart-folder: ./cluster/charts/fuel-streams-publisher
+          version=$(awk '/^version:/ {print $2}' ${{ env.CHART_PATH }}/Chart.yaml)
+          echo "version=$version" >> $GITHUB_OUTPUT
 
-      - name: "Build chart: [fuel-streams v${{ steps.versions.outputs.streams_version }}]"
-        if: contains(steps.charts.outputs.charts, 'fuel-streams')
+      - name: "Build chart: [${{ env.CHART_NAME }} v${{ steps.version.outputs.version }}]"
         uses: bsord/[email protected]
         with:
           useOCIRegistry: true
           registry-url: oci://ghcr.io/fuellabs/helmcharts
           username: ${{ github.repository_owner }}
           access-token: ${{ secrets.GITHUB_TOKEN }}
           force: true
-          chart-folder: ./cluster/charts/fuel-streams
+          chart-folder: ${{ env.CHART_PATH }}
 
       - name: Build Summary
         run: |-
           echo "### Helm Charts Build Summary 📊" >> $GITHUB_STEP_SUMMARY
           echo "| Chart | Version | Status |" >> $GITHUB_STEP_SUMMARY
           echo "|-------|---------|--------|" >> $GITHUB_STEP_SUMMARY
-
-          for chart in ${{ steps.charts.outputs.charts }}; do
-            version=""
-            if [ "$chart" = "fuel-streams-publisher" ]; then
-              version="${{ steps.versions.outputs.publisher_version }}"
-            elif [ "$chart" = "fuel-streams" ]; then
-              version="${{ steps.versions.outputs.streams_version }}"
-            fi
-            echo "| $chart | $version | ✅ Published |" >> $GITHUB_STEP_SUMMARY
-          done
+          echo "| ${{ env.CHART_NAME }} | ${{ steps.version.outputs.version }} | ✅ Published |" >> $GITHUB_STEP_SUMMARY
Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		* @luizstacio @pedronauck @jurshsmith @0xterminator
		* @luizstacio @pedronauck @jurshsmith @0xterminator @cue8it