Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics fix attempt: deferred reporter #3

Draft
wants to merge 2 commits into
base: iceberg-poc
Choose a base branch
from
Draft

Metrics fix attempt: deferred reporter #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
43c78e2
Metrics fix attempt: deferred reporter
Sep 27, 2024
f6bfc60
fix for invalid directory for reporter
Sep 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,7 @@ services:
FLINK_HADOOP_VERSION: "2.8.3-10.0"
AWS_SDK_VERSION: 2.26.10
ICEBERG_VERSION: 1.6.1
NU_DEFERRED_REPORTER_VERSION: 1.17.0
restart: unless-stopped
networks:
- nussknacker-iceberg-net
Expand Down Expand Up @@ -307,6 +308,7 @@ services:
FLINK_HADOOP_VERSION: "2.8.3-10.0"
AWS_SDK_VERSION: 2.26.10
ICEBERG_VERSION: 1.6.1
NU_DEFERRED_REPORTER_VERSION: 1.17.0
restart: unless-stopped
networks:
- nussknacker-iceberg-net
Expand Down
4 changes: 4 additions & 0 deletions flink/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ ARG HADOOP_VERSION="HADOOP_VERSION_IS_UNDEFINED"
ARG FLINK_HADOOP_VERSION="FLINK_HADOOP_VERSION_IS_UNDEFINED"
ARG AWS_SDK_VERSION="AWS_SDK_VERSION_IS_UNDEFINED"
ARG ICEBERG_VERSION="ICEBERG_VERSION_IS_UNDEFINED"
ARG NU_DEFERRED_REPORTER_VERSION="NU_DEFERRED_REPORTER_VERSION_IS_UNDEFINED"

RUN curl -L "https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/${HADOOP_VERSION}/hadoop-common-${HADOOP_VERSION}.jar" \
-o ${FLINK_HOME}/lib/hadoop-common-${HADOOP_VERSION}.jar
Expand All @@ -35,6 +36,9 @@ RUN curl -L https://github.com/knaufk/flink-faker/releases/download/v${FAKER_VER
RUN curl -L https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-flink-runtime-${FLINK_VERSION_SHORT}/${ICEBERG_VERSION}/iceberg-flink-runtime-${FLINK_VERSION_SHORT}-${ICEBERG_VERSION}.jar \
-o ${FLINK_HOME}/lib/iceberg-flink-runtime-${FLINK_VERSION_SHORT}-${ICEBERG_VERSION}.jar

RUN curl -L https://repo1.maven.org/maven2/pl/touk/nussknacker/nussknacker-flink-metrics-deferred-reporter/${NU_DEFERRED_REPORTER_VERSION}/nussknacker-flink-metrics-deferred-reporter-${NU_DEFERRED_REPORTER_VERSION}.jar \
-o ${FLINK_HOME}/plugins/metrics-influx/nussknacker-flink-metrics-deferred-reporter-${NU_DEFERRED_REPORTER_VERSION}.jar

COPY flink-properties.yml /opt/flink/conf/

VOLUME /opt/flink/data
Expand Down
3 changes: 2 additions & 1 deletion flink/flink-properties.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ state.backend.rocksdb.memory.managed: false
state.backend.rocksdb.writebuffer.size: 256m

metrics.reporters: influxdb_reporter
metrics.reporter.influxdb_reporter.factory.class: org.apache.flink.metrics.influxdb.InfluxdbReporterFactory
metrics.reporter.influxdb_reporter.factory.class: pl.touk.nussknacker.engine.flink.metrics.MetricRemovalDeferredToNextReportMetricReporterFactory
metrics.reporter.influxdb_reporter.delegate.factory.class: org.apache.flink.metrics.influxdb.InfluxdbReporterFactory
metrics.reporter.influxdb_reporter.host: telegraf
metrics.reporter.influxdb_reporter.port: 8087
metrics.reporter.influxdb_reporter.db: nussknacker_metrics
Expand Down