From f895d4754d278ddf1ed93c88a1d97a27438db2e4 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Wed, 3 Apr 2024 17:02:27 -0700 Subject: [PATCH 01/24] Break 4192 into smaller PR - 1 of 3 --- .../en/docs/collector/internal-telemetry.md | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 content/en/docs/collector/internal-telemetry.md diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md new file mode 100644 index 000000000000..ede512294318 --- /dev/null +++ b/content/en/docs/collector/internal-telemetry.md @@ -0,0 +1,74 @@ +--- +title: Internal Telemetry +weight: 25 +cSpell:ignore: journalctl kube otecol pprof tracez zpages +--- + +The Collector offers you multiple ways to measure and monitor its health as well +as investigate issues. In this section, you'll learn how to enable internal +observability, what types of telemetry are available, and how best to use them +to monitor your Collector deployment. + +## Enabling observability internal to the Collector + +By default, the Collector exposes service telemetry in two ways currently: + +- internal metrics are exposed via a Prometheus interface which defaults to port + `8888` +- logs are emitted to stdout + +Traces are not exposed by default. There is an effort underway to +[change this](https://github.com/open-telemetry/opentelemetry-collector/issues/7532). +The work includes supporting configuration of the OpenTelemetry SDK used to +produce the Collector's internal telemetry. This feature is currently behind two +feature gates: + +```sh +--feature-gates=telemetry.useOtelWithSDKConfigurationForInternalTelemetry +``` + +The gate `useOtelWithSDKConfigurationForInternalTelemetry` enables the Collector +to parse configuration that aligns with the +[OpenTelemetry Configuration](../configuration/) schema. The support for this +schema is still experimental, but it does allow telemetry to be exported via +OTLP. + +The following configuration can be used in combination with the feature gates +aforementioned to emit internal metrics and traces from the Collector to an OTLP +backend: + +```yaml +service: + telemetry: + metrics: + readers: + - periodic: + interval: 5000 + exporter: + otlp: + protocol: grpc/protobuf + endpoint: https://backend:4317 + traces: + processors: + - batch: + exporter: + otlp: + protocol: grpc/protobuf + endpoint: https://backend2:4317 +``` + +See the configuration's +[example](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/kitchen-sink.yaml) +for additional configuration options. + +Note that this configuration does not support emitting logs as there is no +support for [logs] in OpenTelemetry Go SDK at this time. + + + +To see logs for the Collector: + +On a Linux systemd system, logs can be found using `journalctl`: +`journalctl | grep otelcol` + +or to find only errors: `journalctl | grep otelcol | grep Error` From 85224410cd70d73e7c7375b70f65c6794b7a4052 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Thu, 4 Apr 2024 11:07:45 -0700 Subject: [PATCH 02/24] Fix refcache issue --- static/refcache.json | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/static/refcache.json b/static/refcache.json index 293e8bbcc98b..6f0ced1f0644 100644 --- a/static/refcache.json +++ b/static/refcache.json @@ -3043,6 +3043,10 @@ "StatusCode": 200, "LastSeen": "2024-01-18T19:36:56.082576-05:00" }, + "https://github.com/open-telemetry/opentelemetry-collector/issues/7532": { + "StatusCode": 200, + "LastSeen": "2024-04-04T11:07:15.276911438-07:00" + }, "https://github.com/open-telemetry/opentelemetry-collector/pull/6140": { "StatusCode": 200, "LastSeen": "2024-01-30T05:18:24.402543-05:00" From e2cb980d0e9bf29f7e895b27e0a1537df5d4fa3d Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Thu, 4 Apr 2024 11:49:16 -0700 Subject: [PATCH 03/24] Make copyedits --- .../en/docs/collector/internal-telemetry.md | 35 +++++++++---------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index ede512294318..c72f85f3b610 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -4,20 +4,20 @@ weight: 25 cSpell:ignore: journalctl kube otecol pprof tracez zpages --- -The Collector offers you multiple ways to measure and monitor its health as well -as investigate issues. In this section, you'll learn how to enable internal -observability, what types of telemetry are available, and how best to use them -to monitor your Collector deployment. +The Collector offers you multiple ways to measure and monitor its health. In +this section, you'll learn how to enable internal observability, what types of +telemetry are available, and how best to use them to monitor your Collector +deployment. ## Enabling observability internal to the Collector -By default, the Collector exposes service telemetry in two ways currently: +By default, the Collector exposes service telemetry in two ways: -- internal metrics are exposed via a Prometheus interface which defaults to port - `8888` -- logs are emitted to stdout +- Internal metrics are exposed via a Prometheus interface which defaults to port + `8888`. +- Logs are emitted to stdout. -Traces are not exposed by default. There is an effort underway to +Traces are not exposed by default, but an effort is underway to [change this](https://github.com/open-telemetry/opentelemetry-collector/issues/7532). The work includes supporting configuration of the OpenTelemetry SDK used to produce the Collector's internal telemetry. This feature is currently behind two @@ -29,12 +29,11 @@ feature gates: The gate `useOtelWithSDKConfigurationForInternalTelemetry` enables the Collector to parse configuration that aligns with the -[OpenTelemetry Configuration](../configuration/) schema. The support for this -schema is still experimental, but it does allow telemetry to be exported via -OTLP. +[OpenTelemetry Configuration](../configuration/) schema. Support for this schema +is still experimental, but it does allow telemetry to be exported using OTLP. -The following configuration can be used in combination with the feature gates -aforementioned to emit internal metrics and traces from the Collector to an OTLP +The following configuration can be used in combination with the aforementioned +feature gates to emit internal metrics and traces from the Collector to an OTLP backend: ```yaml @@ -57,12 +56,12 @@ service: endpoint: https://backend2:4317 ``` -See the configuration's -[example](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/kitchen-sink.yaml) +See the +[example configuration](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/kitchen-sink.yaml) for additional configuration options. -Note that this configuration does not support emitting logs as there is no -support for [logs] in OpenTelemetry Go SDK at this time. +> Note that this configuration does not support emitting logs as there is no +> support for logs in OpenTelemetry Go SDK at this time. From e75aa3366938c1c9fa1daa1bf7e1595b8cb1b75c Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Thu, 4 Apr 2024 11:50:44 -0700 Subject: [PATCH 04/24] Add tabbed panes --- content/en/docs/collector/internal-telemetry.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index c72f85f3b610..3d384efcd55d 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -63,11 +63,18 @@ for additional configuration options. > Note that this configuration does not support emitting logs as there is no > support for logs in OpenTelemetry Go SDK at this time. - +You can see logs for the Collector on a Linux systemd system using `journalctl`: -To see logs for the Collector: +{{< tabpane text=true >}} {{% tab All logs %}} -On a Linux systemd system, logs can be found using `journalctl`: -`journalctl | grep otelcol` +```sh +journalctl | grep otelcol +``` + +{{% /tab %}} {{% tab Errors only %}} + +```sh +journalctl | grep otelcol | grep Error +``` -or to find only errors: `journalctl | grep otelcol | grep Error` + {{% /tab %}} {{< /tabpane >}} From c929e4ccf52f859a7bdaed2b79d482ab1bb29fa9 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Thu, 4 Apr 2024 11:52:24 -0700 Subject: [PATCH 05/24] Make markdown linter fix --- content/en/docs/collector/internal-telemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 3d384efcd55d..05da190b876b 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -77,4 +77,4 @@ journalctl | grep otelcol journalctl | grep otelcol | grep Error ``` - {{% /tab %}} {{< /tabpane >}} +{{% /tab %}} {{< /tabpane >}} From 43b2cd23960f8260aee864952a2e53e765fb9bc0 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Thu, 4 Apr 2024 11:57:31 -0700 Subject: [PATCH 06/24] Fix tab labels --- content/en/docs/collector/internal-telemetry.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 05da190b876b..d682108e252b 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -65,13 +65,13 @@ for additional configuration options. You can see logs for the Collector on a Linux systemd system using `journalctl`: -{{< tabpane text=true >}} {{% tab All logs %}} +{{< tabpane text=true >}} {{% tab "All logs" %}} ```sh journalctl | grep otelcol ``` -{{% /tab %}} {{% tab Errors only %}} +{{% /tab %}} {{% tab "Errors only" %}} ```sh journalctl | grep otelcol | grep Error From f7b40e7e1515fa36e7f9511afb081cee429d550f Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Fri, 5 Apr 2024 13:32:27 -0700 Subject: [PATCH 07/24] Modify intro so doc is MVP-ready --- content/en/docs/collector/internal-telemetry.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index d682108e252b..0fb8fe8e71b7 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -5,9 +5,7 @@ cSpell:ignore: journalctl kube otecol pprof tracez zpages --- The Collector offers you multiple ways to measure and monitor its health. In -this section, you'll learn how to enable internal observability, what types of -telemetry are available, and how best to use them to monitor your Collector -deployment. +this section, you'll learn how to enable internal observability. ## Enabling observability internal to the Collector From 5b753d8d022b4678bd9db6754cb18188c35da470 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Fri, 5 Apr 2024 15:04:56 -0700 Subject: [PATCH 08/24] Add content to metrics and logs sections --- .../en/docs/collector/internal-telemetry.md | 131 +++++++++++++++--- 1 file changed, 108 insertions(+), 23 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 0fb8fe8e71b7..1f3cd287e4eb 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -4,18 +4,118 @@ weight: 25 cSpell:ignore: journalctl kube otecol pprof tracez zpages --- -The Collector offers you multiple ways to measure and monitor its health. In +The Collector offers multiple ways to measure and monitor its own health. In this section, you'll learn how to enable internal observability. ## Enabling observability internal to the Collector By default, the Collector exposes service telemetry in two ways: -- Internal metrics are exposed via a Prometheus interface which defaults to port - `8888`. -- Logs are emitted to stdout. +- Internal [metrics](#configuring-metrics) are exposed using a Prometheus + interface which defaults to port `8888`. +- [Logs](#configuring-logs) are emitted to `stdout`. -Traces are not exposed by default, but an effort is underway to +[Traces](#configuring-traces) are not exposed by default but two feature gates +offer experimental support for a configuration based on the OpenTelemetry +Configuration schema. + +### Configuring metrics + +Prometheus metrics are exposed locally on port `8888` and path `/metrics`. For +containerized environments, you may want to expose this port on a public +interface instead of only locally. + +Set the address in the config `service::telemetry::metrics`: + +```yaml +service: + telemetry: + metrics: + address: ':8888' +``` + +To visualize these metrics, you can use the +[Grafana dashboard](https://grafana.com/grafana/dashboards/15983-opentelemetry-collector/), +for example. + +You can enhance the metrics telemetry level using the `level` field. The +following is a list of all possible values and their explanations. + +- `none` indicates that no telemetry data should be collected. +- `basic` is the recommended value and covers the basics of the service + telemetry. +- `normal` adds other indicators on top of basic. +- `detailed` adds dimensions and views to the previous levels. + +For example: + +```yaml +service: + telemetry: + metrics: + level: detailed + address: ':8888' +``` + +The Collector can also be configured to scrape its own metrics and send them +through configured pipelines. For example: + +```yaml +receivers: + prometheus: + config: + scrape_configs: + - job_name: 'otelcol' + scrape_interval: 10s + static_configs: + - targets: ['0.0.0.0:8888'] + metric_relabel_configs: + - source_labels: [__name__] + regex: '.*grpc_io.*' + action: drop +exporters: + debug: +service: + pipelines: + metrics: + receivers: [prometheus] + processors: [] + exporters: [debug] +``` + +### Configuring logs + +You can find log output in `stdout`. The verbosity level for logs defaults to +`INFO`, but you can adjust it in the config `service::telemetry::logs`: + +```yaml +service: + telemetry: + logs: + level: 'debug' +``` + +You can also see logs for the Collector on a Linux systemd system using +`journalctl`: + +{{< tabpane text=true >}} {{% tab "All logs" %}} + +```sh +journalctl | grep otelcol +``` + +{{% /tab %}} {{% tab "Errors only" %}} + +```sh +journalctl | grep otelcol | grep Error +``` + +{{% /tab %}} {{< /tabpane >}} + +### Configuring traces + +Although the Collector does not expose traces by default, an effort is underway +to [change this](https://github.com/open-telemetry/opentelemetry-collector/issues/7532). The work includes supporting configuration of the OpenTelemetry SDK used to produce the Collector's internal telemetry. This feature is currently behind two @@ -27,8 +127,9 @@ feature gates: The gate `useOtelWithSDKConfigurationForInternalTelemetry` enables the Collector to parse configuration that aligns with the -[OpenTelemetry Configuration](../configuration/) schema. Support for this schema -is still experimental, but it does allow telemetry to be exported using OTLP. +[OpenTelemetry Configuration](https://github.com/open-telemetry/opentelemetry-configuration) +schema. Support for this schema is still experimental, but it does allow +telemetry to be exported using OTLP. The following configuration can be used in combination with the aforementioned feature gates to emit internal metrics and traces from the Collector to an OTLP @@ -60,19 +161,3 @@ for additional configuration options. > Note that this configuration does not support emitting logs as there is no > support for logs in OpenTelemetry Go SDK at this time. - -You can see logs for the Collector on a Linux systemd system using `journalctl`: - -{{< tabpane text=true >}} {{% tab "All logs" %}} - -```sh -journalctl | grep otelcol -``` - -{{% /tab %}} {{% tab "Errors only" %}} - -```sh -journalctl | grep otelcol | grep Error -``` - -{{% /tab %}} {{< /tabpane >}} From d0e4a0c34e6266eae21737b89df1a1bfd67c9613 Mon Sep 17 00:00:00 2001 From: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com> Date: Fri, 5 Apr 2024 22:11:49 +0000 Subject: [PATCH 09/24] Results from /fix:refcache --- static/refcache.json | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/static/refcache.json b/static/refcache.json index f93e5d692e89..2b945af723bc 100644 --- a/static/refcache.json +++ b/static/refcache.json @@ -4467,6 +4467,10 @@ "StatusCode": 200, "LastSeen": "2024-01-24T14:54:56.149229+01:00" }, + "https://grafana.com/grafana/dashboards/15983-opentelemetry-collector/": { + "StatusCode": 200, + "LastSeen": "2024-04-05T22:11:27.002931782Z" + }, "https://grafana.com/oss/opentelemetry/": { "StatusCode": 200, "LastSeen": "2024-01-18T08:52:48.999991-05:00" From 3f334434d4ea955c46938cd4fdabdaff0cfe9996 Mon Sep 17 00:00:00 2001 From: Tiffany Hrabusa <30397949+tiffany76@users.noreply.github.com> Date: Mon, 8 Apr 2024 11:30:03 -0700 Subject: [PATCH 10/24] Apply suggestions from review Co-authored-by: Fabrizio Ferri-Benedetti Co-authored-by: Pablo Baeyens --- content/en/docs/collector/internal-telemetry.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 1f3cd287e4eb..cb9fc001d22e 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -1,11 +1,12 @@ --- -title: Internal Telemetry +title: Internal telemetry weight: 25 cSpell:ignore: journalctl kube otecol pprof tracez zpages --- -The Collector offers multiple ways to measure and monitor its own health. In -this section, you'll learn how to enable internal observability. +You can check the health of any Collector instance by checking its own +internal telemetry. Read on to learn how to use this data to diagnose and +troubleshoot Collector issues. ## Enabling observability internal to the Collector @@ -19,10 +20,10 @@ By default, the Collector exposes service telemetry in two ways: offer experimental support for a configuration based on the OpenTelemetry Configuration schema. -### Configuring metrics +### Configure internal metrics Prometheus metrics are exposed locally on port `8888` and path `/metrics`. For -containerized environments, you may want to expose this port on a public +containerized environments, you might want to expose this port on a public interface instead of only locally. Set the address in the config `service::telemetry::metrics`: @@ -79,11 +80,10 @@ service: pipelines: metrics: receivers: [prometheus] - processors: [] exporters: [debug] ``` -### Configuring logs +### Configure internal logs You can find log output in `stdout`. The verbosity level for logs defaults to `INFO`, but you can adjust it in the config `service::telemetry::logs`: From 45c14f0bee28111f3a3963d6f74264f444399996 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Mon, 8 Apr 2024 11:38:30 -0700 Subject: [PATCH 11/24] Make prettier fixes --- content/en/docs/collector/internal-telemetry.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index cb9fc001d22e..74eab4352ebe 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -4,9 +4,9 @@ weight: 25 cSpell:ignore: journalctl kube otecol pprof tracez zpages --- -You can check the health of any Collector instance by checking its own -internal telemetry. Read on to learn how to use this data to diagnose and -troubleshoot Collector issues. +You can check the health of any Collector instance by checking its own internal +telemetry. Read on to learn how to use this data to diagnose and troubleshoot +Collector issues. ## Enabling observability internal to the Collector @@ -112,7 +112,7 @@ journalctl | grep otelcol | grep Error {{% /tab %}} {{< /tabpane >}} -### Configuring traces +### Configure internal traces Although the Collector does not expose traces by default, an effort is underway to From ca60916b46979766c11591021e92c32f5c317ee1 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Mon, 8 Apr 2024 11:42:44 -0700 Subject: [PATCH 12/24] Fix links to headings --- content/en/docs/collector/internal-telemetry.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 74eab4352ebe..0267ea46dc6b 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -12,11 +12,11 @@ Collector issues. By default, the Collector exposes service telemetry in two ways: -- Internal [metrics](#configuring-metrics) are exposed using a Prometheus +- Internal [metrics](#configure-internal-metrics) are exposed using a Prometheus interface which defaults to port `8888`. -- [Logs](#configuring-logs) are emitted to `stdout`. +- [Logs](#configure-internal-logs) are emitted to `stdout`. -[Traces](#configuring-traces) are not exposed by default but two feature gates +[Traces](#configure-internal-traces) are not exposed by default but two feature gates offer experimental support for a configuration based on the OpenTelemetry Configuration schema. From 28d15f7dc42b5f2e71f89c15543e5796f81991e9 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Mon, 8 Apr 2024 11:47:24 -0700 Subject: [PATCH 13/24] Make requested changes to first subsection --- content/en/docs/collector/internal-telemetry.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 0267ea46dc6b..cb60dc6858ac 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -8,16 +8,16 @@ You can check the health of any Collector instance by checking its own internal telemetry. Read on to learn how to use this data to diagnose and troubleshoot Collector issues. -## Enabling observability internal to the Collector +## Activate internal telemetry in the Collector -By default, the Collector exposes service telemetry in two ways: +By default, the Collector exposes its own telemetry in two ways: - Internal [metrics](#configure-internal-metrics) are exposed using a Prometheus interface which defaults to port `8888`. -- [Logs](#configure-internal-logs) are emitted to `stdout`. +- [Logs](#configure-internal-logs) are emitted to `stdout` by default. -[Traces](#configure-internal-traces) are not exposed by default but two feature gates -offer experimental support for a configuration based on the OpenTelemetry +[Traces](#configure-internal-traces) are not exposed by default. Two feature +gates offer experimental support for a configuration based on the OpenTelemetry Configuration schema. ### Configure internal metrics From 5ba87863dfa997df525f785f5521a47704fce16a Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Wed, 10 Apr 2024 15:09:09 -0700 Subject: [PATCH 14/24] Clarify the use of local host for metrics --- content/en/docs/collector/internal-telemetry.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index cb60dc6858ac..2922297017f3 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -22,9 +22,12 @@ Configuration schema. ### Configure internal metrics -Prometheus metrics are exposed locally on port `8888` and path `/metrics`. For +You can configure how internal metrics are generated and exposed by the +Collector. By default, the Collector generates basic metrics about itself and +exposes them for scraping at http://127.0.0.1:8888/metrics. You can expose the +endpoint to one specific or all network interfaces when needed. For containerized environments, you might want to expose this port on a public -interface instead of only locally. +interface. Set the address in the config `service::telemetry::metrics`: @@ -32,7 +35,7 @@ Set the address in the config `service::telemetry::metrics`: service: telemetry: metrics: - address: ':8888' + address: '0.0.0.0:8888' ``` To visualize these metrics, you can use the From 007fce4fbed0b89c1c89259960d1873331ed4571 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Wed, 10 Apr 2024 15:11:57 -0700 Subject: [PATCH 15/24] Fix refcache --- static/refcache.json | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/static/refcache.json b/static/refcache.json index 4d31ec630543..74f0daf2db13 100644 --- a/static/refcache.json +++ b/static/refcache.json @@ -4491,6 +4491,10 @@ "StatusCode": 200, "LastSeen": "2024-04-10T00:09:55.400731+02:00" }, + "https://grafana.com/grafana/dashboards/15983-opentelemetry-collector/": { + "StatusCode": 200, + "LastSeen": "2024-04-10T15:11:30.311778613-07:00" + }, "https://grafana.com/oss/opentelemetry/": { "StatusCode": 200, "LastSeen": "2024-01-18T08:52:48.999991-05:00" From 625af6f18d342df25c5418c89e05e745cb565c68 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Thu, 11 Apr 2024 11:37:37 -0700 Subject: [PATCH 16/24] Add cautionary note to self-monitoring section --- content/en/docs/collector/internal-telemetry.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 2922297017f3..7a56071409cf 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -86,6 +86,14 @@ service: exporters: [debug] ``` +{{% alert title="Caution" color="warning" %}} + + +Self-monitoring is a risky practice. If an issue arises, the source of the problem is unclear and the telemetry is unreliable. + + +{{% /alert %}} + ### Configure internal logs You can find log output in `stdout`. The verbosity level for logs defaults to From 0f3ab4121cf492663b610e4eed6ea88567b5f1f9 Mon Sep 17 00:00:00 2001 From: Tiffany Hrabusa <30397949+tiffany76@users.noreply.github.com> Date: Thu, 11 Apr 2024 11:42:34 -0700 Subject: [PATCH 17/24] Update format of logs note Co-authored-by: Severin Neumann --- content/en/docs/collector/internal-telemetry.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 7a56071409cf..4890612128aa 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -170,5 +170,9 @@ See the [example configuration](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/kitchen-sink.yaml) for additional configuration options. -> Note that this configuration does not support emitting logs as there is no -> support for logs in OpenTelemetry Go SDK at this time. +{{% alert title="Note" %}} + +Note that this configuration does not support emitting logs as there is no +support for logs in OpenTelemetry Go SDK at this time. + +{{% /alert %}} From 97035fceb548e19eb0e80ab8e40cf204e601e446 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Thu, 11 Apr 2024 12:04:30 -0700 Subject: [PATCH 18/24] Make linter fixes --- content/en/docs/collector/internal-telemetry.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 4890612128aa..b2f63dd241a3 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -24,7 +24,7 @@ Configuration schema. You can configure how internal metrics are generated and exposed by the Collector. By default, the Collector generates basic metrics about itself and -exposes them for scraping at http://127.0.0.1:8888/metrics. You can expose the +exposes them for scraping at `http://127.0.0.1:8888/metrics`. You can expose the endpoint to one specific or all network interfaces when needed. For containerized environments, you might want to expose this port on a public interface. @@ -88,9 +88,8 @@ service: {{% alert title="Caution" color="warning" %}} - -Self-monitoring is a risky practice. If an issue arises, the source of the problem is unclear and the telemetry is unreliable. - +Self-monitoring is a risky practice. If an issue arises, the source of the +problem is unclear and the telemetry is unreliable. {{% /alert %}} @@ -172,7 +171,7 @@ for additional configuration options. {{% alert title="Note" %}} -Note that this configuration does not support emitting logs as there is no -support for logs in OpenTelemetry Go SDK at this time. +This configuration does not support emitting logs as there is no support for +logs in OpenTelemetry Go SDK at this time. {{% /alert %}} From 493a38909971cbfd947d8d479e93e9ef54b3dc05 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Thu, 11 Apr 2024 12:40:56 -0700 Subject: [PATCH 19/24] Add reciprocal links --- content/en/docs/collector/internal-telemetry.md | 6 +++--- content/en/docs/collector/troubleshooting.md | 6 ++++++ 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index b2f63dd241a3..d5f7dbaa5bc6 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -4,9 +4,9 @@ weight: 25 cSpell:ignore: journalctl kube otecol pprof tracez zpages --- -You can check the health of any Collector instance by checking its own internal -telemetry. Read on to learn how to use this data to diagnose and troubleshoot -Collector issues. +You can monitor the health of any Collector instance by checking its own +internal telemetry. Read on to learn how to configure this telemetry to help you +[troubleshoot](/docs/collector/troubleshooting/) Collector issues. ## Activate internal telemetry in the Collector diff --git a/content/en/docs/collector/troubleshooting.md b/content/en/docs/collector/troubleshooting.md index d83b7cdb821e..8278d00b678b 100644 --- a/content/en/docs/collector/troubleshooting.md +++ b/content/en/docs/collector/troubleshooting.md @@ -8,6 +8,12 @@ This page describes some options when troubleshooting the health or performance of the OpenTelemetry Collector. The Collector provides a variety of metrics, logs, and extensions for debugging issues. +## Internal telemetry + +You can configure and use the Collector's own +[internal telemetry](/docs/collector/internal-telemetry/) to monitor its +performance. + ## Sending test data For certain types of issues, particularly verifying configuration and debugging From c2fef88b7d3fd6e08c8de3707107f47a4b1a9ad7 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Fri, 12 Apr 2024 11:37:14 -0700 Subject: [PATCH 20/24] Remove traces --- .../en/docs/collector/internal-telemetry.md | 58 ------------------- 1 file changed, 58 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index d5f7dbaa5bc6..083f3c5a204c 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -16,10 +16,6 @@ By default, the Collector exposes its own telemetry in two ways: interface which defaults to port `8888`. - [Logs](#configure-internal-logs) are emitted to `stdout` by default. -[Traces](#configure-internal-traces) are not exposed by default. Two feature -gates offer experimental support for a configuration based on the OpenTelemetry -Configuration schema. - ### Configure internal metrics You can configure how internal metrics are generated and exposed by the @@ -121,57 +117,3 @@ journalctl | grep otelcol | grep Error ``` {{% /tab %}} {{< /tabpane >}} - -### Configure internal traces - -Although the Collector does not expose traces by default, an effort is underway -to -[change this](https://github.com/open-telemetry/opentelemetry-collector/issues/7532). -The work includes supporting configuration of the OpenTelemetry SDK used to -produce the Collector's internal telemetry. This feature is currently behind two -feature gates: - -```sh ---feature-gates=telemetry.useOtelWithSDKConfigurationForInternalTelemetry -``` - -The gate `useOtelWithSDKConfigurationForInternalTelemetry` enables the Collector -to parse configuration that aligns with the -[OpenTelemetry Configuration](https://github.com/open-telemetry/opentelemetry-configuration) -schema. Support for this schema is still experimental, but it does allow -telemetry to be exported using OTLP. - -The following configuration can be used in combination with the aforementioned -feature gates to emit internal metrics and traces from the Collector to an OTLP -backend: - -```yaml -service: - telemetry: - metrics: - readers: - - periodic: - interval: 5000 - exporter: - otlp: - protocol: grpc/protobuf - endpoint: https://backend:4317 - traces: - processors: - - batch: - exporter: - otlp: - protocol: grpc/protobuf - endpoint: https://backend2:4317 -``` - -See the -[example configuration](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/kitchen-sink.yaml) -for additional configuration options. - -{{% alert title="Note" %}} - -This configuration does not support emitting logs as there is no support for -logs in OpenTelemetry Go SDK at this time. - -{{% /alert %}} From eda8108e8c5577fea6195f903591b5e409536eb0 Mon Sep 17 00:00:00 2001 From: Tiffany Hrabusa <30397949+tiffany76@users.noreply.github.com> Date: Tue, 16 Apr 2024 11:51:26 -0700 Subject: [PATCH 21/24] Add review suggestion Co-authored-by: Severin Neumann --- content/en/docs/collector/internal-telemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 083f3c5a204c..45dcbae03546 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -4,7 +4,7 @@ weight: 25 cSpell:ignore: journalctl kube otecol pprof tracez zpages --- -You can monitor the health of any Collector instance by checking its own +You can monitor the health of any OpenTelemetry Collector instance by checking its own internal telemetry. Read on to learn how to configure this telemetry to help you [troubleshoot](/docs/collector/troubleshooting/) Collector issues. From 9f2d1056afe79337d3d277e8dcf4e484913f33ac Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Tue, 16 Apr 2024 11:56:02 -0700 Subject: [PATCH 22/24] Remove Grafana dashboard --- content/en/docs/collector/internal-telemetry.md | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 45dcbae03546..33d1212168ab 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -4,9 +4,9 @@ weight: 25 cSpell:ignore: journalctl kube otecol pprof tracez zpages --- -You can monitor the health of any OpenTelemetry Collector instance by checking its own -internal telemetry. Read on to learn how to configure this telemetry to help you -[troubleshoot](/docs/collector/troubleshooting/) Collector issues. +You can monitor the health of any OpenTelemetry Collector instance by checking +its own internal telemetry. Read on to learn how to configure this telemetry to +help you [troubleshoot](/docs/collector/troubleshooting/) Collector issues. ## Activate internal telemetry in the Collector @@ -34,10 +34,6 @@ service: address: '0.0.0.0:8888' ``` -To visualize these metrics, you can use the -[Grafana dashboard](https://grafana.com/grafana/dashboards/15983-opentelemetry-collector/), -for example. - You can enhance the metrics telemetry level using the `level` field. The following is a list of all possible values and their explanations. From c42680f1f9389aa9aaadc1daa19f38fd51bcb4ee Mon Sep 17 00:00:00 2001 From: Tiffany Hrabusa <30397949+tiffany76@users.noreply.github.com> Date: Tue, 16 Apr 2024 13:13:20 -0700 Subject: [PATCH 23/24] Fix default log emitting Co-authored-by: Pablo Baeyens --- content/en/docs/collector/internal-telemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index 33d1212168ab..a24831de67c1 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -14,7 +14,7 @@ By default, the Collector exposes its own telemetry in two ways: - Internal [metrics](#configure-internal-metrics) are exposed using a Prometheus interface which defaults to port `8888`. -- [Logs](#configure-internal-logs) are emitted to `stdout` by default. +- [Logs](#configure-internal-logs) are emitted to `stderr` by default. ### Configure internal metrics From a9af6e321b8ddc13507027b73da9c43f75b924b3 Mon Sep 17 00:00:00 2001 From: tiffany76 <30397949+tiffany76@users.noreply.github.com> Date: Tue, 16 Apr 2024 13:16:44 -0700 Subject: [PATCH 24/24] Fix other stdout --- content/en/docs/collector/internal-telemetry.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/collector/internal-telemetry.md b/content/en/docs/collector/internal-telemetry.md index a24831de67c1..9405568798a1 100644 --- a/content/en/docs/collector/internal-telemetry.md +++ b/content/en/docs/collector/internal-telemetry.md @@ -87,7 +87,7 @@ problem is unclear and the telemetry is unreliable. ### Configure internal logs -You can find log output in `stdout`. The verbosity level for logs defaults to +You can find log output in `stderr`. The verbosity level for logs defaults to `INFO`, but you can adjust it in the config `service::telemetry::logs`: ```yaml