Unify internal observability documentation - 1 of 3

content/en/docs/collector/internal-telemetry.md

@@ -0,0 +1,163 @@
+---
+title: Internal Telemetry
+weight: 25
+cSpell:ignore: journalctl kube otecol pprof tracez zpages
+---
+
+The Collector offers multiple ways to measure and monitor its own health. In
+this section, you'll learn how to enable internal observability.
+
+## Enabling observability internal to the Collector
+
+By default, the Collector exposes service telemetry in two ways:
+
+- Internal [metrics](#configuring-metrics) are exposed using a Prometheus
+  interface which defaults to port `8888`.
+- [Logs](#configuring-logs) are emitted to `stdout`.
+
+[Traces](#configuring-traces) are not exposed by default but two feature gates
+offer experimental support for a configuration based on the OpenTelemetry
+Configuration schema.
+
+### Configuring metrics
+
+Prometheus metrics are exposed locally on port `8888` and path `/metrics`. For
+containerized environments, you may want to expose this port on a public
+interface instead of only locally.
+
+Set the address in the config `service::telemetry::metrics`:
+
+```yaml
+service:
+  telemetry:
+    metrics:
+      address: ':8888'
+```
+
+To visualize these metrics, you can use the
+[Grafana dashboard](https://grafana.com/grafana/dashboards/15983-opentelemetry-collector/),
+for example.
+
+You can enhance the metrics telemetry level using the `level` field. The
+following is a list of all possible values and their explanations.
+
+- `none` indicates that no telemetry data should be collected.
+- `basic` is the recommended value and covers the basics of the service
+  telemetry.
+- `normal` adds other indicators on top of basic.
+- `detailed` adds dimensions and views to the previous levels.
+
+For example:
+
+```yaml
+service:
+  telemetry:
+    metrics:
+      level: detailed
+      address: ':8888'
+```
+
+The Collector can also be configured to scrape its own metrics and send them
+through configured pipelines. For example:
+
+```yaml
+receivers:
+  prometheus:
+    config:
+      scrape_configs:
+        - job_name: 'otelcol'
+          scrape_interval: 10s
+          static_configs:
+            - targets: ['0.0.0.0:8888']
+          metric_relabel_configs:
+            - source_labels: [__name__]
+              regex: '.*grpc_io.*'
+              action: drop
+exporters:
+  debug:
+service:
+  pipelines:
+    metrics:
+      receivers: [prometheus]
+      processors: []
+      exporters: [debug]
+```
+
+### Configuring logs
+
+You can find log output in `stdout`. The verbosity level for logs defaults to
+`INFO`, but you can adjust it in the config `service::telemetry::logs`:
+
+```yaml
+service:
+  telemetry:
+    logs:
+      level: 'debug'
+```
+
+You can also see logs for the Collector on a Linux systemd system using
+`journalctl`:
+
+{{< tabpane text=true >}} {{% tab "All logs" %}}
+
+```sh
+journalctl | grep otelcol
+```
+
+{{% /tab %}} {{% tab "Errors only" %}}
+
+```sh
+journalctl | grep otelcol | grep Error
+```
+
+{{% /tab %}} {{< /tabpane >}}
+
+### Configuring traces
+
+Although the Collector does not expose traces by default, an effort is underway
+to
+[change this](https://github.com/open-telemetry/opentelemetry-collector/issues/7532).
+The work includes supporting configuration of the OpenTelemetry SDK used to
+produce the Collector's internal telemetry. This feature is currently behind two
+feature gates:
+
+```sh
+--feature-gates=telemetry.useOtelWithSDKConfigurationForInternalTelemetry
+```
+
+The gate `useOtelWithSDKConfigurationForInternalTelemetry` enables the Collector
+to parse configuration that aligns with the
+[OpenTelemetry Configuration](https://github.com/open-telemetry/opentelemetry-configuration)
+schema. Support for this schema is still experimental, but it does allow
+telemetry to be exported using OTLP.
+
+The following configuration can be used in combination with the aforementioned
+feature gates to emit internal metrics and traces from the Collector to an OTLP
+backend:
+
+```yaml
+service:
+  telemetry:
+    metrics:
+      readers:
+        - periodic:
+            interval: 5000
+            exporter:
+              otlp:
+                protocol: grpc/protobuf
+                endpoint: https://backend:4317
+    traces:
+      processors:
+        - batch:
+            exporter:
+              otlp:
+                protocol: grpc/protobuf
+                endpoint: https://backend2:4317
+```
+
+See the
+[example configuration](https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/kitchen-sink.yaml)
+for additional configuration options.
+
+> Note that this configuration does not support emitting logs as there is no
+> support for logs in OpenTelemetry Go SDK at this time.

static/refcache.json

@@ -3047,6 +3047,10 @@
     "StatusCode": 200,
     "LastSeen": "2024-01-18T19:36:56.082576-05:00"
   },
+  "https://github.com/open-telemetry/opentelemetry-collector/issues/7532": {
+    "StatusCode": 200,
+    "LastSeen": "2024-04-04T11:07:15.276911438-07:00"
+  },
   "https://github.com/open-telemetry/opentelemetry-collector/pull/6140": {
     "StatusCode": 200,
     "LastSeen": "2024-01-30T05:18:24.402543-05:00"