Add an Ansible role for OpenTelemetry Collector (#138)

Co-authored-by: v-zhuravlev <[email protected]>
grafana · Mar 12, 2024 · 962ff9a · 962ff9a
1 parent b66a7ff
commit 962ff9a
Show file tree

Hide file tree

Showing 17 changed files with 536 additions and 3 deletions.
diff --git a/.gitignore b/.gitignore
@@ -5,6 +5,7 @@
 .idea
 hosts
 *.log
+yala
 
 # MacOS
 # ---------------------------------------------------

diff --git a/CODEOWNERS b/CODEOWNERS
@@ -1,3 +1,3 @@
 ./                           @ishanjainn
 /roles/grafana               @gardar @ishanjainn
-/roles/grafana_agent         @ishanjainn @v-zhuravlev
+/roles/grafana_agent         @ishanjainn @v-zhuravlev @gardar
diff --git a/README.md b/README.md
@@ -11,7 +11,7 @@
 [![Full Integration Test](https://github.com/grafana/grafana-ansible-collection/actions/workflows/full-integration-test.yml/badge.svg?branch=main)](https://github.com/grafana/grafana-ansible-collection/actions/workflows/full-integration-test.yml)
 [![Lint](https://github.com/grafana/grafana-ansible-collection/actions/workflows/lint.yaml/badge.svg)](https://github.com/grafana/grafana-ansible-collection/actions/workflows/lint.yaml)
 
-This collection (`grafana.grafana`) contains modules and plugins to assist in automating managing of resources in **Grafana** with Ansible.
+This collection (`grafana.grafana`) contains modules and roles to assist in automating the management of resources in **Grafana**, **Grafana Agent**, and **OpenTelemetry Collector** with Ansible.
 
 -   [Ansible collection Documentation](https://docs.ansible.com/ansible/latest/collections/grafana/grafana/)
 -   [Grafana](https://grafana.com)
@@ -46,6 +46,14 @@ collections:
     version: 1.0.0
 ```
 
+## Roles included in the collection
+
+This collection includes the following roles to help set up and manage Grafana, Grafana Agent, and OpenTelemetry Collector:
+
+- **Grafana**: Installs and configures Grafana on your target hosts.
+- **Grafana Agent**: Deploys and configures Grafana Agent, allowing for efficient metrics, logs, and trace data shipping to Grafana Cloud or other endpoints.
+- **OpenTelemetry Collector**: Sets up and configures the OpenTelemetry Collector, enabling advanced observability features through data collection and transmission.
+
 ## Using this collection
 
 You can call modules by their Fully Qualified Collection Namespace (FQCN), such as `grafana.grafana.cloud_stack`:

diff --git a/examples/monitor-multiple-instance-otel.md b/examples/monitor-multiple-instance-otel.md
@@ -0,0 +1,193 @@
+# Scaling OpenTelemetry Collector Deployments Using Grafana Ansible Collection
+This guide is focused on scaling the OpenTelemetry Collector deployment across various Linux hosts by leveraging Ansible, to function both as gateways and agents within your observability architecture. Utilizing the OpenTelemetry Collector in this dual capacity enables a robust collection and forwarding of metrics, traces, and logs to analysis and visualization platforms, such as Grafana Cloud. 
+
+Here, we outline a strategy for deploying and managing the OpenTelemetry Collector's scalable instances throughout your infrastructure with Ansible, enhancing your overall monitoring strategy and data visualization capabilities in Grafana Cloud.
+
+## Before You Begin
+
+To follow this guide, ensure you have:
+
+- Linux hosts.
+- SSH access to each of these Linux hosts.
+- Account permissions to install and configure the OpenTelemetry Collector on these hosts.
+
+## Install the Grafana Ansible collection
+
+The [Grafana Agent role](https://github.com/grafana/grafana-ansible-collection/tree/main/roles/grafana_agent) is available in the Grafana Ansible collection as of the 1.1.0 release.
+
+To install the Grafana Ansible collection, run this command:
+
+```
+ansible-galaxy collection install grafana.grafana
+```
+
+## Create an Ansible inventory file
+
+Next, you will set up your hosts and create an inventory file.
+
+1. Create your hosts and add public SSH keys to them.
+
+  This example uses eight Linux hosts: two Ubuntu hosts, two CentOS hosts, two Fedora hosts, and two Debian hosts.
+
+1. Create an Ansible inventory file.
+
+  The Ansible inventory, which resides in a file named `inventory`, looks similar to this:
+
+  ```
+  146.190.208.216    # hostname = ubuntu-01
+  146.190.208.190    # hostname = ubuntu-02
+  137.184.155.128    # hostname = centos-01
+  146.190.216.129    # hostname = centos-02
+  198.199.82.174     # hostname = debian-01
+  198.199.77.93      # hostname = debian-02
+  143.198.182.156    # hostname = fedora-01
+  143.244.174.246    # hostname = fedora-02
+  ```
+
+  > **Note**: If you are copying the above file, remove the comments (#).
+
+1. Create an `ansible.cfg` file within the same directory as `inventory`, with the following values:
+  ```
+  [defaults]
+  inventory = inventory  # Path to the inventory file
+  private_key_file = ~/.ssh/id_rsa   # Path to my private SSH Key
+  remote_user=root
+  ```
+
+## Use the OpenTelemetry Collector Ansible Role
+
+Next, you'll define an Ansible playbook to apply your chosen or created OpenTelemetry Collector role across your hosts.
+
+Create a file named `deploy-opentelemetry.yml` in the same directory as your `ansible.cfg` and `inventory`. 
+
+```yaml
+- name: Install OpenTelemetry Collector
+  hosts: all
+  become: true
+
+  vars:
+    grafana_cloud_api_key: <Your Grafana.com API Key>        # Example - eyJrIjoiYjI3NjI5MGQxZTcyOTIxYTc0MDgzMGVhNDhlODNhYzA5OTk2Y2U5YiIsIm4iOiJhbnNpYmxldGVzdCIsImlkIjo2NTI5
+    metrics_username: <prometheus-username>                  # Example - 825019
+    logs_username: <loki-username>                           # Example - 411478
+    prometheus_url: <prometheus-push-url>                    # Example - https://prometheus-us-central1.grafana.net/api/prom/push
+    loki_url: <loki-push-url>                                # Example - https://logs-prod-017.grafana.net/loki/api/v1/push
+    tempo_url: <tempo-push-url>                              # Example - tempo-prod-04-prod-us-east-0.grafana.net:443
+    traces_username: <tempo-username>                         # Example - 411478
+
+  tasks:
+    - name: Install OpenTelemetry Collector
+      ansible.builtin.include_role:
+        name: grafana.grafana.opentelemetry_collector
+      vars:
+        otel_collector_extensions:
+          basicauth/grafana_cloud_tempo:
+            # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/basicauthextension
+            client_auth:
+              username: "{{ traces_username }}"
+              password: "{{ grafana_cloud_api_key }}"
+          basicauth/grafana_cloud_prometheus:
+            client_auth:
+              username: "{{ prometheus_url }}"
+              password: "{{ grafana_cloud_api_key }}"
+          basicauth/grafana_cloud_loki:
+            client_auth:
+              username: "{{ logs_username }}"
+              password: "{{ grafana_cloud_api_key }}"
+
+
+        otel_collector_receivers:
+          otlp:
+            # https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver
+            protocols:
+              grpc:
+              http:
+          hostmetrics:
+            # Optional. Host Metrics Receiver added as an example of Infra Monitoring capabilities of the OpenTelemetry Collector
+            # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver
+            scrapers:
+              load:
+              memory:
+
+        otel_collector_processors:
+          batch:
+            # https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor
+          resourcedetection:
+            # Enriches telemetry data with resource information from the host
+            # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor
+            detectors: ["env", "system"]
+            override: false
+          transform/add_resource_attributes_as_metric_attributes:
+            # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor
+            error_mode: ignore
+            metric_statements:
+              - context: datapoint
+                statements:
+                  - set(attributes["deployment.environment"], resource.attributes["deployment.environment"])
+                  - set(attributes["service.version"], resource.attributes["service.version"])
+
+        otel_collector_exporters:
+          otlp/grafana_cloud_traces:
+            # https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter
+            endpoint: "{{ tempo_url }}"
+            auth:
+              authenticator: basicauth/grafana_cloud_tempo
+
+          loki/grafana_cloud_logs:
+            # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/lokiexporter
+            endpoint: "{{ loki_url }}"
+            auth:
+              authenticator: basicauth/grafana_cloud_loki
+
+          prometheusremotewrite/grafana_cloud_metrics:
+            # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusremotewriteexporter
+            endpoint: "{{ prometheus_url }}"
+            add_metric_suffixes: false
+            auth:
+              authenticator: basicauth/grafana_cloud_prometheus
+
+
+        otel_collector_service:
+          extensions: [basicauth/grafana_cloud_tempo, basicauth/grafana_cloud_prometheus, basicauth/grafana_cloud_loki]
+          pipelines:
+            traces:
+              receivers: [otlp]
+              processors: [resourcedetection, batch]
+              exporters: [otlp/grafana_cloud_traces]
+            metrics:
+              receivers: [otlp, hostmetrics]
+              processors: [resourcedetection, transform/add_resource_attributes_as_metric_attributes, batch]
+              exporters: [prometheusremotewrite/grafana_cloud_metrics]
+            logs:
+              receivers: [otlp]
+              processors: [resourcedetection, batch]
+              exporters: [loki/grafana_cloud_logs]
+```
+
+> **Note:** You'll need to adjust the configuration to match the specific telemetry data you intend to collect and where you plan to forward it. The configuration snippet above is a basic example designed for traces, logs and metrics collection via OTLP and forwarding to Grafana Cloud. 
+
+
+## Running the Ansible Playbook
+
+Deploy the OpenTelemetry Collector across your hosts by executing:
+
+```sh
+ansible-playbook deploy-opentelemetry.yml
+```
+
+## Verifying Data Ingestion into Grafana Cloud
+
+Once you've deployed the OpenTelemetry Collector and configured it to forward data to Grafana Cloud, you can verify the ingestion:
+
+- Log into your Grafana Cloud instance.
+- Navigate to the **Explore** section.
+- Select your Grafana Cloud Prometheus data source from the dropdown menu.
+- Execute a query to confirm the reception of metrics, e.g., `{instance="ubuntu-01"}` for a specific host's metrics.
+
+## Visualizing Metrics and Logs in Grafana
+
+With data successfully ingested into Grafana Cloud, you can create custom dashboards to visualize the metrics, logs and traces received from your OpenTelemetry Collector. Utilize Grafana's powerful query builder and visualization tools to derive insights from your data effectively.
+
+- Consider creating dashboards that offer a comprehensive overview of your infrastructure's health and performance.
+- Utilize Grafana's alerting features to proactively manage and respond to issues identified through the OpenTelemetry data.
+
+This guide simplifies the deployment of the OpenTelemetry Collector across multiple Linux hosts using Ansible and illustrates how to visualize collected telemetry data in Grafana Cloud. Tailor the Ansible roles, OpenTelemetry Collector configurations, and Grafana dashboards to suit your specific monitoring and observability requirements.
diff --git a/examples/monitor-multiple-instances.md → examples/monitor-multiple-instances-agent.md b/examples/monitor-multiple-instances.md → examples/monitor-multiple-instances-agent.md
diff --git a/examples/opentelemetry-collector.yml b/examples/opentelemetry-collector.yml
@@ -0,0 +1,43 @@
+- name: Install OpenTelemetry Collector
+  hosts: all
+  become: true
+
+  tasks:
+    - name: Install OpenTelemetry Collector
+      ansible.builtin.include_role:
+        name: grafana.grafana.opentelemetry_collector
+      vars:
+        otel_collector_receivers:
+          otlp:
+            protocols:
+              grpc:
+                endpoint: 0.0.0.0:4317
+              http:
+                endpoint: 0.0.0.0:4318
+        otel_collector_processors:
+          batch:
+
+        otel_collector_exporters:
+          otlp:
+            endpoint: otelcol:4317
+
+        otel_collector_extensions:
+          health_check:
+          pprof:
+          zpages:
+
+        otel_collector_service:
+          extensions: [health_check, pprof, zpages]
+          pipelines:
+            traces:
+              receivers: [otlp]
+              processors: [batch]
+              exporters: [otlp]
+            metrics:
+              receivers: [otlp]
+              processors: [batch]
+              exporters: [otlp]
+            logs:
+              receivers: [otlp]
+              processors: [batch]
+              exporters: [otlp]
diff --git a/roles/grafana_agent/README.md b/roles/grafana_agent/README.md
@@ -1,4 +1,4 @@
-# Role Name
+# Ansible Role for Grafana Agent
 
 Ansible Role to deploy Grafana Agent on Linux hosts. Using this Role, Grafana Agent can be deployed on RedHat, Ubuntu, Debian, CentOS
 and Fedora linux distributions.

diff --git a/roles/opentelemetry_collector/README.md b/roles/opentelemetry_collector/README.md
@@ -0,0 +1,100 @@
+# Ansible Role for OpenTelemetry Collector
+
+This Ansible role to install and configure the OpenTelemetry Collector, which can be used to collect traces, metrics, and logs.
+
+## Requirements
+
+Please ensure that `curl` is intalled on Ansible controller.
+
+## Role Variables
+
+Available variables with their default values are listed below (`defaults/main.yml`):
+
+| Variable Name | Description | Default Value |
+|---------------|-------------|---------------|
+| `otel_collector_version` | Version of OpenTelemetry Collector to install. | `"0.90.1"` |
+| `otel_collector_binary_url` | URL for downloading the OpenTelemetry Collector binary. This URL is constructed based on the collector version, type, and architecture. | `"https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v{{ otel_collector_version }}/{% if otel_collector_type == 'contrib' %}otelcol-contrib_{{ otel_collector_version }}_linux_{{ otel_collector_arch }}{% else %}otelcol_{{ otel_collector_version }}_linux_{{ otel_collector_arch }}{% endif %}.tar.gz"` |
+| `arch_mapping` | Mapping of `ansible_architecture` values to OpenTelemetry Collector binary architecture names. | See below\* |
+| `otel_collector_arch` | Architecture for the OpenTelemetry Collector binary, determined based on the `ansible_architecture` fact. | `"{{ arch_mapping[ansible_architecture] | default('amd64') }}"` |
+| `otel_collector_service_name` | The service name for the OpenTelemetry Collector. | `"otel-collector"` |
+| `otel_collector_type` | Type of the OpenTelemetry Collector (`contrib` includes additional components). | `contrib` |
+| `otel_collector_executable` | The executable name of the OpenTelemetry Collector, changes based on the collector type. | `{% if otel_collector_type == 'contrib' %}otelcol-contrib{% else %}otelcol{% endif %}` |
+| `otel_collector_installation_dir` | Installation directory for the OpenTelemetry Collector. | `"/etc/otel-collector"` |
+| `otel_collector_config_dir` | Directory for OpenTelemetry Collector configuration files. | `"/etc/otel-collector"` |
+| `otel_collector_config_file` | The main configuration file name for the OpenTelemetry Collector. | `"config.yaml"` |
+| `otel_collector_service_user` | The system user under which the OpenTelemetry Collector service will run. | `"otel"` |
+| `otel_collector_service_group` | The system group under which the OpenTelemetry Collector service will run. | `"otel"` |
+| `otel_collector_receivers` | Receivers configuration for the OpenTelemetry Collector. | `""` |
+| `otel_collector_exporters` | Exporters configuration for the OpenTelemetry Collector. | `""` |
+| `otel_collector_processors` | Processors configuration for the OpenTelemetry Collector. | `""` |
+| `otel_collector_extensions` | Extensions configuration for the OpenTelemetry Collector. | `""` |
+| `otel_collector_service` | Service configuration for the OpenTelemetry Collector. | `""` |
+| `otel_collector_connectors` | Connectors configuration for the OpenTelemetry Collector (optional). | `""` |
+
+\* For `arch_mapping`, the default mapping is as follows:
+- `x86_64`: `amd64`
+- `aarch64`: `arm64`
+- `armv7l`: `armhf`
+- `i386`: `i386`
+- `ppc64le`: `ppc64le`
+
+Users of the role can override these variables as needed.
+
+## Example Playbook
+
+Include this role in your playbook with default settings:
+
+```yaml
+- name: Install OpenTelemetry Collector
+  hosts: all
+  become: true
+
+  tasks: 
+    - name: Install OpenTelemetry Collector
+      ansible.builtin.include_role:
+        name: grafana.grafana.opentelemetry_collector
+      vars:
+        otel_collector_receivers:
+          otlp:
+            protocols:
+              grpc:
+                endpoint: 0.0.0.0:4317
+              http:
+                endpoint: 0.0.0.0:4318
+        otel_collector_processors:
+          batch:
+
+        otel_collector_exporters:
+          otlp:
+            endpoint: otelcol:4317
+
+        otel_collector_extensions:
+          health_check:
+          pprof:
+          zpages:
+
+        otel_collector_service:
+          extensions: [health_check, pprof, zpages]
+          pipelines:
+            traces:
+              receivers: [otlp]
+              processors: [batch]
+              exporters: [otlp]
+            metrics:
+              receivers: [otlp]
+              processors: [batch]
+              exporters: [otlp]
+            logs:
+              receivers: [otlp]
+              processors: [batch]
+              exporters: [otlp]
+
+```
+
+## License
+
+See [LICENSE](https://github.com/grafana/grafana-ansible-collection/blob/main/LICENSE)
+
+## Author Information
+
+-   [Ishan Jain](https://github.com/ishanjainn)
-Original file line number
+Diff line change
@@ Expand Up / @@ -5,6 +5,7 @@ @@
     .idea
     hosts
     *.log
+    yala
     # MacOS
     # ---------------------------------------------------
@@ Expand Down @@