From 20baae505bdb3ff9ab9fd89785ce9ab4e528289f Mon Sep 17 00:00:00 2001 From: Marc Sanmiquel Date: Wed, 6 Nov 2024 22:03:23 +0100 Subject: [PATCH 01/32] pyroscope: add stability note for receive_http component (#2046) --- .../reference/components/pyroscope/pyroscope.receive_http.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/sources/reference/components/pyroscope/pyroscope.receive_http.md b/docs/sources/reference/components/pyroscope/pyroscope.receive_http.md index 3b34f393a9..d5f50962f8 100644 --- a/docs/sources/reference/components/pyroscope/pyroscope.receive_http.md +++ b/docs/sources/reference/components/pyroscope/pyroscope.receive_http.md @@ -4,8 +4,12 @@ description: Learn about pyroscope.receive_http title: pyroscope.receive_http --- +Public preview + # pyroscope.receive_http +{{< docs/shared lookup="stability/public_preview.md" source="alloy" version="" >}} + `pyroscope.receive_http` receives profiles over HTTP and forwards them to `pyroscope.*` components capable of receiving profiles. The HTTP API exposed is compatible with the Pyroscope [HTTP ingest API](https://grafana.com/docs/pyroscope/latest/configure-server/about-server-api/). From 31b697c95ecdf7d43fa7790e31cc0f231d39fe8d Mon Sep 17 00:00:00 2001 From: adlotsof <6109332+adlotsof@users.noreply.github.com> Date: Thu, 7 Nov 2024 14:01:27 +0100 Subject: [PATCH 02/32] feat(otelcol/exporter/splunkhec) Add Splunk HTTP Event Collector (HEC) Exporter (#1885) * initial setup * deps * have some docs, fix some bugs * add more docs * fix bugs, proper alloy.Secret * have nested blocks * have default blocks (queue, retry, debug metrics) * have docs for the default blocks * add compability * fix file name * cleanup, changelog, gen docs * linter * Update docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * fix conflicts, deps * Apply suggestions from code review Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: William Dumont * fix docs, fix missing batcherconfig, fix interval in duration, add more validation * fix ordering in go.mod, add converter scaffolding * add converter test data, currently broken test * Update docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * deps * fix spelling of component name * fix converter datatype * fix test type * fix include unit for duration * fix return without assignment * fix address review comment: add native oltp example to docs * fix docs: Apply suggestions from code review Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * remove defaults * fix typo * add missing field * docs: add default values for client * address review comments * additional documentation * add maintainer * update deps * update docs --------- Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> Co-authored-by: William Dumont --- CHANGELOG.md | 1 + .../sources/reference/compatibility/_index.md | 1 + .../otelcol/otelcol.exporter.splunkhec.md | 374 ++++++++++++++++++ go.mod | 3 + go.sum | 6 + internal/component/all/all.go | 1 + .../exporter/splunkhec/config/splunkhec.go | 304 ++++++++++++++ .../splunkhec/config/splunkhec_test.go | 219 ++++++++++ .../otelcol/exporter/splunkhec/splunkhec.go | 89 +++++ .../exporter/splunkhec/splunkhec_test.go | 229 +++++++++++ .../converter_splunkhecexporter.go | 126 ++++++ .../otelcolconvert/testdata/splunkhec.alloy | 56 +++ .../otelcolconvert/testdata/splunkhec.yaml | 62 +++ 13 files changed, 1471 insertions(+) create mode 100644 docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md create mode 100644 internal/component/otelcol/exporter/splunkhec/config/splunkhec.go create mode 100644 internal/component/otelcol/exporter/splunkhec/config/splunkhec_test.go create mode 100644 internal/component/otelcol/exporter/splunkhec/splunkhec.go create mode 100644 internal/component/otelcol/exporter/splunkhec/splunkhec_test.go create mode 100644 internal/converter/internal/otelcolconvert/converter_splunkhecexporter.go create mode 100644 internal/converter/internal/otelcolconvert/testdata/splunkhec.alloy create mode 100644 internal/converter/internal/otelcolconvert/testdata/splunkhec.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index c0f63490e4..db7cbf15b5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,6 +40,7 @@ v1.5.0-rc.0 - Add support to `loki.source.api` to be able to extract the tenant from the HTTP `X-Scope-OrgID` header (@QuentinBisson) - (_Experimental_) Add a `loki.secretfilter` component to redact secrets from collected logs. +- Add `otelcol.exporter.splunkhec` allowing to export otel data to Splunk HEC (@adlotsof) - (_Experimental_) Add a `prometheus.write.queue` component to add an alternative to `prometheus.remote_write` which allowing the writing of metrics to a prometheus endpoint. (@mattdurham) diff --git a/docs/sources/reference/compatibility/_index.md b/docs/sources/reference/compatibility/_index.md index 6315341266..d55681be97 100644 --- a/docs/sources/reference/compatibility/_index.md +++ b/docs/sources/reference/compatibility/_index.md @@ -298,6 +298,7 @@ The following components, grouped by namespace, _export_ OpenTelemetry `otelcol. - [otelcol.exporter.otlp](../components/otelcol/otelcol.exporter.otlp) - [otelcol.exporter.otlphttp](../components/otelcol/otelcol.exporter.otlphttp) - [otelcol.exporter.prometheus](../components/otelcol/otelcol.exporter.prometheus) +- [otelcol.exporter.splunkhec](../components/otelcol/otelcol.exporter.splunkhec) - [otelcol.processor.attributes](../components/otelcol/otelcol.processor.attributes) - [otelcol.processor.batch](../components/otelcol/otelcol.processor.batch) - [otelcol.processor.deltatocumulative](../components/otelcol/otelcol.processor.deltatocumulative) diff --git a/docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md b/docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md new file mode 100644 index 0000000000..b7df971b4f --- /dev/null +++ b/docs/sources/reference/components/otelcol/otelcol.exporter.splunkhec.md @@ -0,0 +1,374 @@ +--- +canonical: https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.exporter.splunkhec/ +description: Learn about otelcol.exporter.splunkhec +title: otelcol.exporter.splunkhec +--- + +Community + +# otelcol.exporter.splunkhec + +{{< docs/shared lookup="stability/community.md" source="alloy" version="" >}} + +`otelcol.exporter.splunkhec` accepts metrics and traces telemetry data from other `otelcol` components and sends it to Splunk HEC. + +{{< admonition type="note" >}} +`otelcol.exporter.splunkhec` is a wrapper over the upstream OpenTelemetry Collector `splunkhec` exporter from the `otelcol-contrib` distribution. +Bug reports or feature requests will be redirected to the upstream repository, if necessary. +{{< /admonition >}} + +You can specify multiple `otelcol.exporter.splunkhec` components by giving them different labels. + +## Usage + +```alloy +otelcol.exporter.splunkhec "LABEL" { + splunk { + token = "YOUR_SPLUNK_TOKEN" + } + client { + endpoint = "http://splunk.yourdomain.com:8088" + } +} +``` + +## Arguments + +The `otelcol.exporter.splunkhec` component does not support any arguments, and is configured +fully through child blocks. + +## Blocks + +The following blocks are supported inside the definition of `otelcol.exporter.splunkhec`: + +Hierarchy | Block | Description | Required +---------------------------|-------------------------------|----------------------------------------------------------------------------|--------- +splunk | [splunk][] | Configures the Splunk HEC exporter. | yes +splunk->otel_to_hec_fields | [otel_to_hec_fields][] | Configures mapping of Open Telemetry to HEC Fields. | no +splunk->telemetry | [telemetry][] | Configures the exporters telemetry. | no +splunk->heartbeat | [heartbeat][] | Configures the exporters heartbeat settings. | no +splunk->batcher | [batcher][] | Configures batching requests based on a timeout and a minimum number of items. | no +client | [client][] | Configures the HTTP client used to send data to Splunk HEC. | yes +retry_on_failure | [retry_on_failure][] | Configures retry mechanism for failed requests. | no +queue | [queue][] | Configures batching of data before sending. | no +debug_metrics | [debug_metrics][] | Configures the metrics that this component generates to monitor its state. | no + + +[splunk]: #splunk-block +[otel_to_hec_fields]: #otel_to_hec_fields-block +[telemetry]: #telemetry-block +[heartbeat]: #heartbeat-block +[batcher]: #batcher-block +[client]: #client-block +[retry_on_failure]: #retry_on_failure-block +[queue]: #queue-block +[debug_metrics]: #debug_metrics-block + +### splunk block + +The `splunk` block configures Splunk HEC specific settings. + +The following arguments are supported: + +Name | Type | Description | Default | Required +---------------------------|----------|-----------------------------------------------------------------|-------------------------------|--------- +`token` | `secret` | Splunk HEC Token. | | yes +`log_data_enabled` | `bool` | Enable sending logs from the exporter. One of `log_data_enabled` or `profiling_data_enabled` must be `true`. | `true` | no +`profiling_data_enabled` | `bool` | Enable sending profiling data from the exporter. One of `log_data_enabled` or `profiling_data_enabled` must be `true`. | `true` | no +`source` | `string` | [Splunk source](https://docs.splunk.com/Splexicon:Source). | `""` | no +`source_type` | `string` | [Splunk source sype](https://docs.splunk.com/Splexicon:Sourcetype). | `""` | no +`index` | `string` | Splunk index name. | `""` | no +`disable_compression` | `bool` | Disable GZip compression. | `false` | no +`max_content_length_logs` | `uint` | Maximum log payload size in bytes. Must be less than 838860800 (~800MB). | `2097152` | no +`max_content_length_metrics` | `uint` | Maximum metric payload size in bytes. Must be less than 838860800 (~800MB). | `2097152` | no +`max_content_length_traces` | `uint` | Maximum trace payload size in bytes. Must be less than 838860800 (~800MB). | `2097152` | no +`max_event_size` | `uint` | Maximum event payload size in bytes. Must be less than 838860800 (~800MB). | `5242880` | no +`splunk_app_name` | `string` | Used to track telemetry for Splunk Apps by name. | `Alloy` | no +`splunk_app_version` | `string` | Used to track telemetry by App version. | `""` | no +`health_path` | `string` | Path for the health API. | `/services/collector/health'` | no +`health_check_enabled` | `bool` | Used to verify Splunk HEC health on exporter startup. | `true` | no +`export_raw` | `bool` | Send only the logs body when targeting HEC raw endpoint. | `false` | no +`use_multi_metrics_format` | `bool` | Use multi-metrics format to save space during ingestion. | `false` | no + + +#### otel_to_hec_fields block + +Name | Type | Description | Default | Required +---------------------------|----------|-----------------------------------------------------------------|-------------------------------|--------- +`severity_text` | `string` | Maps severity text field to a specific HEC field. | `""` | no +`severity_number` | `string` | Maps severity number field to a specific HEC field. | `""` | no + + +#### heartbeat block + +Name | Type | Description | Default | Required +---------------------------|----------|-----------------------------------------------------------------|-------------------------------|--------- +`interval` | `time.Duration` | Time interval for the heartbeat interval, in seconds. | `0s` | no +`startup` | `bool` | Send heartbeat events on exporter startup. | `false` | no + + +#### telemetry block + +Name | Type | Description | Default | Required +---------------------------|-----------------------|-----------------------------------------------------------------|-------------------------------|--------- +`enabled` | `bool` | Enable telemetry inside the exporter. | `false` | no +`override_metrics_names` | `map(string)` | Override metrics for internal metrics in the exporter. | | no + +#### batcher block + +Name | Type | Description | Default | Required +---------------------------|-----------------------|-----------------------------------------------------------------|-------------------------------|--------- +`enabled` | `bool` | Whether to not enqueue batches before sending to the consumerSender. | `false` | no +`flush_timeout` | `time.Duration` | The time after which a batch will be sent regardless of its size. | `200ms` | no +`min_size_items` | `uint` | The number of items at which the batch is sent regardless of the timeout. | `8192` | no +`max_size_items` | `uint` | Maximum number of batch items, if the batch exceeds this value, it will be broken up into smaller batches. Must be greater than or equal to min_size_items. Setting this value to zero disables the maximum size limit. | `0` | no + +### client block + +The `client` block configures the HTTP client used by the component. + +The following arguments are supported: + +Name | Type | Description | Default | Required +--------------------------|------------|-----------------------------------------------------------------------------|---------|--------- +`endpoint` | `string` | The Splunk HEC endpoint to use. | | yes +`read_buffer_size` | `int` | Size of the read buffer the HTTP client uses for reading server responses. | `0` | no +`write_buffer_size` | `int` | Size of the write buffer the HTTP client uses for writing requests. | `0` | no +`timeout` | `duration` | Time to wait before marking a request as failed. | `"15s"` | no +`max_idle_conns` | `int` | Limits the number of idle HTTP connections the client can keep open. | `100` | no +`max_idle_conns_per_host` | `int` | Limits the number of idle HTTP connections the host can keep open. | `2` | no +`max_conns_per_host` | `int` | Limits the total (dialing,active, and idle) number of connections per host. Zero means no limit | `0` | no +`idle_conn_timeout` | `duration` | Time to wait before an idle connection closes itself. | `"45s"` | no +`disable_keep_alives` | `bool` | Disable HTTP keep-alive. | `false` | no +`insecure_skip_verify` | `bool` | Ignores insecure server TLS certificates. | `false` | no + +### retry_on_failure block + +The `retry_on_failure` block configures how failed requests to splunkhec are retried. + +{{< docs/shared lookup="reference/components/otelcol-retry-block.md" source="alloy" version="" >}} + +### queue block + +The `queue` block configures an in-memory buffer of batches before data is sent to the HTTP server. + +{{< docs/shared lookup="reference/components/otelcol-queue-block.md" source="alloy" version="" >}} + +### debug_metrics block + +{{< docs/shared lookup="reference/components/otelcol-debug-metrics-block.md" source="alloy" version="" >}} + +## Exported fields + +The following fields are exported and can be referenced by other components: + +Name | Type | Description +--------|--------------------|----------------------------------------------------------------- +`input` | `otelcol.Consumer` | A value other components can use to send telemetry data to. + +`input` accepts `otelcol.Consumer` data for any telemetry signal (metrics, logs, or traces). + +## Component health + +`otelcol.exporter.splunkhec` is only reported as unhealthy if given an invalid configuration. + +## Debug information + +`otelcol.exporter.splunkhec` does not expose any component-specific debug information. + +## Example + +### Open Telemetry Receiver + +This example forwards metrics, logs and traces send to the `otelcol.receiver.otlp.default` receiver to the Splunk HEC exporter. + +``` +otelcol.receiver.otlp "default" { + grpc { + endpoint = "localhost:4317" + } + + http { + endpoint = "localhost:4318" + compression_algorithms = ["zlib"] + } + + output { + metrics = [otelcol.exporter.splunkhec.default.input] + logs = [otelcol.exporter.splunkhec.default.input] + traces = [otelcol.exporter.splunkhec.default.input] + } +} + +otelcol.exporter.splunkhec "default" { + client { + endpoint = "https://splunkhec.domain.com:8088/services/collector" + timeout = "10s" + max_idle_conns = 200 + max_idle_conns_per_host = 200 + idle_conn_timeout = "10s" + } + + splunk { + token = "SPLUNK_TOKEN" + source = "otel" + sourcetype = "otel" + index = "metrics" + splunk_app_name = "OpenTelemetry-Collector Splunk Exporter" + splunk_app_version = "v0.0.1" +z + otel_to_hec_fields { + severity_text = "otel.log.severity.text" + severity_number = "otel.log.severity.number" + } + + heartbeat { + interval = "30s" + } + + telemetry { + enabled = true + override_metrics_names = { + otelcol_exporter_splunkhec_heartbeats_failed = "app_heartbeats_failed_total", + otelcol_exporter_splunkhec_heartbeats_sent = "app_heartbeats_success_total", + } + extra_attributes = { + custom_key = "custom_value", + dataset_name = "SplunkCloudBeaverStack", + } + } + } +} +``` + +### Forward Prometheus Metrics +This example forwards Prometheus metrics from {{< param "PRODUCT_NAME" >}} through a receiver for conversion to Open Telemetry format before finally sending them to splunkhec. + +```alloy +prometheus.exporter.self "default" { +} + +prometheus.scrape "metamonitoring" { + targets = prometheus.exporter.self.default.targets + forward_to = [otelcol.receiver.prometheus.default.receiver] +} + +otelcol.receiver.prometheus "default" { + output { + metrics = [otelcol.exporter.splunkhec.default.input] + } +} + + +otelcol.exporter.splunkhec "default" { + splunk { + token = "SPLUNK_TOKEN" + } + client { + endpoint = "http://splunkhec.domain.com:8088" + } +} +``` + +### Forward Loki logs + +This example watches for files ending with `.log` in the path `/var/log`, tails these logs with Loki and forwards the logs to the configured Splunk HEC endpoint. The Splunk HEC exporter component is setup to send an heartbeat every 5 seconds. + +```alloy +local.file_match "local_files" { + path_targets = [{"__path__" = "/var/log/*.log"}] + sync_period = "5s" +} + +otelcol.receiver.loki "default" { + output { + logs = [otelcol.processor.attributes.default.input] + } +} + +otelcol.processor.attributes "default" { + action { + key = "host" + action = "upsert" + value = "myhost" + } + + action { + key = "host.name" + action = "upsert" + value = "myhost" + } + + output { + logs = [otelcol.exporter.splunkhec.default.input] + } +} + +loki.source.file "log_scrape" { + targets = local.file_match.local_files.targets + forward_to = [otelcol.receiver.loki.default.receiver] + tail_from_end = false +} + +otelcol.exporter.splunkhec "default" { + retry_on_failure { + enabled = false + } + + client { + endpoint = "http://splunkhec.domain.com:8088" + timeout = "5s" + max_idle_conns = 200 + max_idle_conns_per_host = 200 + idle_conn_timeout = "10s" + write_buffer_size = 8000 + } + + sending_queue { + enabled = false + } + + splunk { + token = "SPLUNK_TOKEN" + source = "otel" + sourcetype = "otel" + index = "devnull" + log_data_enabled = true + + heartbeat { + interval = "5s" + } + + batcher { + flush_timeout = "200ms" + } + + telemetry { + enabled = true + override_metrics_names = { + otelcol_exporter_splunkhec_heartbeats_failed = "app_heartbeats_failed_total", + otelcol_exporter_splunkhec_heartbeats_sent = "app_heartbeats_success_total", + } + extra_attributes = { + host = "myhost", + dataset_name = "SplunkCloudBeaverStack", + } + } + } +} +``` + + + +## Compatible components + +`otelcol.exporter.splunkhec` has exports that can be consumed by the following components: + +- Components that consume [OpenTelemetry `otelcol.Consumer`](../../../compatibility/#opentelemetry-otelcolconsumer-consumers) + +{{< admonition type="note" >}} +Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. +Refer to the linked documentation for more details. +{{< /admonition >}} + + \ No newline at end of file diff --git a/go.mod b/go.mod index 449008346f..591e8b837a 100644 --- a/go.mod +++ b/go.mod @@ -658,6 +658,7 @@ require ( github.com/nicolai86/scaleway-sdk v1.10.2-0.20180628010248-798f60e20bb2 // indirect github.com/ohler55/ojg v1.20.1 // indirect github.com/oklog/ulid v1.3.1 // indirect + github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter v0.112.0 github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/ecsutil v0.112.0 // indirect github.com/open-telemetry/opentelemetry-collector-contrib/internal/common v0.112.0 // indirect github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal v0.112.0 // indirect @@ -668,6 +669,8 @@ require ( github.com/open-telemetry/opentelemetry-collector-contrib/internal/metadataproviders v0.112.0 // indirect github.com/open-telemetry/opentelemetry-collector-contrib/internal/pdatautil v0.112.0 // indirect github.com/open-telemetry/opentelemetry-collector-contrib/internal/sharedcomponent v0.112.0 // indirect + github.com/open-telemetry/opentelemetry-collector-contrib/internal/splunk v0.112.0 // indirect + github.com/open-telemetry/opentelemetry-collector-contrib/pkg/batchperresourceattr v0.112.0 // indirect github.com/open-telemetry/opentelemetry-collector-contrib/pkg/batchpersignal v0.112.0 // indirect github.com/open-telemetry/opentelemetry-collector-contrib/pkg/resourcetotelemetry v0.112.0 // indirect github.com/open-telemetry/opentelemetry-collector-contrib/pkg/sampling v0.112.0 // indirect diff --git a/go.sum b/go.sum index f1c249de2a..92de4ca5eb 100644 --- a/go.sum +++ b/go.sum @@ -1931,6 +1931,8 @@ github.com/open-telemetry/opentelemetry-collector-contrib/exporter/loadbalancing github.com/open-telemetry/opentelemetry-collector-contrib/exporter/loadbalancingexporter v0.112.0/go.mod h1:B7HK7eq+M4PEOaKGOAJeZmNqnabtXXEtnpxK40j1T9o= github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.112.0 h1:wZQdqnXCERudSXjSE4V5NNS9rCb82oYCgjH+c+wPt+k= github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.112.0/go.mod h1:QwYTlmQDuLeaxS0HkIG9K9x45vQhHzL0SvI8inxzMeU= +github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter v0.112.0 h1:bIoCW8VYBEGnvpNYlamlvkPyeoQHCtfGgEuuELJYWYE= +github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter v0.112.0/go.mod h1:7usJQKG52/DDvzJ7Vm5+QEBE1eAYrVhEYbzYFzfkn2Q= github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.112.0 h1:RY0/7LTffj76403QxSlEjb0gnF788Qyfpxc+y32Rd6c= github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.112.0/go.mod h1:1Z84oB3hwUH1B3IsL46csEtu7WA1qQJ/p6USTulGJf4= github.com/open-telemetry/opentelemetry-collector-contrib/extension/bearertokenauthextension v0.112.0 h1:GLh1rnXcY4P2hkMwuMLYCZMjZxze1KnciJXJTOFXOJ8= @@ -1967,6 +1969,10 @@ github.com/open-telemetry/opentelemetry-collector-contrib/internal/pdatautil v0. github.com/open-telemetry/opentelemetry-collector-contrib/internal/pdatautil v0.112.0/go.mod h1:BE54/UEIN+Q5EBlJq/TSc4lf7PtCjxgyBRH7XB/zTYE= github.com/open-telemetry/opentelemetry-collector-contrib/internal/sharedcomponent v0.112.0 h1:g7lqlvAqhkbZ1cPcbDNRxawb0MOzlvm9AevE+0MlknQ= github.com/open-telemetry/opentelemetry-collector-contrib/internal/sharedcomponent v0.112.0/go.mod h1:oep6LZlvIzNGKT4wx5duQuxELChvmOX1YzM/nmBU8BI= +github.com/open-telemetry/opentelemetry-collector-contrib/internal/splunk v0.112.0 h1:Zy0tTlUqipMWRhR7qFvJB+26py6kjAmLrrodQKxXKLc= +github.com/open-telemetry/opentelemetry-collector-contrib/internal/splunk v0.112.0/go.mod h1:WkiKDn5BVT3Cq1y2E0Xf/0D+QhbokmST9+LnlsGqtQ4= +github.com/open-telemetry/opentelemetry-collector-contrib/pkg/batchperresourceattr v0.112.0 h1:1ec6s2PCc7mB8oiAJH/HY75Hhgn5GNgyWU/lEvlFrB0= +github.com/open-telemetry/opentelemetry-collector-contrib/pkg/batchperresourceattr v0.112.0/go.mod h1:DEO0PViQHIA8rWsV+AfdPv2IZGgCXk3/xJsZZqehNe8= github.com/open-telemetry/opentelemetry-collector-contrib/pkg/batchpersignal v0.112.0 h1:wG/lE9KIZfo/R06AJHuOH5Gm1C8zI9FOOE/hI0p3Rfw= github.com/open-telemetry/opentelemetry-collector-contrib/pkg/batchpersignal v0.112.0/go.mod h1:9fbK85xo0GAcvXbaHbnmMrt3dIBWs6FMObh2oaLxI6s= github.com/open-telemetry/opentelemetry-collector-contrib/pkg/datadog v0.112.0 h1:SPKphhxWV0jfv/872sFmC5G3j/qUKGgr5KdGERHYoPQ= diff --git a/internal/component/all/all.go b/internal/component/all/all.go index ef65b8081c..4f58d9fbe6 100644 --- a/internal/component/all/all.go +++ b/internal/component/all/all.go @@ -77,6 +77,7 @@ import ( _ "github.com/grafana/alloy/internal/component/otelcol/exporter/otlp" // Import otelcol.exporter.otlp _ "github.com/grafana/alloy/internal/component/otelcol/exporter/otlphttp" // Import otelcol.exporter.otlphttp _ "github.com/grafana/alloy/internal/component/otelcol/exporter/prometheus" // Import otelcol.exporter.prometheus + _ "github.com/grafana/alloy/internal/component/otelcol/exporter/splunkhec" // Import otelcol.exporter.splunkhec _ "github.com/grafana/alloy/internal/component/otelcol/extension/jaeger_remote_sampling" // Import otelcol.extension.jaeger_remote_sampling _ "github.com/grafana/alloy/internal/component/otelcol/processor/attributes" // Import otelcol.processor.attributes _ "github.com/grafana/alloy/internal/component/otelcol/processor/batch" // Import otelcol.processor.batch diff --git a/internal/component/otelcol/exporter/splunkhec/config/splunkhec.go b/internal/component/otelcol/exporter/splunkhec/config/splunkhec.go new file mode 100644 index 0000000000..e1e043bf1b --- /dev/null +++ b/internal/component/otelcol/exporter/splunkhec/config/splunkhec.go @@ -0,0 +1,304 @@ +package splunkhec_config + +import ( + "errors" + "time" + + "github.com/grafana/alloy/syntax/alloytypes" + "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter" + "go.opentelemetry.io/collector/config/confighttp" + "go.opentelemetry.io/collector/config/configopaque" + "go.opentelemetry.io/collector/config/configretry" + "go.opentelemetry.io/collector/config/configtls" + "go.opentelemetry.io/collector/exporter/exporterbatcher" + "go.opentelemetry.io/collector/exporter/exporterhelper" +) + +type SplunkHecClientArguments struct { + // Endpoint is the Splunk HEC endpoint to send data to. + Endpoint string `alloy:"endpoint,attr"` + // ReadBufferSize for the HTTP client. + ReadBufferSize int `alloy:"read_buffer_size,attr,optional"` + // WriteBufferSize for the HTTP client. + WriteBufferSize int `alloy:"write_buffer_size,attr,optional"` + // Timeout for the HTTP client. Defaults to 15 seconds. + Timeout time.Duration `alloy:"timeout,attr,optional"` + // MaxIdleConns for the HTTP client. + MaxIdleConns *int `alloy:"max_idle_conns,attr,optional"` + // MaxIdleConnsPerHost for the HTTP client. + MaxIdleConnsPerHost *int `alloy:"max_idle_conns_per_host,attr,optional"` + // MaxConnsPerHost for the HTTP client. + MaxConnsPerHost *int `alloy:"max_conns_per_host,attr,optional"` + // IdleConnTimeout for the HTTP client. + IdleConnTimeout *time.Duration `alloy:"idle_conn_timeout,attr,optional"` + // DisableKeepAlives for the HTTP client. + DisableKeepAlives bool `alloy:"disable_keep_alives,attr,optional"` + // TLSSetting for the HTTP client. + InsecureSkipVerify bool `alloy:"insecure_skip_verify,attr,optional"` +} +type SplunkConf struct { + // until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved. + BatcherConfig BatcherConfig `alloy:"batcher,block,optional"` + + // Experimental: This configuration is at the early stage of development and may change without backward compatibility + // until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved. + // LogDataEnabled can be used to disable sending logs by the exporter. + LogDataEnabled bool `alloy:"log_data_enabled,attr,optional"` + // ProfilingDataEnabled can be used to disable sending profiling data by the exporter. + ProfilingDataEnabled bool `alloy:"profiling_data_enabled,attr,optional"` + // HEC Token is the authentication token provided by Splunk: https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector. + Token alloytypes.Secret `alloy:"token,attr"` + // Optional Splunk source: https://docs.splunk.com/Splexicon:Source. + // Sources identify the incoming data. + Source string `alloy:"source,attr,optional"` + // Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype. + SourceType string `alloy:"sourcetype,attr,optional"` + // Splunk index, optional name of the Splunk index. + Index string `alloy:"index,attr,optional"` + // Disable GZip compression. Defaults to false. + DisableCompression bool `alloy:"disable_compression,attr,optional"` + // Maximum log payload size in bytes. Default value is 2097152 bytes (2MiB). + // Maximum allowed value is 838860800 (~ 800 MB). + MaxContentLengthLogs uint `alloy:"max_content_length_logs,attr,optional"` + // Maximum metric payload size in bytes. Default value is 2097152 bytes (2MiB). + // Maximum allowed value is 838860800 (~ 800 MB). + MaxContentLengthMetrics uint `alloy:"max_content_length_metrics,attr,optional"` + // Maximum trace payload size in bytes. Default value is 2097152 bytes (2MiB). + // Maximum allowed value is 838860800 (~ 800 MB). + MaxContentLengthTraces uint `alloy:"max_content_length_traces,attr,optional"` + // Maximum payload size, raw uncompressed. Default value is 5242880 bytes (5MiB). + // Maximum allowed value is 838860800 (~ 800 MB). + MaxEventSize uint `alloy:"max_event_size,attr,optional"` + // App name is used to track telemetry information for Splunk App's using HEC by App name. Defaults to "OpenTelemetry Collector Contrib". + SplunkAppName string `alloy:"splunk_app_name,attr,optional"` + // App version is used to track telemetry information for Splunk App's using HEC by App version. Defaults to the current OpenTelemetry Collector Contrib build version. + SplunkAppVersion string `alloy:"splunk_app_version,attr,optional"` + // HecFields creates a mapping from attributes to HEC fields. + HecFields HecFields `alloy:"otel_to_hec_fields,block,optional"` + // HealthPath for health API, default is '/services/collector/health' + HealthPath string `alloy:"health_path,attr,optional"` + // HecHealthCheckEnabled can be used to verify Splunk HEC health on exporter's startup + HecHealthCheckEnabled bool `alloy:"health_check_enabled,attr,optional"` + // ExportRaw to send only the log's body, targeting a Splunk HEC raw endpoint. + ExportRaw bool `alloy:"export_raw,attr,optional"` + // UseMultiMetricFormat combines metric events to save space during ingestion. + UseMultiMetricFormat bool `alloy:"use_multi_metric_format,attr,optional"` + // Heartbeat is the configuration to enable heartbeat + Heartbeat SplunkHecHeartbeat `alloy:"heartbeat,block,optional"` + // Telemetry is the configuration for splunk hec exporter telemetry + Telemetry SplunkHecTelemetry `alloy:"telemetry,block,optional"` +} + +type BatcherConfig struct { + // Enabled indicates whether to not enqueue batches before sending to the consumerSender. + Enabled bool `alloy:"enabled,attr,optional"` + + // FlushTimeout sets the time after which a batch will be sent regardless of its size. + FlushTimeout time.Duration `alloy:"flush_timeout,attr,optional"` + // MinSizeItems is the number of items (spans, data points or log records for OTLP) at which the batch should be + // sent regardless of the timeout. There is no guarantee that the batch size always greater than this value. + // This option requires the Request to implement RequestItemsCounter interface. Otherwise, it will be ignored. + MinSizeItems int `alloy:"min_size_items,attr,optional"` + // MaxSizeItems is the maximum number of the batch items, i.e. spans, data points or log records for OTLP. + // If the batch size exceeds this value, it will be broken up into smaller batches if possible. + // Setting this value to zero disables the maximum size limit. + MaxSizeItems int `alloy:"max_size_items,attr,optional"` +} + +func (args *BatcherConfig) Convert() *exporterbatcher.Config { + if args == nil { + return nil + } + return &exporterbatcher.Config{ + Enabled: args.Enabled, + FlushTimeout: args.FlushTimeout, + MinSizeConfig: exporterbatcher.MinSizeConfig{MinSizeItems: args.MinSizeItems}, + MaxSizeConfig: exporterbatcher.MaxSizeConfig{MaxSizeItems: args.MaxSizeItems}, + } +} + +type HecFields struct { + // SeverityText informs the exporter to map the severity text field to a specific HEC field. + SeverityText string `alloy:"severity_text,attr,optional"` + // SeverityNumber informs the exporter to map the severity number field to a specific HEC field. + SeverityNumber string `alloy:"severity_number,attr,optional"` +} + +func (args *HecFields) Convert() *splunkhecexporter.OtelToHecFields { + if args == nil { + return nil + } + return &splunkhecexporter.OtelToHecFields{ + SeverityText: args.SeverityText, + SeverityNumber: args.SeverityNumber, + } +} + +// SplunkHecHeartbeat defines the configuration for the Splunk HEC exporter heartbeat. +type SplunkHecHeartbeat struct { + // Interval represents the time interval for the heartbeat interval. If nothing or 0 is set, + // heartbeat is not enabled. + // A heartbeat is an event sent to _internal index with metadata for the current collector/host. + // In seconds + Interval time.Duration `alloy:"interval,attr,optional"` + + // Startup is used to send heartbeat events on exporter's startup. + Startup bool `alloy:"startup,attr,optional"` +} + +func (args *SplunkHecHeartbeat) Convert() *splunkhecexporter.HecHeartbeat { + if args == nil { + return nil + } + return &splunkhecexporter.HecHeartbeat{ + Interval: args.Interval, + Startup: args.Startup, + } +} + +// SplunkHecTelemetry defines the configuration for the Splunk HEC exporter internal telemetry. +type SplunkHecTelemetry struct { + // Enabled can be used to disable sending telemetry data by the exporter. + Enabled bool `alloy:"enabled,attr,optional"` + // Override metrics names for telemetry. + OverrideMetricsNames map[string]string `alloy:"override_metrics_names,attr,optional"` + // extra attributes to be added to telemetry data. + ExtraAttributes map[string]string `alloy:"extra_attributes,attr,optional"` +} + +func (args *SplunkHecTelemetry) Convert() *splunkhecexporter.HecTelemetry { + if args == nil { + return nil + } + return &splunkhecexporter.HecTelemetry{ + Enabled: args.Enabled, + OverrideMetricsNames: args.OverrideMetricsNames, + ExtraAttributes: args.ExtraAttributes, + } +} + +// SplunkHecClientArguments defines the configuration for the Splunk HEC exporter. +type SplunkHecArguments struct { + SplunkHecClientArguments SplunkHecClientArguments `alloy:"client,block"` + QueueSettings exporterhelper.QueueConfig `alloy:"queue,block,optional"` + RetrySettings configretry.BackOffConfig `alloy:"retry_on_failure,block,optional"` + Splunk SplunkConf `alloy:"splunk,block"` +} + +func (args *SplunkHecClientArguments) Convert() *confighttp.ClientConfig { + if args == nil { + return nil + } + return &confighttp.ClientConfig{ + Endpoint: args.Endpoint, + ReadBufferSize: args.ReadBufferSize, + WriteBufferSize: args.WriteBufferSize, + Timeout: args.Timeout, + MaxIdleConns: args.MaxIdleConns, + MaxIdleConnsPerHost: args.MaxIdleConnsPerHost, + MaxConnsPerHost: args.MaxConnsPerHost, + IdleConnTimeout: args.IdleConnTimeout, + DisableKeepAlives: args.DisableKeepAlives, + TLSSetting: configtls.ClientConfig{ + InsecureSkipVerify: args.InsecureSkipVerify, + }, + } +} + +func (args *SplunkHecClientArguments) SetToDefault() { + args.Timeout = 15 * time.Second +} + +func (args *SplunkHecClientArguments) Validate() error { + if args.Endpoint == "" { + return errors.New("missing Splunk hec endpoint") + } + return nil +} + +func (args *SplunkConf) SetToDefault() { + args.BatcherConfig = BatcherConfig{ + Enabled: false, + FlushTimeout: 200 * time.Millisecond, + MinSizeItems: 8192, + MaxSizeItems: 0, + } + args.LogDataEnabled = true + args.ProfilingDataEnabled = true + args.Source = "" + args.SourceType = "" + args.Index = "" + args.DisableCompression = false + args.MaxContentLengthLogs = 2097152 + args.MaxContentLengthMetrics = 2097152 + args.MaxContentLengthTraces = 2097152 + args.MaxEventSize = 5242880 + args.SplunkAppName = "Alloy" + args.SplunkAppVersion = "" + args.HealthPath = "/services/collector/health" + args.HecHealthCheckEnabled = false + args.ExportRaw = false + args.UseMultiMetricFormat = false + args.Heartbeat = SplunkHecHeartbeat{} + args.Telemetry = SplunkHecTelemetry{} +} + +func (args *SplunkConf) Validate() error { + if args.Token == "" { + return errors.New("missing splunk token") + } + if !args.LogDataEnabled && !args.ProfilingDataEnabled { + return errors.New("at least one of log_data_enabled or profiling_data_enabled must be enabled") + } + if args.MaxContentLengthLogs > 838860800 { + return errors.New("max_content_length_logs must be less than 838860800") + } + if args.MaxContentLengthMetrics > 838860800 { + return errors.New("max_content_length_metrics must be less than 838860800") + } + if args.MaxContentLengthTraces > 838860800 { + return errors.New("max_content_length_traces must be less than 838860800") + } + + return nil +} + +// Convert converts args into the upstream type +func (args *SplunkHecArguments) Convert() *splunkhecexporter.Config { + if args == nil { + return nil + } + return &splunkhecexporter.Config{ + ClientConfig: *args.SplunkHecClientArguments.Convert(), + QueueSettings: args.QueueSettings, + BackOffConfig: args.RetrySettings, + BatcherConfig: *args.Splunk.BatcherConfig.Convert(), + LogDataEnabled: args.Splunk.LogDataEnabled, + ProfilingDataEnabled: args.Splunk.ProfilingDataEnabled, + Token: configopaque.String(args.Splunk.Token), + Source: args.Splunk.Source, + SourceType: args.Splunk.SourceType, + Index: args.Splunk.Index, + DisableCompression: args.Splunk.DisableCompression, + MaxContentLengthLogs: args.Splunk.MaxContentLengthLogs, + MaxContentLengthMetrics: args.Splunk.MaxContentLengthMetrics, + MaxContentLengthTraces: args.Splunk.MaxContentLengthTraces, + MaxEventSize: args.Splunk.MaxEventSize, + SplunkAppName: args.Splunk.SplunkAppName, + SplunkAppVersion: args.Splunk.SplunkAppVersion, + HecFields: *args.Splunk.HecFields.Convert(), + HealthPath: args.Splunk.HealthPath, + HecHealthCheckEnabled: args.Splunk.HecHealthCheckEnabled, + ExportRaw: args.Splunk.ExportRaw, + UseMultiMetricFormat: args.Splunk.UseMultiMetricFormat, + Heartbeat: *args.Splunk.Heartbeat.Convert(), + Telemetry: *args.Splunk.Telemetry.Convert(), + } +} + +func (args *SplunkHecArguments) SetToDefault() { + args.SplunkHecClientArguments.SetToDefault() + args.QueueSettings = exporterhelper.NewDefaultQueueConfig() + args.RetrySettings = configretry.NewDefaultBackOffConfig() + args.Splunk.SetToDefault() +} diff --git a/internal/component/otelcol/exporter/splunkhec/config/splunkhec_test.go b/internal/component/otelcol/exporter/splunkhec/config/splunkhec_test.go new file mode 100644 index 0000000000..b226c641e2 --- /dev/null +++ b/internal/component/otelcol/exporter/splunkhec/config/splunkhec_test.go @@ -0,0 +1,219 @@ +package splunkhec_config_test + +import ( + "testing" + + splunkhec_config "github.com/grafana/alloy/internal/component/otelcol/exporter/splunkhec/config" + "github.com/grafana/alloy/syntax" + "github.com/stretchr/testify/require" +) + +func TestUnmarshalSplunkHecClientArguments(t *testing.T) { + for _, tt := range []struct { + name string + cfg string + expectErr bool + }{ + { + name: "valid client config", + cfg: ` + endpoint = "http://localhost:8088" + timeout = "10s" + insecure_skip_verify = true + max_idle_conns = 10 + `, + }, + { + name: "invalid client config", + cfg: ` + timeout = "10s" + insecure_skip_verify = true + max_idle_conns = 10 + `, + expectErr: true, + }, + } { + tt := tt + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + var sl splunkhec_config.SplunkHecClientArguments + err := syntax.Unmarshal([]byte(tt.cfg), &sl) + if tt.expectErr { + require.Error(t, err) + } else { + require.NoError(t, err) + } + }) + } +} + +func TestUnmarshalSplunkConf(t *testing.T) { + for _, tt := range []struct { + name string + cfg string + expectErr bool + }{ + { + name: "valid splunk config", + cfg: ` + log_data_enabled = true + profiling_data_enabled = true + token = "abc" + source = "def" + sourcetype = "ghi" + index = "jkl" + disable_compression = true + max_content_length_logs = 100 + max_content_length_metrics = 200 + max_content_length_traces = 300 + max_event_size = 400 + + `, + }, + { + name: "invalid splunk config - no token", + cfg: ` + log_data_enabled = true + profiling_data_enabled = true + source = "def" + sourcetype = "ghi" + index = "jkl" + disable_compression = true + max_content_length_logs = 100 + max_content_length_metrics = 200 + max_content_length_traces = 300 + max_event_size = 400 + `, + expectErr: true, + }, + { + name: "valid splunk config - nested blocks", + cfg: ` + log_data_enabled = true + profiling_data_enabled = true + source = "def" + sourcetype = "ghi" + token = "jkl" + disable_compression = true + max_content_length_logs = 100 + max_content_length_metrics = 200 + max_content_length_traces = 300 + max_event_size = 400 + heartbeat { + interval = "10s" + startup = true + } + telemetry { + enabled = true + } + + `, + }, + } { + tt := tt + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + var sl splunkhec_config.SplunkConf + err := syntax.Unmarshal([]byte(tt.cfg), &sl) + if tt.expectErr { + require.Error(t, err) + } else { + require.NoError(t, err) + } + }) + } +} + +func TestUnmarshalSplunkHecArguments(t *testing.T) { + for _, tt := range []struct { + name string + cfg string + expectErr bool + }{ + { + name: "valid splunkhec arguments", + cfg: ` + splunk { + log_data_enabled = true + profiling_data_enabled = true + token = "abc" + source = "def" + } + client { + endpoint = "http://localhost:8088" + timeout = "10s" + insecure_skip_verify = true + max_idle_conns = 10 + } + `, + }, { + name: "invalidvalid splunkhec arguments, no token", + cfg: ` + splunk { + log_data_enabled = true + profiling_data_enabled = true + source = "def" + } + client { + endpoint = "http://localhost:8088" + timeout = "10s" + insecure_skip_verify = true + max_idle_conns = 10 + } + `, + expectErr: true, + }, + { + name: "invalid splunkhec arguments, no endpoint", + cfg: ` + splunk { + log_data_enabled = true + profiling_data_enabled = true + source = "def" + token = "abc" + } + client { + timeout = "10s" + insecure_skip_verify = true + max_idle_conns = 10 + } + `, + expectErr: true, + }, + { + name: "valid splunkhec arguments, with hearthbeat config", + cfg: ` + splunk { + log_data_enabled = true + profiling_data_enabled = true + source = "def" + token = "abc" + heartbeat { + interval = "10s" + startup = true + } + } + client { + endpoint = "http://localhost:8088" + timeout = "10s" + insecure_skip_verify = true + max_idle_conns = 10 + } + `, + expectErr: false, + }, + } { + tt := tt + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + var sl splunkhec_config.SplunkHecArguments + err := syntax.Unmarshal([]byte(tt.cfg), &sl) + if tt.expectErr { + require.Error(t, err) + } else { + require.NoError(t, err) + } + }) + } + +} diff --git a/internal/component/otelcol/exporter/splunkhec/splunkhec.go b/internal/component/otelcol/exporter/splunkhec/splunkhec.go new file mode 100644 index 0000000000..2b4e385f85 --- /dev/null +++ b/internal/component/otelcol/exporter/splunkhec/splunkhec.go @@ -0,0 +1,89 @@ +// package splunkhec provides an otel.exporter splunkhec component +// Maintainers for the Grafana Alloy wrapper: +// - @adlotsof +// - @PatMis16 +package splunkhec + +import ( + "github.com/grafana/alloy/internal/component" + "github.com/grafana/alloy/internal/component/otelcol" + otelcolCfg "github.com/grafana/alloy/internal/component/otelcol/config" + "github.com/grafana/alloy/internal/component/otelcol/exporter" + splunkhec_config "github.com/grafana/alloy/internal/component/otelcol/exporter/splunkhec/config" + "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter" + otelcomponent "go.opentelemetry.io/collector/component" + otelextension "go.opentelemetry.io/collector/extension" + "go.opentelemetry.io/collector/pipeline" +) + +func init() { + component.Register(component.Registration{ + Name: "otelcol.exporter.splunkhec", + Community: true, + Args: Arguments{}, + Exports: otelcol.ConsumerExports{}, + Build: func(opts component.Options, args component.Arguments) (component.Component, error) { + fact := splunkhecexporter.NewFactory() + return exporter.New(opts, fact, args.(Arguments), exporter.TypeAll) + }, + }) +} + +type Arguments struct { + Client splunkhec_config.SplunkHecClientArguments `alloy:"client,block"` + Queue otelcol.QueueArguments `alloy:"sending_queue,block,optional"` + Retry otelcol.RetryArguments `alloy:"retry_on_failure,block,optional"` + + // Splunk specific configuration settings + Splunk splunkhec_config.SplunkConf `alloy:"splunk,block"` + + // DebugMetrics configures component internal metrics. Optional. + DebugMetrics otelcolCfg.DebugMetricsArguments `alloy:"debug_metrics,block,optional"` +} + +var _ exporter.Arguments = Arguments{} + +func (args *Arguments) SetToDefault() { + args.Queue.SetToDefault() + args.Retry.SetToDefault() + args.Client.SetToDefault() + args.Splunk.SetToDefault() + args.DebugMetrics.SetToDefault() +} + +func (args Arguments) Convert() (otelcomponent.Config, error) { + + return (&splunkhec_config.SplunkHecArguments{ + Splunk: args.Splunk, + QueueSettings: *args.Queue.Convert(), + RetrySettings: *args.Retry.Convert(), + SplunkHecClientArguments: args.Client, + }).Convert(), nil +} + +func (args *Arguments) Validate() error { + if err := args.Client.Validate(); err != nil { + return err + } + if err := args.Splunk.Validate(); err != nil { + return err + } + if err := args.Queue.Validate(); err != nil { + return err + } + + return nil +} +func (args Arguments) DebugMetricsConfig() otelcolCfg.DebugMetricsArguments { + return args.DebugMetrics +} + +// Extensions implements exporter.Arguments. +func (args Arguments) Extensions() map[otelcomponent.ID]otelextension.Extension { + return nil +} + +// Exporters implements exporter.Arguments. +func (args Arguments) Exporters() map[pipeline.Signal]map[otelcomponent.ID]otelcomponent.Component { + return nil +} diff --git a/internal/component/otelcol/exporter/splunkhec/splunkhec_test.go b/internal/component/otelcol/exporter/splunkhec/splunkhec_test.go new file mode 100644 index 0000000000..5cf6ad923c --- /dev/null +++ b/internal/component/otelcol/exporter/splunkhec/splunkhec_test.go @@ -0,0 +1,229 @@ +package splunkhec_test + +import ( + "testing" + "time" + + "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter" + + "github.com/grafana/alloy/internal/component/otelcol/exporter/splunkhec" + "github.com/grafana/alloy/syntax" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/collector/config/configauth" + "go.opentelemetry.io/collector/config/confighttp" + "go.opentelemetry.io/collector/config/configopaque" + "go.opentelemetry.io/collector/config/configretry" + "go.opentelemetry.io/collector/config/configtls" + "go.opentelemetry.io/collector/exporter/exporterbatcher" + "go.opentelemetry.io/collector/exporter/exporterhelper" +) + +func TestConfigConversion(t *testing.T) { + expectedCustomise := splunkhecexporter.Config{ + ClientConfig: confighttp.ClientConfig{ + Endpoint: "http://localhost:8088", ProxyURL: "", + TLSSetting: configtls.ClientConfig{ + Config: configtls.Config{ + CAFile: "", + CAPem: "", + IncludeSystemCACertsPool: false, + CertFile: "", + CertPem: "", + KeyFile: "", + KeyPem: "", + MinVersion: "", + MaxVersion: "", + CipherSuites: []string(nil), ReloadInterval: 0, + }, + Insecure: false, + InsecureSkipVerify: true, + ServerName: "", + }, + ReadBufferSize: 0, + WriteBufferSize: 0, + Timeout: 10000000000, + Headers: map[string]configopaque.String(nil), + Auth: (*configauth.Authentication)(nil), + Compression: "", + MaxIdleConns: (*int)(nil), + MaxIdleConnsPerHost: (*int)(nil), + MaxConnsPerHost: (*int)(nil), + IdleConnTimeout: (*time.Duration)(nil), + DisableKeepAlives: false, + HTTP2ReadIdleTimeout: 0, + HTTP2PingTimeout: 0, + Cookies: (*confighttp.CookiesConfig)(nil), + }, + QueueSettings: exporterhelper.QueueConfig{ + Enabled: true, + NumConsumers: 10, + QueueSize: 1000, + StorageID: nil, + }, + BackOffConfig: configretry.BackOffConfig{ + Enabled: true, + InitialInterval: 5000000000, + RandomizationFactor: 0.5, + Multiplier: 1.5, + MaxInterval: 30000000000, + MaxElapsedTime: 300000000000, + }, + BatcherConfig: exporterbatcher.Config{ + Enabled: false, + FlushTimeout: 200000000, + MinSizeConfig: exporterbatcher.MinSizeConfig{MinSizeItems: 8192}, + MaxSizeConfig: exporterbatcher.MaxSizeConfig{MaxSizeItems: 0}, + }, + LogDataEnabled: true, + ProfilingDataEnabled: true, + Token: "token", + Source: "source", + SourceType: "sourcetype", + Index: "index", + DisableCompression: false, + MaxContentLengthLogs: 0x200000, + MaxContentLengthMetrics: 0x200000, + MaxContentLengthTraces: 0x200000, + MaxEventSize: 0x500000, + SplunkAppName: "Alloy", + SplunkAppVersion: "", + HecFields: splunkhecexporter.OtelToHecFields{SeverityText: "", SeverityNumber: ""}, + HealthPath: "/services/collector/health", + HecHealthCheckEnabled: false, + ExportRaw: false, + UseMultiMetricFormat: false, + Heartbeat: splunkhecexporter.HecHeartbeat{Interval: 0, Startup: false}, + Telemetry: splunkhecexporter.HecTelemetry{ + Enabled: false, + OverrideMetricsNames: map[string]string(nil), + ExtraAttributes: map[string]string(nil), + }, + } + + expectedMinimal := &splunkhecexporter.Config{ + ClientConfig: confighttp.ClientConfig{ + Endpoint: "http://localhost:8088", + ProxyURL: "", + TLSSetting: configtls.ClientConfig{ + Config: configtls.Config{ + CAFile: "", + CAPem: "", + IncludeSystemCACertsPool: false, + CertFile: "", + CertPem: "", + KeyFile: "", + KeyPem: "", + MinVersion: "", + MaxVersion: "", + CipherSuites: []string(nil), + ReloadInterval: 0, + }, + Insecure: false, + InsecureSkipVerify: false, + ServerName: "", + }, ReadBufferSize: 0, + WriteBufferSize: 0, + Timeout: 15000000000, + Headers: map[string]configopaque.String(nil), + Auth: (*configauth.Authentication)(nil), + Compression: "", MaxIdleConns: (*int)(nil), + MaxIdleConnsPerHost: (*int)(nil), + MaxConnsPerHost: (*int)(nil), + IdleConnTimeout: (*time.Duration)(nil), + DisableKeepAlives: false, + HTTP2ReadIdleTimeout: 0, + HTTP2PingTimeout: 0, + Cookies: (*confighttp.CookiesConfig)(nil)}, + QueueSettings: exporterhelper.QueueConfig{ + Enabled: true, + NumConsumers: 10, + QueueSize: 1000, + StorageID: (nil), + }, + BackOffConfig: configretry.BackOffConfig{ + Enabled: true, + InitialInterval: 5000000000, + RandomizationFactor: 0.5, + Multiplier: 1.5, + MaxInterval: 30000000000, + MaxElapsedTime: 300000000000, + }, + BatcherConfig: exporterbatcher.Config{Enabled: false, + FlushTimeout: 200000000, + MinSizeConfig: exporterbatcher.MinSizeConfig{MinSizeItems: 8192}, + MaxSizeConfig: exporterbatcher.MaxSizeConfig{MaxSizeItems: 0}}, + LogDataEnabled: true, + ProfilingDataEnabled: true, + Token: "token", Source: "", + SourceType: "", Index: "", + DisableCompression: false, + MaxContentLengthLogs: 0x200000, + MaxContentLengthMetrics: 0x200000, + MaxContentLengthTraces: 0x200000, + MaxEventSize: 0x500000, + SplunkAppName: "Alloy", + HecFields: splunkhecexporter.OtelToHecFields{SeverityText: "", SeverityNumber: ""}, + HealthPath: "/services/collector/health", HecHealthCheckEnabled: false, + ExportRaw: false, + UseMultiMetricFormat: false, + Heartbeat: splunkhecexporter.HecHeartbeat{Interval: 0, Startup: false}, + Telemetry: splunkhecexporter.HecTelemetry{ + Enabled: false, + OverrideMetricsNames: map[string]string(nil), + ExtraAttributes: map[string]string(nil), + }} + + tests := []struct { + testName string + alloyCfg string + expected *splunkhecexporter.Config + }{ + { + testName: "full customise", + alloyCfg: ` + splunk { + token = "token" + source = "source" + sourcetype = "sourcetype" + index = "index" + } + client { + endpoint = "http://localhost:8088" + timeout = "10s" + insecure_skip_verify = true + } + `, + expected: &expectedCustomise, + }, + { + testName: "minimal customise", + alloyCfg: ` + splunk { + token = "token" + } + client { + endpoint = "http://localhost:8088" + } + `, + expected: expectedMinimal, + }, + } + for _, tt := range tests { + tt := tt + t.Run(tt.testName, func(t *testing.T) { + t.Parallel() + var args splunkhec.Arguments + err := syntax.Unmarshal([]byte(tt.alloyCfg), &args) + if err != nil { + t.Fatal(err) + } + + cfg, err := args.Convert() + if err != nil { + t.Fatal(err) + } + + require.Equal(t, tt.expected, cfg) + }) + } +} diff --git a/internal/converter/internal/otelcolconvert/converter_splunkhecexporter.go b/internal/converter/internal/otelcolconvert/converter_splunkhecexporter.go new file mode 100644 index 0000000000..a586267f5b --- /dev/null +++ b/internal/converter/internal/otelcolconvert/converter_splunkhecexporter.go @@ -0,0 +1,126 @@ +package otelcolconvert + +import ( + "fmt" + + "github.com/grafana/alloy/internal/component/otelcol/exporter/splunkhec" + splunkhec_config "github.com/grafana/alloy/internal/component/otelcol/exporter/splunkhec/config" + "github.com/grafana/alloy/internal/converter/diag" + "github.com/grafana/alloy/internal/converter/internal/common" + "github.com/grafana/alloy/syntax/alloytypes" + "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter" + "go.opentelemetry.io/collector/component" + "go.opentelemetry.io/collector/component/componentstatus" + "go.opentelemetry.io/collector/exporter/exporterbatcher" +) + +func init() { + converters = append(converters, splunkhecExporterConverter{}) +} + +type splunkhecExporterConverter struct{} + +func (splunkhecExporterConverter) Factory() component.Factory { return splunkhecexporter.NewFactory() } + +func (splunkhecExporterConverter) InputComponentName() string { + return "otelcol.exporter.splunkhec" +} + +func (splunkhecExporterConverter) ConvertAndAppend(state *State, id componentstatus.InstanceID, cfg component.Config) diag.Diagnostics { + var diags diag.Diagnostics + + label := state.AlloyComponentLabel() + + args := toSplunkHecExporter(cfg.(*splunkhecexporter.Config)) + block := common.NewBlockWithOverride([]string{"otelcol", "exporter", "splunkhec"}, label, args) + + diags.Add( + diag.SeverityLevelInfo, + fmt.Sprintf("Converted %s into %s", StringifyInstanceID(id), StringifyBlock(block)), + ) + + state.Body().AppendBlock(block) + return diags +} + +func toSplunkHecExporter(cfg *splunkhecexporter.Config) *splunkhec.Arguments { + return &splunkhec.Arguments{ + Client: toSplunkHecHTTPClientArguments(cfg), + Retry: toRetryArguments(cfg.BackOffConfig), + Queue: toQueueArguments(cfg.QueueSettings), + Splunk: toSplunkConfig(cfg), + DebugMetrics: common.DefaultValue[splunkhec.Arguments]().DebugMetrics, + } +} + +func toSplunkHecHTTPClientArguments(cfg *splunkhecexporter.Config) splunkhec_config.SplunkHecClientArguments { + return splunkhec_config.SplunkHecClientArguments{ + Endpoint: cfg.Endpoint, + Timeout: cfg.Timeout, + ReadBufferSize: int(cfg.ReadBufferSize), + WriteBufferSize: int(cfg.WriteBufferSize), + MaxIdleConns: cfg.MaxIdleConns, + MaxIdleConnsPerHost: cfg.MaxIdleConnsPerHost, + MaxConnsPerHost: cfg.MaxConnsPerHost, + IdleConnTimeout: cfg.IdleConnTimeout, + DisableKeepAlives: cfg.DisableKeepAlives, + InsecureSkipVerify: cfg.TLSSetting.Insecure, + } +} + +func toSplunkConfig(cfg *splunkhecexporter.Config) splunkhec_config.SplunkConf { + return splunkhec_config.SplunkConf{ + Token: alloytypes.Secret(cfg.Token.String()), + Source: cfg.Source, + SourceType: cfg.SourceType, + Index: cfg.Index, + LogDataEnabled: cfg.LogDataEnabled, + ProfilingDataEnabled: cfg.ProfilingDataEnabled, + DisableCompression: cfg.DisableCompression, + MaxContentLengthLogs: cfg.MaxContentLengthLogs, + MaxContentLengthMetrics: cfg.MaxContentLengthMetrics, + MaxContentLengthTraces: cfg.MaxContentLengthTraces, + MaxEventSize: cfg.MaxEventSize, + SplunkAppName: cfg.SplunkAppName, + SplunkAppVersion: cfg.SplunkAppVersion, + HealthPath: cfg.HealthPath, + HecHealthCheckEnabled: cfg.HecHealthCheckEnabled, + ExportRaw: cfg.ExportRaw, + UseMultiMetricFormat: cfg.UseMultiMetricFormat, + Heartbeat: toSplunkHecHeartbeat(cfg.Heartbeat), + Telemetry: toSplunkHecTelemetry(cfg.Telemetry), + BatcherConfig: toSplunkHecBatcherConfig(cfg.BatcherConfig), + HecFields: toSplunkHecFields(cfg.HecFields), + } +} + +func toSplunkHecHeartbeat(cfg splunkhecexporter.HecHeartbeat) splunkhec_config.SplunkHecHeartbeat { + return splunkhec_config.SplunkHecHeartbeat{ + Interval: cfg.Interval, + Startup: cfg.Startup, + } +} + +func toSplunkHecTelemetry(cfg splunkhecexporter.HecTelemetry) splunkhec_config.SplunkHecTelemetry { + return splunkhec_config.SplunkHecTelemetry{ + Enabled: cfg.Enabled, + OverrideMetricsNames: cfg.OverrideMetricsNames, + ExtraAttributes: cfg.ExtraAttributes, + } +} + +func toSplunkHecBatcherConfig(cfg exporterbatcher.Config) splunkhec_config.BatcherConfig { + return splunkhec_config.BatcherConfig{ + Enabled: cfg.Enabled, + FlushTimeout: cfg.FlushTimeout, + MinSizeItems: cfg.MinSizeItems, + MaxSizeItems: cfg.MaxSizeItems, + } +} + +func toSplunkHecFields(cfg splunkhecexporter.OtelToHecFields) splunkhec_config.HecFields { + return splunkhec_config.HecFields{ + SeverityText: cfg.SeverityText, + SeverityNumber: cfg.SeverityNumber, + } +} diff --git a/internal/converter/internal/otelcolconvert/testdata/splunkhec.alloy b/internal/converter/internal/otelcolconvert/testdata/splunkhec.alloy new file mode 100644 index 0000000000..3ace51fad1 --- /dev/null +++ b/internal/converter/internal/otelcolconvert/testdata/splunkhec.alloy @@ -0,0 +1,56 @@ +otelcol.receiver.otlp "default" { + grpc { + endpoint = "localhost:4317" + } + + http { + endpoint = "localhost:4318" + compression_algorithms = ["zlib"] + } + + output { + metrics = [otelcol.exporter.splunkhec.default.input] + logs = [otelcol.exporter.splunkhec.default.input] + traces = [otelcol.exporter.splunkhec.default.input] + } +} + +otelcol.exporter.splunkhec "default" { + client { + endpoint = "https://splunk:8088/services/collector" + timeout = "10s" + max_idle_conns = 200 + max_idle_conns_per_host = 200 + idle_conn_timeout = "10s" + } + + splunk { + token = "[REDACTED]" + source = "otel" + sourcetype = "otel" + index = "metrics" + splunk_app_name = "OpenTelemetry-Collector Splunk Exporter" + splunk_app_version = "v0.0.1" + + otel_to_hec_fields { + severity_text = "otel.log.severity.text" + severity_number = "otel.log.severity.number" + } + + heartbeat { + interval = "30s" + } + + telemetry { + enabled = true + override_metrics_names = { + otelcol_exporter_splunkhec_heartbeats_failed = "app_heartbeats_failed_total", + otelcol_exporter_splunkhec_heartbeats_sent = "app_heartbeats_success_total", + } + extra_attributes = { + custom_key = "custom_value", + dataset_name = "SplunkCloudBeaverStack", + } + } + } +} diff --git a/internal/converter/internal/otelcolconvert/testdata/splunkhec.yaml b/internal/converter/internal/otelcolconvert/testdata/splunkhec.yaml new file mode 100644 index 0000000000..708effd844 --- /dev/null +++ b/internal/converter/internal/otelcolconvert/testdata/splunkhec.yaml @@ -0,0 +1,62 @@ + +receivers: + otlp: + protocols: + grpc: + http: + compression_algorithms: + - "zlib" + +exporters: + splunk_hec: + # Splunk HTTP Event Collector token. + token: "00000000-0000-0000-0000-0000000000000" + # URL to a Splunk instance to send data to. + endpoint: "https://splunk:8088/services/collector" + # Optional Splunk source: https://docs.splunk.com/Splexicon:Source + source: "otel" + # Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype + sourcetype: "otel" + # Splunk index, optional name of the Splunk index targeted. + index: "metrics" + # Maximum HTTP connections to use simultaneously when sending data. Defaults to 100. + max_idle_conns: 200 + # Whether to disable gzip compression over HTTP. Defaults to false. + disable_compression: false + # HTTP timeout when sending data. Defaults to 10s. + timeout: 10s + tls: + # Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false. + insecure_skip_verify: false + # Path to the CA cert to verify the server being connected to. + ca_file: /certs/ExampleCA.crt + # Path to the TLS cert to use for client connections when TLS client auth is required. + cert_file: /certs/HECclient.crt + # Path to the TLS key to use for TLS required connections. + key_file: /certs/HECclient.key + # Application name is used to track telemetry information for Splunk App's using HEC by App name. + splunk_app_name: "OpenTelemetry-Collector Splunk Exporter" + # Application version is used to track telemetry information for Splunk App's using HEC by App version. + splunk_app_version: "v0.0.1" + heartbeat: + interval: 30s + telemetry: + enabled: true + override_metrics_names: + otelcol_exporter_splunkhec_heartbeats_sent: app_heartbeats_success_total + otelcol_exporter_splunkhec_heartbeats_failed: app_heartbeats_failed_total + extra_attributes: + dataset_name: SplunkCloudBeaverStack + custom_key: custom_value + +service: + pipelines: + metrics: + receivers: [otlp] + exporters: [splunk_hec] + traces: + receivers: [otlp] + exporters: [splunk_hec] + logs: + receivers: [otlp] + exporters: [splunk_hec] \ No newline at end of file From 9198a3ab717740046949a5b1a089e8a25d751f99 Mon Sep 17 00:00:00 2001 From: Sam DeHaan Date: Thu, 7 Nov 2024 16:42:47 -0500 Subject: [PATCH 03/32] fix: tiny typo in release notes for 1.5 (#2055) --- docs/sources/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/release-notes.md b/docs/sources/release-notes.md index b50125abf5..acb732a9da 100644 --- a/docs/sources/release-notes.md +++ b/docs/sources/release-notes.md @@ -31,7 +31,7 @@ Both components are very similar. More information can be found in the [announce ### Breaking change: Change default value of `revision` in `import.git` The default value was changed from `"HEAD"` to `"main"`. -Setting the `revision` to `"HEAD"`, `"FETCH_HEAD"`, `"ORIG_HEAD"`, `"MERGE_HEAD"` or `"CHERRY_PICK_HEAD"` is not longer allowed. +Setting the `revision` to `"HEAD"`, `"FETCH_HEAD"`, `"ORIG_HEAD"`, `"MERGE_HEAD"` or `"CHERRY_PICK_HEAD"` is no longer allowed. ## v1.4 From 395e8cd0b8909d291934da20d9d978505f6ef3fb Mon Sep 17 00:00:00 2001 From: Sam DeHaan Date: Fri, 8 Nov 2024 10:05:39 -0500 Subject: [PATCH 04/32] fix: Update postgres exporter (#2019) * Update postgres exporter * Update changelog * Use postgres exporter branch that implements exporter package * Add TODO for future maintainers --- CHANGELOG.md | 2 ++ go.mod | 4 +++- go.sum | 8 ++++---- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index db7cbf15b5..3436e24eb9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,6 +68,8 @@ v1.5.0-rc.0 ### Bugfixes +- Fixed an issue in the `prometheus.exporter.postgres` component that would leak goroutines when the target was not reachable (@dehaansa) + - Fixed a bug in `import.git` which caused a `"non-fast-forward update"` error message. (@ptodev) - Do not log error on clean shutdown of `loki.source.journal`. (@thampiotr) diff --git a/go.mod b/go.mod index 591e8b837a..2265f3d386 100644 --- a/go.mod +++ b/go.mod @@ -899,7 +899,9 @@ replace ( // https://github.com/grafana/cadvisor/tree/grafana-v0.47-noglobals github.com/google/cadvisor => github.com/grafana/cadvisor v0.0.0-20240729082359-1f04a91701e2 - github.com/prometheus-community/postgres_exporter => github.com/grafana/postgres_exporter v0.15.1-0.20240417113938-9358270470dd + // TODO(dehaansa): integrate the changes from the exporter-package-v0.15.0 branch into at least the + // grafana fork of the exporter, or completely into upstream + github.com/prometheus-community/postgres_exporter => github.com/grafana/postgres_exporter v0.15.1-0.20241105053755-e0a51174f168 // TODO(marctc): remove once this PR is merged upstream: https://github.com/prometheus/mysqld_exporter/pull/774 github.com/prometheus/mysqld_exporter => github.com/grafana/mysqld_exporter v0.12.2-0.20231005125903-364b9c41e595 diff --git a/go.sum b/go.sum index 92de4ca5eb..568a6f297c 100644 --- a/go.sum +++ b/go.sum @@ -180,8 +180,8 @@ github.com/ClickHouse/clickhouse-go v1.5.4/go.mod h1:EaI/sW7Azgz9UATzd5ZdZHRUhHg github.com/Code-Hex/go-generics-cache v1.5.1 h1:6vhZGc5M7Y/YD8cIUcY8kcuQLB4cHR7U+0KMqAA0KcU= github.com/Code-Hex/go-generics-cache v1.5.1/go.mod h1:qxcC9kRVrct9rHeiYpFWSoW1vxyillCVzX13KZG8dl4= github.com/DATA-DOG/go-sqlmock v1.4.1/go.mod h1:f/Ixk793poVmq4qj/V1dPUg2JEAKC73Q5eFN3EC/SaM= -github.com/DATA-DOG/go-sqlmock v1.5.0 h1:Shsta01QNfFxHCfpW6YH2STWB0MudeXXEWMr20OEh60= -github.com/DATA-DOG/go-sqlmock v1.5.0/go.mod h1:f/Ixk793poVmq4qj/V1dPUg2JEAKC73Q5eFN3EC/SaM= +github.com/DATA-DOG/go-sqlmock v1.5.2 h1:OcvFkGmslmlZibjAjaHm3L//6LiuBgolP7OputlJIzU= +github.com/DATA-DOG/go-sqlmock v1.5.2/go.mod h1:88MAG/4G7SMwSE3CeA0ZKzrT5CiOU3OJ+JlNzwDqpNU= github.com/DataDog/agent-payload/v5 v5.0.134 h1:h0oP3vDTOsjW1uKIZxKsCjOV/40jkY2Y+42GKAVH9ig= github.com/DataDog/agent-payload/v5 v5.0.134/go.mod h1:FgVQKmVdqdmZTbxIptqJC/l+xEzdiXsaAOs/vGAvWzs= github.com/DataDog/datadog-agent/cmd/agent/common/path v0.57.1 h1:9WpqKeK4bAc8pSb0sK4fY03bUOqhWUZdGoVh55KBaNI= @@ -1254,8 +1254,8 @@ github.com/grafana/opentelemetry-collector/processor/batchprocessor v0.0.0-20241 github.com/grafana/opentelemetry-collector/processor/batchprocessor v0.0.0-20241104164848-8ea9d0a3e17a/go.mod h1:QLQ31rGjPuMc/nGw4rL4HzQI9F0jVAPEmC342chxoqA= github.com/grafana/opentelemetry-collector/service v0.0.0-20241104164848-8ea9d0a3e17a h1:ZycgUSrrwtB2x1fMdLD88J2k8886/HvX1tYHlaOH/hg= github.com/grafana/opentelemetry-collector/service v0.0.0-20241104164848-8ea9d0a3e17a/go.mod h1:VTLnax+DjHal3q7WKQO0ITjWdfPTq2txaoNRcVXYzgE= -github.com/grafana/postgres_exporter v0.15.1-0.20240417113938-9358270470dd h1:vNHdecaOmYgSHMEQRgyzWacV++N38Jp8qLZg0RCsfFo= -github.com/grafana/postgres_exporter v0.15.1-0.20240417113938-9358270470dd/go.mod h1:kR16GJ0ZwWVQ2osW3pgtDJU1a/GXpufrwio0kLG14cg= +github.com/grafana/postgres_exporter v0.15.1-0.20241105053755-e0a51174f168 h1:I7FyVTtge/3G5YHVOMDG0l4If6W+kXbFDqtzj5gCSGs= +github.com/grafana/postgres_exporter v0.15.1-0.20241105053755-e0a51174f168/go.mod h1:dMrETGkSetWByp2XGsm8g6pRVh/ibnrDxKsN4BqnGNg= github.com/grafana/prometheus v1.8.2-0.20240514135907-13889ba362e6 h1:kih3d3M3dxAmrpFLvnIxFzWx8KMQyKxQwKgWP67C/Fg= github.com/grafana/prometheus v1.8.2-0.20240514135907-13889ba362e6/go.mod h1:yv4MwOn3yHMQ6MZGHPg/U7Fcyqf+rxqiZfSur6myVtc= github.com/grafana/pyroscope-go/godeltaprof v0.1.8 h1:iwOtYXeeVSAeYefJNaxDytgjKtUuKQbJqgAIjlnicKg= From e6f887d3f0b39bda46a8303fa2092e635a1749b6 Mon Sep 17 00:00:00 2001 From: yoyosir Date: Fri, 8 Nov 2024 10:46:00 -0500 Subject: [PATCH 05/32] Support kubernetes_role argument for prometheus.operator.servicemonitors (#2023) * Support kubernetes_role argument for prometheus.operator.servicemonitors * Update docs/sources/reference/components/prometheus/prometheus.operator.servicemonitors.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Use DefaultArguments to handle default * Fix doc * Update docs/sources/reference/components/prometheus/prometheus.operator.servicemonitors.md Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> * Add validation --------- Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> --- CHANGELOG.md | 2 + .../prometheus.operator.servicemonitors.md | 9 ++- .../prometheus/operator/common/crdmanager.go | 3 +- .../configgen/config_gen_servicemonitor.go | 13 ++- .../config_gen_servicemonitor_test.go | 80 ++++++++++++++++++- .../component/prometheus/operator/types.go | 8 ++ 6 files changed, 105 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3436e24eb9..0c8019e348 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,6 +39,8 @@ v1.5.0-rc.0 - Add support to `loki.source.api` to be able to extract the tenant from the HTTP `X-Scope-OrgID` header (@QuentinBisson) +- Add support to `prometheus.operator.servicemonitors` to allow `endpointslice` role. (@yoyosir) + - (_Experimental_) Add a `loki.secretfilter` component to redact secrets from collected logs. - Add `otelcol.exporter.splunkhec` allowing to export otel data to Splunk HEC (@adlotsof) diff --git a/docs/sources/reference/components/prometheus/prometheus.operator.servicemonitors.md b/docs/sources/reference/components/prometheus/prometheus.operator.servicemonitors.md index c5182b7171..67b2fcc4d3 100644 --- a/docs/sources/reference/components/prometheus/prometheus.operator.servicemonitors.md +++ b/docs/sources/reference/components/prometheus/prometheus.operator.servicemonitors.md @@ -32,10 +32,11 @@ prometheus.operator.servicemonitors "LABEL" { The following arguments are supported: -Name | Type | Description | Default | Required ----- | ---- | ----------- | ------- | -------- -`forward_to` | `list(MetricsReceiver)` | List of receivers to send scraped metrics to. | | yes -`namespaces` | `list(string)` | List of namespaces to search for ServiceMonitor resources. If not specified, all namespaces will be searched. || no +Name | Type | Description | Default | Required +---- | ---- |----------------------------------------------------------------------------------------------------------------------------|-----------| -------- +`forward_to` | `list(MetricsReceiver)` | List of receivers to send scraped metrics to. | | yes +`namespaces` | `list(string)` | List of namespaces to search for ServiceMonitor resources. If not specified, all namespaces will be searched. | | no +`kubernetes_role` | `string` | The Kubernetes role used for discovery. Supports `endpoints` or `endpointslice`. | `endpoints` | no ## Blocks diff --git a/internal/component/prometheus/operator/common/crdmanager.go b/internal/component/prometheus/operator/common/crdmanager.go index f5c13da577..e24b99159b 100644 --- a/internal/component/prometheus/operator/common/crdmanager.go +++ b/internal/component/prometheus/operator/common/crdmanager.go @@ -4,6 +4,7 @@ import ( "context" "errors" "fmt" + promk8s "github.com/prometheus/prometheus/discovery/kubernetes" "sort" "strings" "sync" @@ -486,7 +487,7 @@ func (c *crdManager) addServiceMonitor(sm *promopv1.ServiceMonitor) { mapKeys := []string{} for i, ep := range sm.Spec.Endpoints { var scrapeConfig *config.ScrapeConfig - scrapeConfig, err = gen.GenerateServiceMonitorConfig(sm, ep, i) + scrapeConfig, err = gen.GenerateServiceMonitorConfig(sm, ep, i, promk8s.Role(c.args.KubernetesRole)) if err != nil { // TODO(jcreixell): Generate Kubernetes event to inform of this error when running `kubectl get `. level.Error(c.logger).Log("name", sm.Name, "err", err, "msg", "error generating scrapeconfig from serviceMonitor") diff --git a/internal/component/prometheus/operator/configgen/config_gen_servicemonitor.go b/internal/component/prometheus/operator/configgen/config_gen_servicemonitor.go index 5893f07646..542f3c8ab7 100644 --- a/internal/component/prometheus/operator/configgen/config_gen_servicemonitor.go +++ b/internal/component/prometheus/operator/configgen/config_gen_servicemonitor.go @@ -16,7 +16,7 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) -func (cg *ConfigGenerator) GenerateServiceMonitorConfig(m *promopv1.ServiceMonitor, ep promopv1.Endpoint, i int) (cfg *config.ScrapeConfig, err error) { +func (cg *ConfigGenerator) GenerateServiceMonitorConfig(m *promopv1.ServiceMonitor, ep promopv1.Endpoint, i int, role promk8s.Role) (cfg *config.ScrapeConfig, err error) { cfg = cg.generateDefaultScrapeConfig() cfg.JobName = fmt.Sprintf("serviceMonitor/%s/%s/%d", m.Namespace, m.Name, i) @@ -24,7 +24,7 @@ func (cg *ConfigGenerator) GenerateServiceMonitorConfig(m *promopv1.ServiceMonit if ep.HonorTimestamps != nil { cfg.HonorTimestamps = *ep.HonorTimestamps } - dConfig := cg.generateK8SSDConfig(m.Spec.NamespaceSelector, m.Namespace, promk8s.RoleEndpoint, m.Spec.AttachMetadata) + dConfig := cg.generateK8SSDConfig(m.Spec.NamespaceSelector, m.Namespace, role, m.Spec.AttachMetadata) cfg.ServiceDiscoveryConfigs = append(cfg.ServiceDiscoveryConfigs, dConfig) if ep.Interval != "" { @@ -153,6 +153,10 @@ func (cg *ConfigGenerator) GenerateServiceMonitorConfig(m *promopv1.ServiceMonit } } + labelPortName := "__meta_kubernetes_endpoint_port_name" + if role == promk8s.RoleEndpointSlice { + labelPortName = "__meta_kubernetes_endpointslice_port_name" + } // Filter targets based on correct port for the endpoint. if ep.Port != "" { regex, err := relabel.NewRegexp(ep.Port) @@ -160,7 +164,7 @@ func (cg *ConfigGenerator) GenerateServiceMonitorConfig(m *promopv1.ServiceMonit return nil, fmt.Errorf("parsing Port as regex: %w", err) } relabels.add(&relabel.Config{ - SourceLabels: model.LabelNames{"__meta_kubernetes_endpoint_port_name"}, + SourceLabels: model.LabelNames{model.LabelName(labelPortName)}, Action: "keep", Regex: regex, }) @@ -191,6 +195,9 @@ func (cg *ConfigGenerator) GenerateServiceMonitorConfig(m *promopv1.ServiceMonit } sourceLabels := model.LabelNames{"__meta_kubernetes_endpoint_address_target_kind", "__meta_kubernetes_endpoint_address_target_name"} + if role == promk8s.RoleEndpointSlice { + sourceLabels = model.LabelNames{"__meta_kubernetes_endpointslice_address_target_kind", "__meta_kubernetes_endpointslice_address_target_name"} + } // Relabel namespace and pod and service labels into proper labels. // Relabel node labels with meta labels available with Prometheus >= v2.3. relabels.add(&relabel.Config{ diff --git a/internal/component/prometheus/operator/configgen/config_gen_servicemonitor_test.go b/internal/component/prometheus/operator/configgen/config_gen_servicemonitor_test.go index 17d4791abf..2b772b3e3a 100644 --- a/internal/component/prometheus/operator/configgen/config_gen_servicemonitor_test.go +++ b/internal/component/prometheus/operator/configgen/config_gen_servicemonitor_test.go @@ -32,6 +32,7 @@ func TestGenerateServiceMonitorConfig(t *testing.T) { name string m *promopv1.ServiceMonitor ep promopv1.Endpoint + role promk8s.Role expectedRelabels string expectedMetricRelabels string expected *config.ScrapeConfig @@ -44,7 +45,8 @@ func TestGenerateServiceMonitorConfig(t *testing.T) { Name: "svcmonitor", }, }, - ep: promopv1.Endpoint{}, + ep: promopv1.Endpoint{}, + role: promk8s.RoleEndpoint, expectedRelabels: util.Untab(` - target_label: __meta_foo replacement: bar @@ -110,6 +112,7 @@ func TestGenerateServiceMonitorConfig(t *testing.T) { ep: promopv1.Endpoint{ TargetPort: &intstr.IntOrString{StrVal: "http_metrics", Type: intstr.String}, }, + role: promk8s.RoleEndpoint, expectedRelabels: util.Untab(` - target_label: __meta_foo replacement: bar @@ -180,6 +183,7 @@ func TestGenerateServiceMonitorConfig(t *testing.T) { ep: promopv1.Endpoint{ TargetPort: &intstr.IntOrString{IntVal: 4242, Type: intstr.Int}, }, + role: promk8s.RoleEndpoint, expectedRelabels: util.Untab(` - target_label: __meta_foo replacement: bar @@ -239,6 +243,77 @@ func TestGenerateServiceMonitorConfig(t *testing.T) { }, }, }, + { + name: "role_endpointslice", + m: &promopv1.ServiceMonitor{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "operator", + Name: "svcmonitor", + }, + }, + ep: promopv1.Endpoint{ + TargetPort: &intstr.IntOrString{IntVal: 4242, Type: intstr.Int}, + }, + role: promk8s.RoleEndpointSlice, + expectedRelabels: util.Untab(` + - target_label: __meta_foo + replacement: bar + - source_labels: [job] + target_label: __tmp_prometheus_job_name + - source_labels: ["__meta_kubernetes_pod_container_port_number"] + regex: "4242" + action: "keep" + - source_labels: [__meta_kubernetes_endpointslice_address_target_kind, __meta_kubernetes_endpointslice_address_target_name] + regex: Node;(.*) + target_label: node + replacement: ${1} + - source_labels: [__meta_kubernetes_endpointslice_address_target_kind, __meta_kubernetes_endpointslice_address_target_name] + regex: Pod;(.*) + target_label: pod + action: replace + replacement: ${1} + - source_labels: [__meta_kubernetes_namespace] + target_label: namespace + - source_labels: [__meta_kubernetes_service_name] + target_label: service + - source_labels: [__meta_kubernetes_pod_container_name] + target_label: container + - source_labels: [__meta_kubernetes_pod_name] + target_label: pod + - source_labels: [__meta_kubernetes_pod_phase] + regex: (Failed|Succeeded) + action: drop + - source_labels: [__meta_kubernetes_service_name] + target_label: job + replacement: ${1} + - target_label: endpoint + replacement: "4242" + `), + expected: &config.ScrapeConfig{ + JobName: "serviceMonitor/operator/svcmonitor/1", + HonorTimestamps: true, + ScrapeInterval: model.Duration(time.Minute), + ScrapeTimeout: model.Duration(10 * time.Second), + ScrapeProtocols: config.DefaultScrapeProtocols, + EnableCompression: true, + MetricsPath: "/metrics", + Scheme: "http", + HTTPClientConfig: commonConfig.HTTPClientConfig{ + FollowRedirects: true, + EnableHTTP2: true, + }, + ServiceDiscoveryConfigs: discovery.Configs{ + &promk8s.SDConfig{ + Role: "endpointslice", + + NamespaceDiscovery: promk8s.NamespaceDiscovery{ + IncludeOwnNamespace: false, + Names: []string{"operator"}, + }, + }, + }, + }, + }, { name: "everything", m: &promopv1.ServiceMonitor{ @@ -308,6 +383,7 @@ func TestGenerateServiceMonitorConfig(t *testing.T) { }, }, }, + role: promk8s.RoleEndpoint, expectedRelabels: util.Untab(` - target_label: __meta_foo replacement: bar @@ -427,7 +503,7 @@ func TestGenerateServiceMonitorConfig(t *testing.T) { {TargetLabel: "__meta_foo", Replacement: "bar"}, }, } - cfg, err := cg.GenerateServiceMonitorConfig(tc.m, tc.ep, 1) + cfg, err := cg.GenerateServiceMonitorConfig(tc.m, tc.ep, 1, tc.role) require.NoError(t, err) // check relabel configs separately rlcs := cfg.RelabelConfigs diff --git a/internal/component/prometheus/operator/types.go b/internal/component/prometheus/operator/types.go index 8ed8cc4149..ed5be3d6e0 100644 --- a/internal/component/prometheus/operator/types.go +++ b/internal/component/prometheus/operator/types.go @@ -1,6 +1,8 @@ package operator import ( + "fmt" + promk8s "github.com/prometheus/prometheus/discovery/kubernetes" "time" "github.com/grafana/alloy/internal/component/common/config" @@ -24,6 +26,8 @@ type Arguments struct { // Namespaces to search for monitor resources. Empty implies All namespaces Namespaces []string `alloy:"namespaces,attr,optional"` + KubernetesRole string `alloy:"kubernetes_role,attr,optional"` + // LabelSelector allows filtering discovered monitor resources by labels LabelSelector *config.LabelSelector `alloy:"selector,block,optional"` @@ -54,6 +58,7 @@ var DefaultArguments = Arguments{ Client: kubernetes.ClientArguments{ HTTPClientConfig: config.DefaultHTTPClientConfig, }, + KubernetesRole: string(promk8s.RoleEndpoint), } // SetToDefault implements syntax.Defaulter. @@ -66,6 +71,9 @@ func (args *Arguments) Validate() error { if len(args.Namespaces) == 0 { args.Namespaces = []string{apiv1.NamespaceAll} } + if args.KubernetesRole != string(promk8s.RoleEndpointSlice) && args.KubernetesRole != string(promk8s.RoleEndpoint) { + return fmt.Errorf("only endpoints and endpointslice are supported") + } return nil } From f3108e71c5ce44b125e80d1ce0bd26ef997e66bc Mon Sep 17 00:00:00 2001 From: Paulin Todev Date: Mon, 11 Nov 2024 11:35:26 +0000 Subject: [PATCH 06/32] Prototype targets.merge function (array.combine_maps) (#1826) * Prototype map.inner_join function * Rename to "targets.merge", remove unnecessary params * Add failure tests * Delete incorrect comment * rename targets.merge to array.combine_maps, mark it as experimental, make is more permissive and add tests * update changelog * Update CHANGELOG.md Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> * Update docs/sources/reference/stdlib/array.md Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> * Update docs/sources/reference/stdlib/array.md Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> * Update docs/sources/reference/stdlib/array.md Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> * rename mapCombine to combineMaps * document panic * add equal test * add more tests * Update syntax/internal/stdlib/stdlib.go Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> * Update syntax/internal/stdlib/stdlib.go Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> * add examples in doc * fix error propagation * remove value nul on len function * refactor code into a traversal function * update doc to avoid modifying the experimental shared doc --------- Co-authored-by: William Dumont Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> --- CHANGELOG.md | 3 + docs/sources/reference/stdlib/array.md | 51 ++++++ .../controller/component_references.go | 24 ++- .../runtime/internal/controller/loader.go | 2 +- syntax/internal/stdlib/stdlib.go | 145 +++++++++++++++++- syntax/internal/value/value.go | 15 +- syntax/internal/value/value_test.go | 6 + syntax/vm/error.go | 3 + syntax/vm/vm.go | 6 + syntax/vm/vm_stdlib_test.go | 133 ++++++++++++++++ 10 files changed, 383 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c8019e348..b2b15938fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,6 +33,7 @@ v1.5.0-rc.0 ### Features - Add the function `path_join` to the stdlib. (@wildum) + - Add `pyroscope.receive_http` component to receive and forward Pyroscope profiles (@marcsanmi) - Add support to `loki.source.syslog` for the RFC3164 format ("BSD syslog"). (@sushain97) @@ -47,6 +48,8 @@ v1.5.0-rc.0 - (_Experimental_) Add a `prometheus.write.queue` component to add an alternative to `prometheus.remote_write` which allowing the writing of metrics to a prometheus endpoint. (@mattdurham) +- (_Experimental_) Add the `arrary.combine_maps` function to the stdlib. (@ptodev, @wildum) + ### Enhancements - The `mimir.rules.kubernetes` component now supports adding extra label matchers diff --git a/docs/sources/reference/stdlib/array.md b/docs/sources/reference/stdlib/array.md index 482cc60e65..b9fb947bdc 100644 --- a/docs/sources/reference/stdlib/array.md +++ b/docs/sources/reference/stdlib/array.md @@ -32,3 +32,54 @@ Elements within the list can be any type. > array.concat([[1, 2], [3, 4]], [[5, 6]]) [[1, 2], [3, 4], [5, 6]] ``` + +## array.combine_maps + +> **EXPERIMENTAL**: This is an [experimental][] feature. Experimental +> features are subject to frequent breaking changes, and may be removed with +> no equivalent replacement. The `stability.level` flag must be set to `experimental` +> to use the feature. + +The `array.combine_maps` function allows you to join two arrays of maps if certain keys have matching values in both maps. It's particularly useful when combining labels of targets coming from different `prometheus.discovery.*` or `prometheus.exporter.*` components. +It takes three arguments: + +* The first two arguments are a of type `list(map(string))`. The keys of the map are strings. + The value for each key could be of any Alloy type such as a `string`, `integer`, `map`, or a `capsule`. +* The third input is an `array` containing strings. The strings are the keys whose value has to match for maps to be combined. + +The maps that don't contain all the keys provided in the third argument will be discarded. When maps are combined and both contain the same keys, the last value from the second argument will be used. + +Pseudo function code: +``` +for every map in arg1: + for every map in arg2: + if the condition key matches in both: + merge maps and add to result +``` + +### Examples + +```alloy +> array.combine_maps([{"instance"="1.1.1.1", "team"="A"}], [{"instance"="1.1.1.1", "cluster"="prod"}], ["instance"]) +[{"instance"="1.1.1.1", "team"="A", "cluster"="prod"}] + +// Second map overrides the team in the first map +> array.combine_maps([{"instance"="1.1.1.1", "team"="A"}], [{"instance"="1.1.1.1", "team"="B"}], ["instance"]) +[{"instance"="1.1.1.1", "team"="B"}] + +// If multiple maps from the first argument match with multiple maps from the second argument, different combinations will be created. +> array.combine_maps([{"instance"="1.1.1.1", "team"="A"}, {"instance"="1.1.1.1", "team"="B"}], [{"instance"="1.1.1.1", "cluster"="prod"}, {"instance"="1.1.1.1", "cluster"="ops"}], ["instance"]) +[{"instance"="1.1.1.1", "team"="A", "cluster"="prod"}, {"instance"="1.1.1.1", "team"="A", "cluster"="ops"}, {"instance"="1.1.1.1", "team"="B", "cluster"="prod"}, {"instance"="1.1.1.1", "team"="B", "cluster"="ops"}] +``` + +Examples using discovery and exporter components: +```alloy +> array.combine_maps(discovery.kubernetes.k8s_pods.targets, prometheus.exporter.postgres, ["instance"]) + +> array.combine_maps(prometheus.exporter.redis.default.targets, [{"instance"="1.1.1.1", "testLabelKey" = "testLabelVal"}], ["instance"]) +``` + +You can find more examples in the [tests][]. + +[tests]: https://github.com/grafana/alloy/blob/main/syntax/vm/vm_stdlib_test.go +[experimental]: https://grafana.com/docs/release-life-cycle/ \ No newline at end of file diff --git a/internal/runtime/internal/controller/component_references.go b/internal/runtime/internal/controller/component_references.go index cc5205dfdc..5754878042 100644 --- a/internal/runtime/internal/controller/component_references.go +++ b/internal/runtime/internal/controller/component_references.go @@ -5,6 +5,7 @@ import ( "strings" "github.com/go-kit/log" + "github.com/grafana/alloy/internal/featuregate" "github.com/grafana/alloy/internal/runtime/internal/dag" "github.com/grafana/alloy/internal/runtime/logging/level" "github.com/grafana/alloy/syntax/ast" @@ -18,6 +19,17 @@ import ( // will be (field_a, field_b, field_c). type Traversal []*ast.Ident +// String returns a dot-separated string representation of the field names in the traversal. +// For example, a traversal of fields [field_a, field_b, field_c] returns "field_a.field_b.field_c". +// Returns an empty string if the traversal contains no fields. +func (t Traversal) String() string { + var fieldNames []string + for _, field := range t { + fieldNames = append(fieldNames, field.Name) + } + return strings.Join(fieldNames, ".") +} + // Reference describes an Alloy expression reference to a BlockNode. type Reference struct { Target BlockNode // BlockNode being referenced @@ -29,7 +41,7 @@ type Reference struct { // ComponentReferences returns the list of references a component is making to // other components. -func ComponentReferences(cn dag.Node, g *dag.Graph, l log.Logger, scope *vm.Scope) ([]Reference, diag.Diagnostics) { +func ComponentReferences(cn dag.Node, g *dag.Graph, l log.Logger, scope *vm.Scope, minStability featuregate.Stability) ([]Reference, diag.Diagnostics) { var ( traversals []Traversal @@ -63,6 +75,16 @@ func ComponentReferences(cn dag.Node, g *dag.Graph, l log.Logger, scope *vm.Scop refs = append(refs, ref) } else if scope.IsStdlibDeprecated(t[0].Name) { level.Warn(l).Log("msg", "this stdlib function is deprecated; please refer to the documentation for updated usage and alternatives", "function", t[0].Name) + } else if funcName := t.String(); scope.IsStdlibExperimental(funcName) { + if err := featuregate.CheckAllowed(featuregate.StabilityExperimental, minStability, funcName); err != nil { + diags = append(diags, diag.Diagnostic{ + Severity: diag.SeverityLevelError, + Message: err.Error(), + StartPos: ast.StartPos(t[0]).Position(), + EndPos: ast.StartPos(t[len(t)-1]).Position(), + }) + continue + } } } diff --git a/internal/runtime/internal/controller/loader.go b/internal/runtime/internal/controller/loader.go index 8cbe0061fe..fae75f5865 100644 --- a/internal/runtime/internal/controller/loader.go +++ b/internal/runtime/internal/controller/loader.go @@ -615,7 +615,7 @@ func (l *Loader) wireGraphEdges(g *dag.Graph) diag.Diagnostics { // Finally, wire component references. l.cache.mut.RLock() - refs, nodeDiags := ComponentReferences(n, g, l.log, l.cache.scope) + refs, nodeDiags := ComponentReferences(n, g, l.log, l.cache.scope, l.globals.MinStability) l.cache.mut.RUnlock() for _, ref := range refs { g.AddEdge(dag.Edge{From: n, To: ref.Target}) diff --git a/syntax/internal/stdlib/stdlib.go b/syntax/internal/stdlib/stdlib.go index 995e538d4b..85dadd9c67 100644 --- a/syntax/internal/stdlib/stdlib.go +++ b/syntax/internal/stdlib/stdlib.go @@ -17,7 +17,15 @@ import ( "gopkg.in/yaml.v3" ) -// There identifiers are deprecated in favour of the namespaced ones. +// TODO: refactor the stdlib to have consistent naming between namespaces and identifiers. + +// ExperimentalIdentifiers contains the full name (namespace + identifier's name) of stdlib +// identifiers that are considered "experimental". +var ExperimentalIdentifiers = map[string]bool{ + "array.combine_maps": true, +} + +// These identifiers are deprecated in favour of the namespaced ones. var DeprecatedIdentifiers = map[string]interface{}{ "env": os.Getenv, "nonsensitive": nonSensitive, @@ -86,7 +94,8 @@ var str = map[string]interface{}{ } var array = map[string]interface{}{ - "concat": concat, + "concat": concat, + "combine_maps": combineMaps, } var convert = map[string]interface{}{ @@ -146,6 +155,138 @@ var concat = value.RawFunction(func(funcValue value.Value, args ...value.Value) return value.Array(raw...), nil }) +// This function assumes that the types of the value.Value objects are correct. +func shouldJoin(left value.Value, right value.Value, conditions value.Value) (bool, error) { + for i := 0; i < conditions.Len(); i++ { + condition := conditions.Index(i).Text() + + leftVal, ok := left.Key(condition) + if !ok { + return false, nil + } + + rightVal, ok := right.Key(condition) + if !ok { + return false, nil + } + + if !leftVal.Equal(rightVal) { + return false, nil + } + } + return true, nil +} + +// Merge two maps. +// If a key exists in both maps, the value from the right map will be used. +func concatMaps(left, right value.Value) (value.Value, error) { + res := make(map[string]value.Value) + + for _, key := range left.Keys() { + val, ok := left.Key(key) + if !ok { + return value.Null, fmt.Errorf("concatMaps: key %s not found in left map while iterating - this should never happen", key) + } + res[key] = val + } + + for _, key := range right.Keys() { + val, ok := right.Key(key) + if !ok { + return value.Null, fmt.Errorf("concatMaps: key %s not found in right map while iterating - this should never happen", key) + } + res[key] = val + } + + return value.Object(res), nil +} + +// Inputs: +// args[0]: []map[string]string: lhs array +// args[1]: []map[string]string: rhs array +// args[2]: []string: merge conditions +var combineMaps = value.RawFunction(func(funcValue value.Value, args ...value.Value) (value.Value, error) { + if len(args) != 3 { + return value.Value{}, fmt.Errorf("combine_maps: expected 3 arguments, got %d", len(args)) + } + + // Validate args[0] and args[1] + for i := range []int{0, 1} { + if args[i].Type() != value.TypeArray { + return value.Null, value.ArgError{ + Function: funcValue, + Argument: args[i], + Index: i, + Inner: value.TypeError{ + Value: args[i], + Expected: value.TypeArray, + }, + } + } + for j := 0; j < args[i].Len(); j++ { + if args[i].Index(j).Type() != value.TypeObject { + return value.Null, value.ArgError{ + Function: funcValue, + Argument: args[i].Index(j), + Index: j, + Inner: value.TypeError{ + Value: args[i].Index(j), + Expected: value.TypeObject, + }, + } + } + } + } + + // Validate args[2] + if args[2].Type() != value.TypeArray { + return value.Null, value.ArgError{ + Function: funcValue, + Argument: args[2], + Index: 2, + Inner: value.TypeError{ + Value: args[2], + Expected: value.TypeArray, + }, + } + } + if args[2].Len() == 0 { + return value.Null, value.ArgError{ + Function: funcValue, + Argument: args[2], + Index: 2, + Inner: fmt.Errorf("combine_maps: merge conditions must not be empty"), + } + } + + // We cannot preallocate the size of the result array, because we don't know + // how well the merge is going to go. If none of the merge conditions are met, + // the result array will be empty. + res := []value.Value{} + + for i := 0; i < args[0].Len(); i++ { + for j := 0; j < args[1].Len(); j++ { + left := args[0].Index(i) + right := args[1].Index(j) + + join, err := shouldJoin(left, right, args[2]) + if err != nil { + return value.Null, err + } + + if join { + val, err := concatMaps(left, right) + if err != nil { + return value.Null, err + } + res = append(res, val) + } + } + } + + return value.Array(res...), nil +}) + func jsonDecode(in string) (interface{}, error) { var res interface{} err := json.Unmarshal([]byte(in), &res) diff --git a/syntax/internal/value/value.go b/syntax/internal/value/value.go index 3c8554b88c..829449370e 100644 --- a/syntax/internal/value/value.go +++ b/syntax/internal/value/value.go @@ -396,7 +396,7 @@ func (v Value) Key(key string) (index Value, ok bool) { // // An ArgError will be returned if one of the arguments is invalid. An Error // will be returned if the function call returns an error or if the number of -// arguments doesn't match. +// arguments doesn't match func (v Value) Call(args ...Value) (Value, error) { if v.ty != TypeFunction { panic("syntax/value: Call called on non-function type") @@ -553,3 +553,16 @@ func convertGoNumber(nval Number, target reflect.Type) reflect.Value { panic("unsupported number conversion") } + +// Equal will result in panic if the values are funcs, maps or slices +func (v Value) Equal(rhs Value) bool { + if v.Type() != rhs.Type() { + return false + } + + if !v.rv.Equal(rhs.rv) { + return false + } + + return true +} diff --git a/syntax/internal/value/value_test.go b/syntax/internal/value/value_test.go index fbebcabdd7..ec52f192c9 100644 --- a/syntax/internal/value/value_test.go +++ b/syntax/internal/value/value_test.go @@ -140,6 +140,12 @@ func TestValue_Call(t *testing.T) { require.Equal(t, int64(15+43), res.Int()) }) + t.Run("equal - string", func(t *testing.T) { + v := value.String("aa") + w := value.String("aa") + require.True(t, v.Equal(w)) + }) + t.Run("fully variadic", func(t *testing.T) { add := func(nums ...int) int { var sum int diff --git a/syntax/vm/error.go b/syntax/vm/error.go index 7f3ada3d5a..38d9528a5e 100644 --- a/syntax/vm/error.go +++ b/syntax/vm/error.go @@ -44,6 +44,9 @@ func makeDiagnostic(err error, assoc map[value.Value]ast.Node) error { case value.FieldError: fmt.Fprintf(&expr, ".%s", ne.Field) val = ne.Value + case value.ArgError: + message = ne.Inner.Error() + val = ne.Argument } cause = val diff --git a/syntax/vm/vm.go b/syntax/vm/vm.go index 71b2893ccc..f8d0e44341 100644 --- a/syntax/vm/vm.go +++ b/syntax/vm/vm.go @@ -509,3 +509,9 @@ func (s *Scope) IsStdlibDeprecated(name string) bool { _, exist := stdlib.DeprecatedIdentifiers[name] return exist } + +// IsStdlibExperimental returns true if the scoped identifier is experimental. +func (s *Scope) IsStdlibExperimental(fullName string) bool { + _, exist := stdlib.ExperimentalIdentifiers[fullName] + return exist +} diff --git a/syntax/vm/vm_stdlib_test.go b/syntax/vm/vm_stdlib_test.go index e8009ae157..36a74c29ab 100644 --- a/syntax/vm/vm_stdlib_test.go +++ b/syntax/vm/vm_stdlib_test.go @@ -39,6 +39,99 @@ func TestVM_Stdlib(t *testing.T) { {"encoding.from_yaml nil field", "encoding.from_yaml(`foo: null`)", map[string]interface{}{"foo": nil}}, {"encoding.from_yaml nil array element", `encoding.from_yaml("[0, null]")`, []interface{}{0, nil}}, {"encoding.from_base64", `encoding.from_base64("Zm9vYmFyMTIzIT8kKiYoKSctPUB+")`, string(`foobar123!?$*&()'-=@~`)}, + + // Map tests + { + // Basic case. No conflicting key/val pairs. + "array.combine_maps", + `array.combine_maps([{"a" = "a1", "b" = "b1"}], [{"a" = "a1", "c" = "c1"}], ["a"])`, + []map[string]interface{}{{"a": "a1", "b": "b1", "c": "c1"}}, + }, + { + // The first array has 2 maps, each with the same key/val pairs. + "array.combine_maps", + `array.combine_maps([{"a" = "a1", "b" = "b1"}, {"a" = "a1", "b" = "b1"}], [{"a" = "a1", "c" = "c1"}], ["a"])`, + []map[string]interface{}{{"a": "a1", "b": "b1", "c": "c1"}, {"a": "a1", "b": "b1", "c": "c1"}}, + }, + { + // Non-unique merge criteria. + "array.combine_maps", + `array.combine_maps([{"pod" = "a", "lbl" = "q"}, {"pod" = "b", "lbl" = "q"}], [{"pod" = "c", "lbl" = "q"}, {"pod" = "d", "lbl" = "q"}], ["lbl"])`, + []map[string]interface{}{{"lbl": "q", "pod": "c"}, {"lbl": "q", "pod": "d"}, {"lbl": "q", "pod": "c"}, {"lbl": "q", "pod": "d"}}, + }, + { + // Basic case. Integer and string values. + "array.combine_maps", + `array.combine_maps([{"a" = 1, "b" = 2.2}], [{"a" = 1, "c" = "c1"}], ["a"])`, + []map[string]interface{}{{"a": 1, "b": 2.2, "c": "c1"}}, + }, + { + // The second map will override a value from the first. + "array.combine_maps", + `array.combine_maps([{"a" = 1, "b" = 2.2}], [{"a" = 1, "b" = "3.3"}], ["a"])`, + []map[string]interface{}{{"a": 1, "b": "3.3"}}, + }, + { + // Not enough matches for a join. + "array.combine_maps", + `array.combine_maps([{"a" = 1, "b" = 2.2}], [{"a" = 2, "b" = "3.3"}], ["a"])`, + []map[string]interface{}{}, + }, + { + // Not enough matches for a join. + // The "a" value has differing types. + "array.combine_maps", + `array.combine_maps([{"a" = 1, "b" = 2.2}], [{"a" = "1", "b" = "3.3"}], ["a"])`, + []map[string]interface{}{}, + }, + { + // Basic case. Some values are arrays and maps. + "array.combine_maps", + `array.combine_maps([{"a" = 1, "b" = [1,2,3]}], [{"a" = 1, "c" = {"d" = {"e" = 10}}}], ["a"])`, + []map[string]interface{}{{"a": 1, "b": []interface{}{1, 2, 3}, "c": map[string]interface{}{"d": map[string]interface{}{"e": 10}}}}, + }, + { + // Join key not present in ARG2 + "array.combine_maps", + `array.combine_maps([{"a" = 1, "n" = 1.1}], [{"a" = 1, "n" = 2.1}, {"n" = 2.2}], ["a"])`, + []map[string]interface{}{{"a": 1, "n": 2.1}}, + }, + { + // Join key not present in ARG1 + "array.combine_maps", + `array.combine_maps([{"a" = 1, "n" = 1.1}, {"n" = 1.2}], [{"a" = 1, "n" = 2.1}], ["a"])`, + []map[string]interface{}{{"a": 1, "n": 2.1}}, + }, + { + // Join with multiple keys + "array.combine_maps", + `array.combine_maps([{"a" = 1, "b" = 3, "n" = 1.1}], [{"a" = 1, "b" = 3, "n" = 2.1}], ["a", "b"])`, + []map[string]interface{}{{"a": 1, "b": 3, "n": 2.1}}, + }, + { + // Join with multiple keys + // Some maps don't match all keys + "array.combine_maps", + `array.combine_maps([{"a" = 1, "n" = 1.1}, {"a" = 1, "b" = 3, "n" = 1.1}, {"b" = 3, "n" = 1.1}], [{"a" = 1, "n" = 2.3}, {"b" = 1, "n" = 2.3}, {"a" = 1, "b" = 3, "n" = 2.1}], ["a", "b"])`, + []map[string]interface{}{{"a": 1, "b": 3, "n": 2.1}}, + }, + { + // Join with multiple keys + // No match because one key is missing + "array.combine_maps", + `array.combine_maps([{"a" = 1, "n" = 1.1}, {"a" = 1, "b" = 3, "n" = 1.1}, {"b" = 3, "n" = 1.1}], [{"a" = 1, "n" = 2.3}, {"b" = 1, "n" = 2.3}, {"a" = 1, "b" = 3, "n" = 2.1}], ["a", "b", "c"])`, + []map[string]interface{}{}, + }, + { + // Multi match ends up with len(ARG1) * len(ARG2) maps + "array.combine_maps", + `array.combine_maps([{"a" = 1, "n" = 1.1}, {"a" = 1, "n" = 1.2}, {"a" = 1, "n" = 1.3}], [{"a" = 1, "n" = 2.1}, {"a" = 1, "n" = 2.2}, {"a" = 1, "n" = 2.3}], ["a"])`, + []map[string]interface{}{ + {"a": 1, "n": 2.1}, {"a": 1, "n": 2.2}, {"a": 1, "n": 2.3}, + {"a": 1, "n": 2.1}, {"a": 1, "n": 2.2}, {"a": 1, "n": 2.3}, + {"a": 1, "n": 2.1}, {"a": 1, "n": 2.2}, {"a": 1, "n": 2.3}, + }, + }, } for _, tc := range tt { @@ -55,6 +148,46 @@ func TestVM_Stdlib(t *testing.T) { } } +func TestVM_Stdlib_Errors(t *testing.T) { + tt := []struct { + name string + input string + expectedErr string + }{ + // Map tests + { + // Error: invalid RHS type - string. + "array.combine_maps", + `array.combine_maps([{"a" = "a1", "b" = "b1"}], "a", ["a"])`, + `"a" should be array, got string`, + }, + { + // Error: invalid RHS type - an array with strings. + "array.combine_maps", + `array.combine_maps([{"a" = "a1", "b" = "b1"}], ["a"], ["a"])`, + `"a" should be object, got string`, + }, + { + "array.combine_maps", + `array.combine_maps([{"a" = "a1", "b" = "b1"}], [{"a" = "a1", "c" = "b1"}], [])`, + `combine_maps: merge conditions must not be empty`, + }, + } + + for _, tc := range tt { + t.Run(tc.name, func(t *testing.T) { + expr, err := parser.ParseExpression(tc.input) + require.NoError(t, err) + + eval := vm.New(expr) + + rv := reflect.New(reflect.TypeOf([]map[string]interface{}{})) + err = eval.Evaluate(nil, rv.Interface()) + require.ErrorContains(t, err, tc.expectedErr) + }) + } +} + func TestStdlibCoalesce(t *testing.T) { t.Setenv("TEST_VAR2", "Hello!") From 3cf2bcd92d7dcd5ef90ae962542e3cedefa10990 Mon Sep 17 00:00:00 2001 From: Sam DeHaan Date: Mon, 11 Nov 2024 07:33:01 -0500 Subject: [PATCH 07/32] Implement an initial version of the support bundle in Alloy (#2009) * Implement an initial version of the support bundle in Alloy * Add documentation for support bundle * Update changelog * Update docs/sources/troubleshoot/support_bundle.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/troubleshoot/support_bundle.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/troubleshoot/support_bundle.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/troubleshoot/support_bundle.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Initial PR feedback * Rewrite http service to use logging library internal to alloy * Revert accidental commit of e2e test changes * Fix comment on exported function * Clean up added host variable that is no longer used * Refactor usage of logger in http service * Update internal/service/http/http.go Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> * implement PR feedback * Hide support bundle behind public preview stability level * Update docs based on feedback * Update docs/sources/troubleshoot/support_bundle.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/troubleshoot/support_bundle.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/troubleshoot/support_bundle.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/troubleshoot/support_bundle.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/troubleshoot/support_bundle.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/troubleshoot/support_bundle.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * More PR feedback in docs * Fix race condition in logger * Add a note about backward-compatibility exception --------- Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> --- CHANGELOG.md | 2 + docs/sources/reference/cli/run.md | 2 + docs/sources/troubleshoot/support_bundle.md | 51 +++++ go.mod | 2 + go.sum | 2 + internal/alloycli/cmd_run.go | 23 ++- internal/runtime/logging/logger.go | 71 ++++++- internal/service/http/http.go | 106 ++++++++-- internal/service/http/http_test.go | 2 +- internal/service/http/supportbundle.go | 211 ++++++++++++++++++++ 10 files changed, 445 insertions(+), 27 deletions(-) create mode 100644 docs/sources/troubleshoot/support_bundle.md create mode 100644 internal/service/http/supportbundle.go diff --git a/CHANGELOG.md b/CHANGELOG.md index b2b15938fb..27e9b9873f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,8 @@ v1.5.0-rc.0 ### Features +- Add support bundle generation via the API endpoint /-/support (@dehaansa) + - Add the function `path_join` to the stdlib. (@wildum) - Add `pyroscope.receive_http` component to receive and forward Pyroscope profiles (@marcsanmi) diff --git a/docs/sources/reference/cli/run.md b/docs/sources/reference/cli/run.md index f246bb894e..c154696260 100644 --- a/docs/sources/reference/cli/run.md +++ b/docs/sources/reference/cli/run.md @@ -42,6 +42,7 @@ The following flags are supported: * `--server.http.ui-path-prefix`: Base path where the UI is exposed (default `/`). * `--storage.path`: Base directory where components can store data (default `data-alloy/`). * `--disable-reporting`: Disable [data collection][] (default `false`). +* `--disable-support-bundle`: Disable [support bundle][] endpoint (default `false`). * `--cluster.enabled`: Start {{< param "PRODUCT_NAME" >}} in clustered mode (default `false`). * `--cluster.node-name`: The name to use for this node (defaults to the environment's hostname). * `--cluster.join-addresses`: Comma-separated list of addresses to join the cluster at (default `""`). Mutually exclusive with `--cluster.discover-peers`. @@ -178,6 +179,7 @@ Refer to [alloy convert][] for more details on how `extra-args` work. [go-discover]: https://github.com/hashicorp/go-discover [in-memory HTTP traffic]: ../../../get-started/component_controller/#in-memory-traffic [data collection]: ../../../data-collection/ +[support bundle]: ../../../troubleshoot/support_bundle [components]: ../../get-started/components/ [component controller]: ../../../get-started/component_controller/ [UI]: ../../../troubleshoot/debug/#clustering-page diff --git a/docs/sources/troubleshoot/support_bundle.md b/docs/sources/troubleshoot/support_bundle.md new file mode 100644 index 0000000000..2bb870bc5b --- /dev/null +++ b/docs/sources/troubleshoot/support_bundle.md @@ -0,0 +1,51 @@ +--- +canonical: https://grafana.com/docs/alloy/latest/troubleshoot/support_bundle/ +description: Learn how to generate a support bundle +title: Generate a support bundle +menuTitle: Generate a support bundle +weight: 300 +--- + +Public preview + +# Generate a support bundle + +{{< docs/public-preview product="Generate support bundle" >}} + +The `/-/support?duration=N` endpoint returns a support bundle, a zip file that contains information +about a running {{< param "PRODUCT_NAME" >}} instance, and can be used as a baseline of information when trying +to debug an issue. + +This feature is not covered by our [backward-compatibility][backward-compatibility] guarantees. + +{{< admonition type="note" >}} +This endpoint is enabled by default, but may be disabled using the `--disable-support-bundle` runtime flag. +{{< /admonition >}} + +The duration parameter is optional, must be less than or equal to the +configured HTTP server write timeout, and if not provided, defaults to it. +The endpoint is only exposed to the {{< param "PRODUCT_NAME" >}} HTTP server listen address, which +defaults to `localhost:12345`. + +The support bundle contains all information in plain text, so you can +inspect it before sharing to verify that no sensitive information has leaked. + +In addition, you can inspect the [supportbundle implementation](https://github.com/grafana/alloy/tree/internal/service/http/supportbundle.go) +to verify the code used to generate these bundles. + +A support bundle contains the following data: +* `alloy-components.json` contains information about the [components][components] running on this {{< param "PRODUCT_NAME" >}} instance, generated by the +`/api/v0/web/components` endpoint. +* `alloy-logs.txt` contains the logs during the bundle generation. +* `alloy-metadata.yaml` contains the {{< param "PRODUCT_NAME" >}} build version and the installation's operating system, architecture, and uptime. +* `alloy-metrics.txt` contains a snapshot of the internal metrics for {{< param "PRODUCT_NAME" >}}. +* `alloy-peers.json` contains information about the identified cluster peers of this {{< param "PRODUCT_NAME" >}} instance, generated by the +`/api/v0/web/peers` endpoint. +* `alloy-runtime-flags.txt` contains the values of the runtime flags available in {{< param "PRODUCT_NAME" >}}. +* The `pprof/` directory contains Go runtime profiling data (CPU, heap, goroutine, mutex, block profiles) as exported by the pprof package. +Refer to the [profile][profile] documentation for more details on how to use this information. + +[profile]: ../profile +[components]: ../../get-started/components/ +[alloy-repo]: https://github.com/grafana/alloy/issues +[backward-compatibility]: ../../introduction/backward-compatibility \ No newline at end of file diff --git a/go.mod b/go.mod index 2265f3d386..1f1e516f04 100644 --- a/go.mod +++ b/go.mod @@ -845,6 +845,8 @@ require ( go.opentelemetry.io/otel/exporters/stdout/stdoutlog v0.7.0 // indirect ) +require github.com/mackerelio/go-osstat v0.2.5 + // NOTE: replace directives below must always be *temporary*. // // Adding a replace directive to change a module to a fork of a module will diff --git a/go.sum b/go.sum index 568a6f297c..5aa02756c4 100644 --- a/go.sum +++ b/go.sum @@ -1702,6 +1702,8 @@ github.com/lufia/plan9stats v0.0.0-20220913051719-115f729f3c8c h1:VtwQ41oftZwlMn github.com/lufia/plan9stats v0.0.0-20220913051719-115f729f3c8c/go.mod h1:JKx41uQRwqlTZabZc+kILPrO/3jlKnQ2Z8b7YiVw5cE= github.com/lyft/protoc-gen-validate v0.0.0-20180911180927-64fcb82c878e/go.mod h1:XbGvPuh87YZc5TdIa2/I4pLk0QoUACkjt2znoq26NVQ= github.com/lyft/protoc-gen-validate v0.0.13/go.mod h1:XbGvPuh87YZc5TdIa2/I4pLk0QoUACkjt2znoq26NVQ= +github.com/mackerelio/go-osstat v0.2.5 h1:+MqTbZUhoIt4m8qzkVoXUJg1EuifwlAJSk4Yl2GXh+o= +github.com/mackerelio/go-osstat v0.2.5/go.mod h1:atxwWF+POUZcdtR1wnsUcQxTytoHG4uhl2AKKzrOajY= github.com/magefile/mage v1.15.0 h1:BvGheCMAsG3bWUDbZ8AyXXpCNwU9u5CB6sM+HNb9HYg= github.com/magefile/mage v1.15.0/go.mod h1:z5UZb/iS3GoOSn0JgWuiw7dxlurVYTu+/jHXqQg881A= github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= diff --git a/internal/alloycli/cmd_run.go b/internal/alloycli/cmd_run.go index 33f9798eb7..306a030c55 100644 --- a/internal/alloycli/cmd_run.go +++ b/internal/alloycli/cmd_run.go @@ -23,6 +23,7 @@ import ( "github.com/grafana/ckit/peer" "github.com/prometheus/client_golang/prometheus" "github.com/spf13/cobra" + "github.com/spf13/pflag" "go.opentelemetry.io/otel" "golang.org/x/exp/maps" @@ -64,6 +65,7 @@ func runCommand() *cobra.Command { clusterAdvInterfaces: advertise.DefaultInterfaces, clusterMaxJoinPeers: 5, clusterRejoinInterval: 60 * time.Second, + disableSupportBundle: false, } cmd := &cobra.Command{ @@ -100,7 +102,7 @@ depending on the nature of the reload error. SilenceUsage: true, RunE: func(cmd *cobra.Command, args []string) error { - return r.Run(args[0]) + return r.Run(cmd, args[0]) }, } @@ -111,6 +113,8 @@ depending on the nature of the reload error. cmd.Flags().StringVar(&r.uiPrefix, "server.http.ui-path-prefix", r.uiPrefix, "Prefix to serve the HTTP UI at") cmd.Flags(). BoolVar(&r.enablePprof, "server.http.enable-pprof", r.enablePprof, "Enable /debug/pprof profiling endpoints.") + cmd.Flags(). + BoolVar(&r.disableSupportBundle, "server.http.disable-support-bundle", r.disableSupportBundle, "Disable /-/support support bundle retrieval.") // Cluster flags cmd.Flags(). @@ -184,9 +188,10 @@ type alloyRun struct { configBypassConversionErrors bool configExtraArgs string enableCommunityComps bool + disableSupportBundle bool } -func (fr *alloyRun) Run(configPath string) error { +func (fr *alloyRun) Run(cmd *cobra.Command, configPath string) error { var wg sync.WaitGroup defer wg.Wait() @@ -275,8 +280,15 @@ func (fr *alloyRun) Run(configPath string) error { return err } + runtimeFlags := []string{} + if !fr.disableSupportBundle { + cmd.Flags().VisitAll(func(f *pflag.Flag) { + runtimeFlags = append(runtimeFlags, fmt.Sprintf("%s=%s", f.Name, f.Value.String())) + }) + } + httpService := httpservice.New(httpservice.Options{ - Logger: log.With(l, "service", "http"), + Logger: l, Tracer: t, Gatherer: prometheus.DefaultGatherer, @@ -286,6 +298,11 @@ func (fr *alloyRun) Run(configPath string) error { HTTPListenAddr: fr.httpListenAddr, MemoryListenAddr: fr.inMemoryAddr, EnablePProf: fr.enablePprof, + MinStability: fr.minStability, + BundleContext: httpservice.SupportBundleContext{ + RuntimeFlags: runtimeFlags, + DisableSupportBundle: fr.disableSupportBundle, + }, }) remoteCfgService, err := remotecfgservice.New(remotecfgservice.Options{ diff --git a/internal/runtime/logging/logger.go b/internal/runtime/logging/logger.go index 5bc2402772..4e1b5bd5a3 100644 --- a/internal/runtime/logging/logger.go +++ b/internal/runtime/logging/logger.go @@ -55,6 +55,12 @@ func New(w io.Writer, o Options) (*Logger, error) { return l, nil } +// NewNop returns a logger that does nothing +func NewNop() *Logger { + l, _ := NewDeferred(io.Discard) + return l +} + // NewDeferred creates a new logger with the default log level and format. // The logger is not updated during initialization. func NewDeferred(w io.Writer) (*Logger, error) { @@ -63,7 +69,6 @@ func NewDeferred(w io.Writer) (*Logger, error) { format formatVar writer writerVar ) - l := &Logger{ inner: w, @@ -104,11 +109,10 @@ func (l *Logger) Update(o Options) error { l.level.Set(slogLevel(o.Level).Level()) l.format.Set(o.Format) - newWriter := l.inner + l.writer.SetInnerWriter(l.inner) if len(o.WriteTo) > 0 { - newWriter = io.MultiWriter(l.inner, &lokiWriter{o.WriteTo}) + l.writer.SetLokiWriter(&lokiWriter{o.WriteTo}) } - l.writer.Set(newWriter) // Build all our deferred handlers if l.deferredSlog != nil { @@ -133,6 +137,14 @@ func (l *Logger) Update(o Options) error { return nil } +func (l *Logger) SetTemporaryWriter(w io.Writer) { + l.writer.SetTemporaryWriter(w) +} + +func (l *Logger) RemoveTemporaryWriter() { + l.writer.RemoveTemporaryWriter() +} + // Log implements log.Logger. func (l *Logger) Log(kvps ...interface{}) error { // Buffer logs before confirming log format is configured in `logging` block @@ -215,24 +227,63 @@ func (f *formatVar) Set(format Format) { type writerVar struct { mut sync.RWMutex - w io.Writer + + lokiWriter *lokiWriter + innerWriter io.Writer + tmpWriter io.Writer } -func (w *writerVar) Set(inner io.Writer) { +func (w *writerVar) SetTemporaryWriter(writer io.Writer) { w.mut.Lock() defer w.mut.Unlock() - w.w = inner + w.tmpWriter = writer } -func (w *writerVar) Write(p []byte) (n int, err error) { +func (w *writerVar) RemoveTemporaryWriter() { + w.mut.Lock() + defer w.mut.Unlock() + w.tmpWriter = nil +} + +func (w *writerVar) SetInnerWriter(writer io.Writer) { + w.mut.Lock() + defer w.mut.Unlock() + w.innerWriter = writer +} + +func (w *writerVar) SetLokiWriter(writer *lokiWriter) { + w.mut.Lock() + defer w.mut.Unlock() + w.lokiWriter = writer +} + +func (w *writerVar) Write(p []byte) (int, error) { w.mut.RLock() defer w.mut.RUnlock() - if w.w == nil { + if w.innerWriter == nil { return 0, fmt.Errorf("no writer available") } - return w.w.Write(p) + // The following is effectively an io.Multiwriter, but without updating + // the Multiwriter each time tmpWriter is added or removed. + if _, err := w.innerWriter.Write(p); err != nil { + return 0, err + } + + if w.lokiWriter != nil { + if _, err := w.lokiWriter.Write(p); err != nil { + return 0, err + } + } + + if w.tmpWriter != nil { + if _, err := w.tmpWriter.Write(p); err != nil { + return 0, err + } + } + + return len(p), nil } type bufferedItem struct { diff --git a/internal/service/http/http.go b/internal/service/http/http.go index ff070b330b..590802b9b4 100644 --- a/internal/service/http/http.go +++ b/internal/service/http/http.go @@ -2,6 +2,7 @@ package http import ( + "bytes" "context" "crypto/tls" "fmt" @@ -11,14 +12,17 @@ import ( "os" "path" "sort" + "strconv" "strings" "sync" + "time" "github.com/go-kit/log" "github.com/gorilla/mux" "github.com/grafana/alloy/internal/component" "github.com/grafana/alloy/internal/featuregate" alloy_runtime "github.com/grafana/alloy/internal/runtime" + "github.com/grafana/alloy/internal/runtime/logging" "github.com/grafana/alloy/internal/runtime/logging/level" "github.com/grafana/alloy/internal/service" "github.com/grafana/alloy/internal/service/remotecfg" @@ -40,16 +44,18 @@ const ServiceName = "http" // Options are used to configure the HTTP service. Options are constant for the // lifetime of the HTTP service. type Options struct { - Logger log.Logger // Where to send logs. + Logger *logging.Logger // Where to send logs. Tracer trace.TracerProvider // Where to send traces. Gatherer prometheus.Gatherer // Where to collect metrics from. ReadyFunc func() bool ReloadFunc func() (*alloy_runtime.Source, error) - HTTPListenAddr string // Address to listen for HTTP traffic on. - MemoryListenAddr string // Address to accept in-memory traffic on. - EnablePProf bool // Whether pprof endpoints should be exposed. + HTTPListenAddr string // Address to listen for HTTP traffic on. + MemoryListenAddr string // Address to accept in-memory traffic on. + EnablePProf bool // Whether pprof endpoints should be exposed. + MinStability featuregate.Stability // Minimum stability level to utilize for feature gates + BundleContext SupportBundleContext // Context for delivering a support bundle } // Arguments holds runtime settings for the HTTP service. @@ -58,14 +64,20 @@ type Arguments struct { } type Service struct { - log log.Logger - tracer trace.TracerProvider - gatherer prometheus.Gatherer - opts Options + // globalLogger allows us to leverage the logging struct for setting a temporary + // logger for support bundle usage and still leverage log.With for logging in the service + globalLogger *logging.Logger + log log.Logger + tracer trace.TracerProvider + gatherer prometheus.Gatherer + opts Options winMut sync.Mutex win *server.WinCertStoreHandler + // Used to enforce single-flight requests to supportHandler + supportBundleMut sync.Mutex + // publicLis and tcpLis are used to lazily enable TLS, since TLS is // optionally configurable at runtime. // @@ -95,7 +107,7 @@ func New(opts Options) *Service { ) if l == nil { - l = log.NewNopLogger() + l = logging.NewNop() } if t == nil { t = noop.NewTracerProvider() @@ -113,10 +125,11 @@ func New(opts Options) *Service { _ = publicLis.SetInner(tcpLis) return &Service{ - log: l, - tracer: t, - gatherer: r, - opts: opts, + globalLogger: l, + log: log.With(l, "service", "http"), + tracer: t, + gatherer: r, + opts: opts, publicLis: publicLis, tcpLis: tcpLis, @@ -211,6 +224,9 @@ func (s *Service) Run(ctx context.Context, host service.Host) error { }).Methods(http.MethodGet, http.MethodPost) } + // Wire in support bundle generator + r.HandleFunc("/-/support", s.supportHandler).Methods("GET") + // Wire custom service handlers for services which depend on the http // service. // @@ -243,6 +259,70 @@ func (s *Service) Run(ctx context.Context, host service.Host) error { return nil } +func (s *Service) supportHandler(rw http.ResponseWriter, r *http.Request) { + s.supportBundleMut.Lock() + defer s.supportBundleMut.Unlock() + + // TODO(dehaansa) remove this check once the support bundle is generally available + if !s.opts.MinStability.Permits(featuregate.StabilityPublicPreview) { + rw.WriteHeader(http.StatusForbidden) + _, _ = rw.Write([]byte("support bundle generation is only available in public preview. Use" + + " --stability.level command-line flag to enable public-preview features")) + return + } + + if s.opts.BundleContext.DisableSupportBundle { + rw.WriteHeader(http.StatusForbidden) + _, _ = rw.Write([]byte("support bundle generation is disabled; it can be re-enabled by removing the --disable-support-bundle flag")) + return + } + + duration := getServerWriteTimeout(r) + if r.URL.Query().Has("duration") { + d, err := strconv.Atoi(r.URL.Query().Get("duration")) + if err != nil { + http.Error(rw, fmt.Sprintf("duration value (in seconds) should be a positive integer: %s", err), http.StatusBadRequest) + return + } + if d < 1 { + http.Error(rw, "duration value (in seconds) should be larger than 1", http.StatusBadRequest) + return + } + if float64(d) > duration.Seconds() { + http.Error(rw, "duration value exceeds the server's write timeout", http.StatusBadRequest) + return + } + duration = time.Duration(d) * time.Second + } + ctx, cancel := context.WithTimeout(context.Background(), duration) + defer cancel() + + var logsBuffer bytes.Buffer + syncBuff := log.NewSyncWriter(&logsBuffer) + s.globalLogger.SetTemporaryWriter(syncBuff) + defer func() { + s.globalLogger.RemoveTemporaryWriter() + }() + + bundle, err := ExportSupportBundle(ctx, s.opts.BundleContext.RuntimeFlags, s.opts.HTTPListenAddr, s.Data().(Data).DialFunc) + if err != nil { + http.Error(rw, err.Error(), http.StatusInternalServerError) + return + } + if err := ServeSupportBundle(rw, bundle, &logsBuffer); err != nil { + http.Error(rw, err.Error(), http.StatusInternalServerError) + return + } +} + +func getServerWriteTimeout(r *http.Request) time.Duration { + srv, ok := r.Context().Value(http.ServerContextKey).(*http.Server) + if ok && srv.WriteTimeout != 0 { + return srv.WriteTimeout + } + return 30 * time.Second +} + // getServiceRoutes returns a sorted list of service routes for services which // depend on the HTTP service. // diff --git a/internal/service/http/http_test.go b/internal/service/http/http_test.go index bfc212471d..2481fcd6cb 100644 --- a/internal/service/http/http_test.go +++ b/internal/service/http/http_test.go @@ -169,7 +169,7 @@ func newTestEnvironment(t *testing.T) (*testEnvironment, error) { } svc := New(Options{ - Logger: util.TestLogger(t), + Logger: util.TestAlloyLogger(t), Tracer: noop.NewTracerProvider(), Gatherer: prometheus.NewRegistry(), diff --git a/internal/service/http/supportbundle.go b/internal/service/http/supportbundle.go new file mode 100644 index 0000000000..3c75c35150 --- /dev/null +++ b/internal/service/http/supportbundle.go @@ -0,0 +1,211 @@ +package http + +import ( + "archive/zip" + "bytes" + "context" + "fmt" + "io" + "net/http" + "path/filepath" + "runtime" + "runtime/pprof" + "strings" + "time" + + "github.com/grafana/alloy/internal/build" + "github.com/grafana/alloy/internal/static/server" + "github.com/mackerelio/go-osstat/uptime" + "gopkg.in/yaml.v3" +) + +// SupportBundleContext groups the relevant context that is used in the HTTP +// service config for the support bundle +type SupportBundleContext struct { + DisableSupportBundle bool // Whether support bundle endpoint should be disabled. + RuntimeFlags []string // Alloy runtime flags to send with support bundle +} + +// Bundle collects all the data that is exposed as a support bundle. +type Bundle struct { + meta []byte + alloyMetrics []byte + components []byte + peers []byte + runtimeFlags []byte + heapBuf *bytes.Buffer + goroutineBuf *bytes.Buffer + blockBuf *bytes.Buffer + mutexBuf *bytes.Buffer + cpuBuf *bytes.Buffer +} + +// Metadata contains general runtime information about the current Alloy environment. +type Metadata struct { + BuildVersion string `yaml:"build_version"` + OS string `yaml:"os"` + Architecture string `yaml:"architecture"` + Uptime float64 `yaml:"uptime"` +} + +// ExportSupportBundle gathers the information required for the support bundle. +func ExportSupportBundle(ctx context.Context, runtimeFlags []string, srvAddress string, dialContext server.DialContextFunc) (*Bundle, error) { + // The block profiler is disabled by default. Temporarily enable recording + // of all blocking events. Also, temporarily record all mutex contentions, + // and defer restoring of earlier mutex profiling fraction. + runtime.SetBlockProfileRate(1) + old := runtime.SetMutexProfileFraction(1) + defer func() { + runtime.SetBlockProfileRate(0) + runtime.SetMutexProfileFraction(old) + }() + + // Gather runtime metadata. + ut, err := uptime.Get() + if err != nil { + return nil, err + } + m := Metadata{ + BuildVersion: build.Version, + OS: runtime.GOOS, + Architecture: runtime.GOARCH, + Uptime: ut.Seconds(), + } + meta, err := yaml.Marshal(m) + if err != nil { + return nil, fmt.Errorf("failed to marshal support bundle metadata: %s", err) + } + + var httpClient http.Client + httpClient.Transport = &http.Transport{DialContext: dialContext} + // Gather Alloy's own metrics. + alloyMetrics, err := retrieveAPIEndpoint(httpClient, srvAddress, "metrics") + if err != nil { + return nil, fmt.Errorf("failed to get internal Alloy metrics: %s", err) + } + // Gather running component configuration + components, err := retrieveAPIEndpoint(httpClient, srvAddress, "api/v0/web/components") + if err != nil { + return nil, fmt.Errorf("failed to get component details: %s", err) + } + // Gather cluster peers information + peers, err := retrieveAPIEndpoint(httpClient, srvAddress, "api/v0/web/peers") + if err != nil { + return nil, fmt.Errorf("failed to get peer details: %s", err) + } + + // Export pprof data. + var ( + cpuBuf bytes.Buffer + heapBuf bytes.Buffer + goroutineBuf bytes.Buffer + blockBuf bytes.Buffer + mutexBuf bytes.Buffer + ) + err = pprof.StartCPUProfile(&cpuBuf) + if err != nil { + return nil, err + } + deadline, _ := ctx.Deadline() + // Sleep for the remaining of the context deadline, but leave some time for + // the rest of the bundle to be exported successfully. + time.Sleep(time.Until(deadline) - 200*time.Millisecond) + pprof.StopCPUProfile() + + p := pprof.Lookup("heap") + if err := p.WriteTo(&heapBuf, 0); err != nil { + return nil, err + } + p = pprof.Lookup("goroutine") + if err := p.WriteTo(&goroutineBuf, 0); err != nil { + return nil, err + } + p = pprof.Lookup("block") + if err := p.WriteTo(&blockBuf, 0); err != nil { + return nil, err + } + p = pprof.Lookup("mutex") + if err := p.WriteTo(&mutexBuf, 0); err != nil { + return nil, err + } + + // Finally, bundle everything up to be served, either as a zip from + // memory, or exported to a directory. + bundle := &Bundle{ + meta: meta, + alloyMetrics: alloyMetrics, + components: components, + peers: peers, + runtimeFlags: []byte(strings.Join(runtimeFlags, "\n")), + heapBuf: &heapBuf, + goroutineBuf: &goroutineBuf, + blockBuf: &blockBuf, + mutexBuf: &mutexBuf, + cpuBuf: &cpuBuf, + } + + return bundle, nil +} + +func retrieveAPIEndpoint(httpClient http.Client, srvAddress, endpoint string) ([]byte, error) { + url := fmt.Sprintf("http://%s/%s", srvAddress, endpoint) + resp, err := httpClient.Get(url) + if err != nil { + return nil, err + } + res, err := io.ReadAll(resp.Body) + if err != nil { + return nil, err + } + + return res, nil +} + +// ServeSupportBundle the collected data and logs as a zip file over the given +// http.ResponseWriter. +func ServeSupportBundle(rw http.ResponseWriter, b *Bundle, logsBuf *bytes.Buffer) error { + zw := zip.NewWriter(rw) + rw.Header().Set("Content-Type", "application/zip") + rw.Header().Set("Content-Disposition", "attachment; filename=\"alloy-support-bundle.zip\"") + + zipStructure := map[string][]byte{ + "alloy-metadata.yaml": b.meta, + "alloy-components.json": b.components, + "alloy-peers.json": b.peers, + "alloy-metrics.txt": b.alloyMetrics, + "alloy-runtime-flags.txt": b.runtimeFlags, + "alloy-logs.txt": logsBuf.Bytes(), + "pprof/cpu.pprof": b.cpuBuf.Bytes(), + "pprof/heap.pprof": b.heapBuf.Bytes(), + "pprof/goroutine.pprof": b.goroutineBuf.Bytes(), + "pprof/mutex.pprof": b.mutexBuf.Bytes(), + "pprof/block.pprof": b.blockBuf.Bytes(), + } + + for fn, b := range zipStructure { + if b != nil { + path := append([]string{"alloy-support-bundle"}, strings.Split(fn, "/")...) + if err := writeByteSlice(zw, b, path...); err != nil { + return err + } + } + } + + err := zw.Close() + if err != nil { + return fmt.Errorf("failed to flush the zip writer: %v", err) + } + return nil +} + +func writeByteSlice(zw *zip.Writer, b []byte, fn ...string) error { + f, err := zw.Create(filepath.Join(fn...)) + if err != nil { + return err + } + _, err = f.Write(b) + if err != nil { + return err + } + return nil +} From 1bbd2a5a90a8006b16130d2b48222531a972fcf8 Mon Sep 17 00:00:00 2001 From: Piotr <17101802+thampiotr@users.noreply.github.com> Date: Mon, 11 Nov 2024 13:12:58 +0000 Subject: [PATCH 08/32] fix changelog items (#2060) --- CHANGELOG.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 27e9b9873f..482d29c936 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,7 +14,15 @@ Main (unreleased) - Add `add_cloudwatch_timestamp` to `prometheus.exporter.cloudwatch` metrics. (@captncraig) -v1.5.0-rc.0 +- Add support to `prometheus.operator.servicemonitors` to allow `endpointslice` role. (@yoyosir) + +- Add `otelcol.exporter.splunkhec` allowing to export otel data to Splunk HEC (@adlotsof) + +### Bugfixes + +- Fixed an issue in the `prometheus.exporter.postgres` component that would leak goroutines when the target was not reachable (@dehaansa) + +v1.5.0-rc.1 ----------------- ### Breaking changes @@ -42,10 +50,7 @@ v1.5.0-rc.0 - Add support to `loki.source.api` to be able to extract the tenant from the HTTP `X-Scope-OrgID` header (@QuentinBisson) -- Add support to `prometheus.operator.servicemonitors` to allow `endpointslice` role. (@yoyosir) - - (_Experimental_) Add a `loki.secretfilter` component to redact secrets from collected logs. -- Add `otelcol.exporter.splunkhec` allowing to export otel data to Splunk HEC (@adlotsof) - (_Experimental_) Add a `prometheus.write.queue` component to add an alternative to `prometheus.remote_write` which allowing the writing of metrics to a prometheus endpoint. (@mattdurham) @@ -75,8 +80,6 @@ v1.5.0-rc.0 ### Bugfixes -- Fixed an issue in the `prometheus.exporter.postgres` component that would leak goroutines when the target was not reachable (@dehaansa) - - Fixed a bug in `import.git` which caused a `"non-fast-forward update"` error message. (@ptodev) - Do not log error on clean shutdown of `loki.source.journal`. (@thampiotr) From a539919c23e8006e23ad61f05f6f89ee2a95ba58 Mon Sep 17 00:00:00 2001 From: Piotr <17101802+thampiotr@users.noreply.github.com> Date: Mon, 11 Nov 2024 14:24:50 +0000 Subject: [PATCH 09/32] fix typo (#2062) --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 482d29c936..91f22faedd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -55,7 +55,7 @@ v1.5.0-rc.1 - (_Experimental_) Add a `prometheus.write.queue` component to add an alternative to `prometheus.remote_write` which allowing the writing of metrics to a prometheus endpoint. (@mattdurham) -- (_Experimental_) Add the `arrary.combine_maps` function to the stdlib. (@ptodev, @wildum) +- (_Experimental_) Add the `array.combine_maps` function to the stdlib. (@ptodev, @wildum) ### Enhancements From 894942088144652b057d15a2aa182e69fc79f0fc Mon Sep 17 00:00:00 2001 From: Piotr <17101802+thampiotr@users.noreply.github.com> Date: Wed, 13 Nov 2024 11:33:05 +0000 Subject: [PATCH 10/32] Prep main for v1.5.0 (#2077) --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 91f22faedd..28fc106bc5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,7 @@ Main (unreleased) - Fixed an issue in the `prometheus.exporter.postgres` component that would leak goroutines when the target was not reachable (@dehaansa) -v1.5.0-rc.1 +v1.5.0 ----------------- ### Breaking changes From a682502b8b4f31e85ad878f9d50fd3de931b5054 Mon Sep 17 00:00:00 2001 From: Piotr <17101802+thampiotr@users.noreply.github.com> Date: Wed, 13 Nov 2024 13:43:30 +0000 Subject: [PATCH 11/32] Upgrade helm chart for v1.5.0 (#2080) --- operations/helm/charts/alloy/CHANGELOG.md | 5 +++++ operations/helm/charts/alloy/Chart.yaml | 4 ++-- operations/helm/charts/alloy/README.md | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../clustering/alloy/templates/controllers/statefulset.yaml | 2 +- .../alloy/templates/controllers/deployment.yaml | 2 +- .../alloy/templates/controllers/deployment.yaml | 2 +- .../alloy/templates/controllers/statefulset.yaml | 2 +- .../alloy/templates/controllers/statefulset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/deployment.yaml | 2 +- .../alloy/templates/controllers/deployment.yaml | 2 +- .../alloy/templates/controllers/statefulset.yaml | 2 +- .../alloy/templates/controllers/statefulset.yaml | 2 +- .../custom-config/alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../tests/envFrom/alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../extra-env/alloy/templates/controllers/daemonset.yaml | 2 +- .../extra-ports/alloy/templates/controllers/daemonset.yaml | 2 +- .../faro-ingress/alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../host-alias/alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/deployment.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../tests/nonroot/alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/daemonset.yaml | 2 +- .../sidecars/alloy/templates/controllers/daemonset.yaml | 2 +- .../alloy/templates/controllers/deployment.yaml | 2 +- .../alloy/templates/controllers/deployment.yaml | 2 +- 38 files changed, 43 insertions(+), 38 deletions(-) diff --git a/operations/helm/charts/alloy/CHANGELOG.md b/operations/helm/charts/alloy/CHANGELOG.md index fc66d83346..cfe067dfc5 100644 --- a/operations/helm/charts/alloy/CHANGELOG.md +++ b/operations/helm/charts/alloy/CHANGELOG.md @@ -9,8 +9,13 @@ internal API changes are not present. Unreleased ---------- + +0.10.0 (2024-11-13) +---------- ### Enhancements + - Add support for adding hostAliases to the Helm chart. (@duncan485) +- Update to Grafana Alloy v1.5.0. (@thampiotr) 0.9.2 (2024-10-18) ------------------ diff --git a/operations/helm/charts/alloy/Chart.yaml b/operations/helm/charts/alloy/Chart.yaml index 27c934861a..ced9b35779 100644 --- a/operations/helm/charts/alloy/Chart.yaml +++ b/operations/helm/charts/alloy/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: alloy description: 'Grafana Alloy' type: application -version: 0.9.2 -appVersion: 'v1.4.3' +version: 0.10.0 +appVersion: 'v1.5.0' icon: https://raw.githubusercontent.com/grafana/alloy/main/docs/sources/assets/alloy_icon_orange.svg dependencies: diff --git a/operations/helm/charts/alloy/README.md b/operations/helm/charts/alloy/README.md index e0bce43085..ee53707b14 100644 --- a/operations/helm/charts/alloy/README.md +++ b/operations/helm/charts/alloy/README.md @@ -1,6 +1,6 @@ # Grafana Alloy Helm chart -![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![Version: 0.9.2](https://img.shields.io/badge/Version-0.9.2-informational?style=flat-square) ![AppVersion: v1.4.3](https://img.shields.io/badge/AppVersion-v1.4.3-informational?style=flat-square) +![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![Version: 0.10.0](https://img.shields.io/badge/Version-0.10.0-informational?style=flat-square) ![AppVersion: v1.5.0](https://img.shields.io/badge/AppVersion-v1.5.0-informational?style=flat-square) Helm chart for deploying [Grafana Alloy][] to Kubernetes. diff --git a/operations/helm/tests/additional-serviceaccount-label/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/additional-serviceaccount-label/alloy/templates/controllers/daemonset.yaml index 7ccc31efd2..4d9474b7ef 100644 --- a/operations/helm/tests/additional-serviceaccount-label/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/additional-serviceaccount-label/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/clustering/alloy/templates/controllers/statefulset.yaml b/operations/helm/tests/clustering/alloy/templates/controllers/statefulset.yaml index 3c6c0173e5..8972a08e6b 100644 --- a/operations/helm/tests/clustering/alloy/templates/controllers/statefulset.yaml +++ b/operations/helm/tests/clustering/alloy/templates/controllers/statefulset.yaml @@ -30,7 +30,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/controller-deployment-pdb-max-unavailable/alloy/templates/controllers/deployment.yaml b/operations/helm/tests/controller-deployment-pdb-max-unavailable/alloy/templates/controllers/deployment.yaml index dae7f92479..72a0d49a3d 100644 --- a/operations/helm/tests/controller-deployment-pdb-max-unavailable/alloy/templates/controllers/deployment.yaml +++ b/operations/helm/tests/controller-deployment-pdb-max-unavailable/alloy/templates/controllers/deployment.yaml @@ -28,7 +28,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/controller-deployment-pdb-min-available/alloy/templates/controllers/deployment.yaml b/operations/helm/tests/controller-deployment-pdb-min-available/alloy/templates/controllers/deployment.yaml index dae7f92479..72a0d49a3d 100644 --- a/operations/helm/tests/controller-deployment-pdb-min-available/alloy/templates/controllers/deployment.yaml +++ b/operations/helm/tests/controller-deployment-pdb-min-available/alloy/templates/controllers/deployment.yaml @@ -28,7 +28,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/controller-statefulset-pdb-max-unavailable/alloy/templates/controllers/statefulset.yaml b/operations/helm/tests/controller-statefulset-pdb-max-unavailable/alloy/templates/controllers/statefulset.yaml index 2d63337067..e953031d61 100644 --- a/operations/helm/tests/controller-statefulset-pdb-max-unavailable/alloy/templates/controllers/statefulset.yaml +++ b/operations/helm/tests/controller-statefulset-pdb-max-unavailable/alloy/templates/controllers/statefulset.yaml @@ -30,7 +30,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/controller-statefulset-pdb-min-available/alloy/templates/controllers/statefulset.yaml b/operations/helm/tests/controller-statefulset-pdb-min-available/alloy/templates/controllers/statefulset.yaml index 2d63337067..e953031d61 100644 --- a/operations/helm/tests/controller-statefulset-pdb-min-available/alloy/templates/controllers/statefulset.yaml +++ b/operations/helm/tests/controller-statefulset-pdb-min-available/alloy/templates/controllers/statefulset.yaml @@ -30,7 +30,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/controller-volumes-extra/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/controller-volumes-extra/alloy/templates/controllers/daemonset.yaml index b57436d247..6dce3c8fa5 100644 --- a/operations/helm/tests/controller-volumes-extra/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/controller-volumes-extra/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/create-daemonset-hostnetwork/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/create-daemonset-hostnetwork/alloy/templates/controllers/daemonset.yaml index ad51d8310d..1d75b14409 100644 --- a/operations/helm/tests/create-daemonset-hostnetwork/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/create-daemonset-hostnetwork/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/create-daemonset/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/create-daemonset/alloy/templates/controllers/daemonset.yaml index 7ccc31efd2..4d9474b7ef 100644 --- a/operations/helm/tests/create-daemonset/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/create-daemonset/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/create-deployment-autoscaling/alloy/templates/controllers/deployment.yaml b/operations/helm/tests/create-deployment-autoscaling/alloy/templates/controllers/deployment.yaml index a1df33be37..b53d9ad8ed 100644 --- a/operations/helm/tests/create-deployment-autoscaling/alloy/templates/controllers/deployment.yaml +++ b/operations/helm/tests/create-deployment-autoscaling/alloy/templates/controllers/deployment.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/create-deployment/alloy/templates/controllers/deployment.yaml b/operations/helm/tests/create-deployment/alloy/templates/controllers/deployment.yaml index dae7f92479..72a0d49a3d 100644 --- a/operations/helm/tests/create-deployment/alloy/templates/controllers/deployment.yaml +++ b/operations/helm/tests/create-deployment/alloy/templates/controllers/deployment.yaml @@ -28,7 +28,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/create-statefulset-autoscaling/alloy/templates/controllers/statefulset.yaml b/operations/helm/tests/create-statefulset-autoscaling/alloy/templates/controllers/statefulset.yaml index 74ca132539..9146f6b8d2 100644 --- a/operations/helm/tests/create-statefulset-autoscaling/alloy/templates/controllers/statefulset.yaml +++ b/operations/helm/tests/create-statefulset-autoscaling/alloy/templates/controllers/statefulset.yaml @@ -29,7 +29,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/create-statefulset/alloy/templates/controllers/statefulset.yaml b/operations/helm/tests/create-statefulset/alloy/templates/controllers/statefulset.yaml index 2d63337067..e953031d61 100644 --- a/operations/helm/tests/create-statefulset/alloy/templates/controllers/statefulset.yaml +++ b/operations/helm/tests/create-statefulset/alloy/templates/controllers/statefulset.yaml @@ -30,7 +30,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/custom-config/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/custom-config/alloy/templates/controllers/daemonset.yaml index 7ccc31efd2..4d9474b7ef 100644 --- a/operations/helm/tests/custom-config/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/custom-config/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/default-values/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/default-values/alloy/templates/controllers/daemonset.yaml index 7ccc31efd2..4d9474b7ef 100644 --- a/operations/helm/tests/default-values/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/default-values/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/enable-servicemonitor-tls/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/enable-servicemonitor-tls/alloy/templates/controllers/daemonset.yaml index 6d43e773c3..07bbf1331e 100644 --- a/operations/helm/tests/enable-servicemonitor-tls/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/enable-servicemonitor-tls/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/enable-servicemonitor/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/enable-servicemonitor/alloy/templates/controllers/daemonset.yaml index 7ccc31efd2..4d9474b7ef 100644 --- a/operations/helm/tests/enable-servicemonitor/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/enable-servicemonitor/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/envFrom/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/envFrom/alloy/templates/controllers/daemonset.yaml index 7d721e3b27..71bb1d1eec 100644 --- a/operations/helm/tests/envFrom/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/envFrom/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/existing-config/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/existing-config/alloy/templates/controllers/daemonset.yaml index 965ae9e429..ebea934e31 100644 --- a/operations/helm/tests/existing-config/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/existing-config/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/extra-env/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/extra-env/alloy/templates/controllers/daemonset.yaml index 66d83eeb13..cb9bb328c8 100644 --- a/operations/helm/tests/extra-env/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/extra-env/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/extra-ports/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/extra-ports/alloy/templates/controllers/daemonset.yaml index fc871cdca5..cd9a8d391e 100644 --- a/operations/helm/tests/extra-ports/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/extra-ports/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/faro-ingress/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/faro-ingress/alloy/templates/controllers/daemonset.yaml index 7cf86c80da..405a4242ab 100644 --- a/operations/helm/tests/faro-ingress/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/faro-ingress/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/global-image-pullsecrets/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/global-image-pullsecrets/alloy/templates/controllers/daemonset.yaml index 875430dd0f..a7e2c3a7f8 100644 --- a/operations/helm/tests/global-image-pullsecrets/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/global-image-pullsecrets/alloy/templates/controllers/daemonset.yaml @@ -32,7 +32,7 @@ spec: - name: global-cred containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/global-image-registry/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/global-image-registry/alloy/templates/controllers/daemonset.yaml index 1933902acc..422a6edd09 100644 --- a/operations/helm/tests/global-image-registry/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/global-image-registry/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: quay.io/grafana/alloy:v1.4.3 + image: quay.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/host-alias/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/host-alias/alloy/templates/controllers/daemonset.yaml index af29f80c4d..3cf8558ef9 100644 --- a/operations/helm/tests/host-alias/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/host-alias/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/initcontainers/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/initcontainers/alloy/templates/controllers/daemonset.yaml index 73f378bfeb..11935c9c47 100644 --- a/operations/helm/tests/initcontainers/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/initcontainers/alloy/templates/controllers/daemonset.yaml @@ -45,7 +45,7 @@ spec: name: geoip containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/lifecycle-hooks/alloy/templates/controllers/deployment.yaml b/operations/helm/tests/lifecycle-hooks/alloy/templates/controllers/deployment.yaml index 0f70f1da0a..af71461cf7 100644 --- a/operations/helm/tests/lifecycle-hooks/alloy/templates/controllers/deployment.yaml +++ b/operations/helm/tests/lifecycle-hooks/alloy/templates/controllers/deployment.yaml @@ -28,7 +28,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/local-image-pullsecrets/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/local-image-pullsecrets/alloy/templates/controllers/daemonset.yaml index cd0ef136e3..19cb2adcf3 100644 --- a/operations/helm/tests/local-image-pullsecrets/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/local-image-pullsecrets/alloy/templates/controllers/daemonset.yaml @@ -29,7 +29,7 @@ spec: - name: local-cred containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/local-image-registry/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/local-image-registry/alloy/templates/controllers/daemonset.yaml index 1933902acc..422a6edd09 100644 --- a/operations/helm/tests/local-image-registry/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/local-image-registry/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: quay.io/grafana/alloy:v1.4.3 + image: quay.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/nodeselectors-and-tolerations/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/nodeselectors-and-tolerations/alloy/templates/controllers/daemonset.yaml index 1f378083e0..0164da26d1 100644 --- a/operations/helm/tests/nodeselectors-and-tolerations/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/nodeselectors-and-tolerations/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/nonroot/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/nonroot/alloy/templates/controllers/daemonset.yaml index d66f119ac8..842a1cb3b1 100644 --- a/operations/helm/tests/nonroot/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/nonroot/alloy/templates/controllers/daemonset.yaml @@ -29,7 +29,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/pod_annotations/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/pod_annotations/alloy/templates/controllers/daemonset.yaml index 2216d62453..48000f3227 100644 --- a/operations/helm/tests/pod_annotations/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/pod_annotations/alloy/templates/controllers/daemonset.yaml @@ -28,7 +28,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/sidecars/alloy/templates/controllers/daemonset.yaml b/operations/helm/tests/sidecars/alloy/templates/controllers/daemonset.yaml index 897f2367d1..0c096d1939 100644 --- a/operations/helm/tests/sidecars/alloy/templates/controllers/daemonset.yaml +++ b/operations/helm/tests/sidecars/alloy/templates/controllers/daemonset.yaml @@ -27,7 +27,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/termination-grace-period/alloy/templates/controllers/deployment.yaml b/operations/helm/tests/termination-grace-period/alloy/templates/controllers/deployment.yaml index e5801c1e39..42cb679036 100644 --- a/operations/helm/tests/termination-grace-period/alloy/templates/controllers/deployment.yaml +++ b/operations/helm/tests/termination-grace-period/alloy/templates/controllers/deployment.yaml @@ -28,7 +28,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run diff --git a/operations/helm/tests/topologyspreadconstraints/alloy/templates/controllers/deployment.yaml b/operations/helm/tests/topologyspreadconstraints/alloy/templates/controllers/deployment.yaml index 1c8d9f9e99..a832d7368a 100644 --- a/operations/helm/tests/topologyspreadconstraints/alloy/templates/controllers/deployment.yaml +++ b/operations/helm/tests/topologyspreadconstraints/alloy/templates/controllers/deployment.yaml @@ -28,7 +28,7 @@ spec: serviceAccountName: alloy containers: - name: alloy - image: docker.io/grafana/alloy:v1.4.3 + image: docker.io/grafana/alloy:v1.5.0 imagePullPolicy: IfNotPresent args: - run From 9b73e94a90062ab866d0e140a088440797fe0afe Mon Sep 17 00:00:00 2001 From: Piotr <17101802+thampiotr@users.noreply.github.com> Date: Wed, 13 Nov 2024 14:29:04 +0000 Subject: [PATCH 12/32] Update release instructions (#2081) --- docs/developer/release/11-update-otel.md | 13 ------------- docs/developer/release/README.md | 1 - 2 files changed, 14 deletions(-) delete mode 100644 docs/developer/release/11-update-otel.md diff --git a/docs/developer/release/11-update-otel.md b/docs/developer/release/11-update-otel.md deleted file mode 100644 index 9ff6d7b92e..0000000000 --- a/docs/developer/release/11-update-otel.md +++ /dev/null @@ -1,13 +0,0 @@ -# Update Open Telemetry Contrib - -Grafana Alloy is listed as a distribution of the OpenTelemetry Collector. If there are any new OTel components that Grafana Alloy needs to be associated with, then open a PR in [OpenTelemetry Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) and add Alloy to the list of distributions. [Example](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/653ab064bb797ed2b4ae599936a7b9cfdad18a29/receiver/kafkareceiver/README.md?plain=1#L7) - -## Steps - -1. Determine if there are any new OTEL components by looking at the changelog. - -2. Create a PR in OpenTelemetry Contrib. - -3. Find those OTEL components in contrib and add Grafana Alloy as a distribution. - -4. Tag Juraci ([jpkrohling](https://github.com/jpkrohling)) on the PR. diff --git a/docs/developer/release/README.md b/docs/developer/release/README.md index 7f9d345e48..8167a54afe 100644 --- a/docs/developer/release/README.md +++ b/docs/developer/release/README.md @@ -41,7 +41,6 @@ responsible for ownership of the following workflows: 5. [Update Helm Charts](./8-update-helm-charts.md) 6. [Update Homebrew](./9-update-homebrew.md) 7. [Announce Release](./10-announce-release.md) -8. [Update OTEL Contrib](./11-update-otel.md) ## Patch Release Publish - latest version (`1.15.1`, `1.15.2`...) 1. [Update the "main" and "release/" branches](./3-update-version-in-code.md) From 218a6837a8d80e55064b07bb8a52c597df1b2d23 Mon Sep 17 00:00:00 2001 From: Ilya Reshetnikov Date: Thu, 14 Nov 2024 07:25:17 +1100 Subject: [PATCH 13/32] Update prometheus-metrics.md (#2051) * Update prometheus-metrics.md fixing a typo * Update prometheus-metrics.md * Update prometheus-metrics.md --- docs/sources/collect/prometheus-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/collect/prometheus-metrics.md b/docs/sources/collect/prometheus-metrics.md index e0819a4ed1..3faa8d5b48 100644 --- a/docs/sources/collect/prometheus-metrics.md +++ b/docs/sources/collect/prometheus-metrics.md @@ -34,7 +34,7 @@ This topic describes how to: Before components can collect Prometheus metrics, you must have a component responsible for writing those metrics somewhere. -The [prometheus.remote_write][] component is responsible for delivering Prometheus metrics to one or Prometheus-compatible endpoints. +The [prometheus.remote_write][] component is responsible for delivering Prometheus metrics to one or more Prometheus-compatible endpoints. After a `prometheus.remote_write` component is defined, you can use other {{< param "PRODUCT_NAME" >}} components to forward metrics to it. To configure a `prometheus.remote_write` component for metrics delivery, complete the following steps: From 7290a0661889477de51665648e2acce3891254c9 Mon Sep 17 00:00:00 2001 From: William Dumont Date: Thu, 14 Nov 2024 15:45:53 +0100 Subject: [PATCH 14/32] Add integration test for OTLP internal metrics (#2058) * add integration test for OTLP internal metrics * rename otel gen app --- .../common/metrics_assert.go | 4 +- .../{otel-metrics-gen => otel-gen}/Dockerfile | 2 +- .../{otel-metrics-gen => otel-gen}/main.go | 46 ++++++++++--- .../configs/tempo/tempo.yaml | 64 +++++++++++++++++++ .../cmd/integration-tests/docker-compose.yaml | 17 ++++- .../tests/otlp-metrics/config.alloy | 31 ++++++++- .../otlp_alloy_integration_metrics_test.go | 23 +++++++ 7 files changed, 171 insertions(+), 16 deletions(-) rename internal/cmd/integration-tests/configs/{otel-metrics-gen => otel-gen}/Dockerfile (78%) rename internal/cmd/integration-tests/configs/{otel-metrics-gen => otel-gen}/main.go (63%) create mode 100644 internal/cmd/integration-tests/configs/tempo/tempo.yaml create mode 100644 internal/cmd/integration-tests/tests/otlp-metrics/otlp_alloy_integration_metrics_test.go diff --git a/internal/cmd/integration-tests/common/metrics_assert.go b/internal/cmd/integration-tests/common/metrics_assert.go index 51f2d97daa..b924130672 100644 --- a/internal/cmd/integration-tests/common/metrics_assert.go +++ b/internal/cmd/integration-tests/common/metrics_assert.go @@ -24,7 +24,7 @@ var PromDefaultHistogramMetric = []string{ "golang_native_histogram", } -// Default metrics list according to what the otel-metrics-gen app is generating. +// Default metrics list according to what the otel-gen app is generating. var OtelDefaultMetrics = []string{ "example_counter", "example_float_counter", @@ -34,7 +34,7 @@ var OtelDefaultMetrics = []string{ "example_float_histogram_bucket", } -// Default histogram metrics list according to what the otel-metrics-gen app is generating. +// Default histogram metrics list according to what the otel-gen app is generating. var OtelDefaultHistogramMetrics = []string{ "example_exponential_histogram", "example_exponential_float_histogram", diff --git a/internal/cmd/integration-tests/configs/otel-metrics-gen/Dockerfile b/internal/cmd/integration-tests/configs/otel-gen/Dockerfile similarity index 78% rename from internal/cmd/integration-tests/configs/otel-metrics-gen/Dockerfile rename to internal/cmd/integration-tests/configs/otel-gen/Dockerfile index f07a688fee..7297f48ea7 100644 --- a/internal/cmd/integration-tests/configs/otel-metrics-gen/Dockerfile +++ b/internal/cmd/integration-tests/configs/otel-gen/Dockerfile @@ -3,7 +3,7 @@ WORKDIR /app/ COPY go.mod go.sum ./ COPY syntax/go.mod syntax/go.sum ./syntax/ RUN go mod download -COPY ./internal/cmd/integration-tests/configs/otel-metrics-gen/ ./ +COPY ./internal/cmd/integration-tests/configs/otel-gen/ ./ RUN CGO_ENABLED=0 go build -o main main.go FROM alpine:3.18 COPY --from=build /app/main /app/main diff --git a/internal/cmd/integration-tests/configs/otel-metrics-gen/main.go b/internal/cmd/integration-tests/configs/otel-gen/main.go similarity index 63% rename from internal/cmd/integration-tests/configs/otel-metrics-gen/main.go rename to internal/cmd/integration-tests/configs/otel-gen/main.go index 6f88d7725b..b3ade63f22 100644 --- a/internal/cmd/integration-tests/configs/otel-metrics-gen/main.go +++ b/internal/cmd/integration-tests/configs/otel-gen/main.go @@ -8,9 +8,10 @@ import ( "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" - "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" sdkmetric "go.opentelemetry.io/otel/sdk/metric" "go.opentelemetry.io/otel/sdk/resource" + "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.7.0" ) @@ -22,29 +23,51 @@ func main() { if !ok { otlpExporterEndpoint = "localhost:4318" } - exporter, err := otlpmetrichttp.New(ctx, + + // Setting up the trace exporter + traceExporter, err := otlptracehttp.New(ctx, + otlptracehttp.WithInsecure(), + otlptracehttp.WithEndpoint(otlpExporterEndpoint), + ) + if err != nil { + log.Fatalf("failed to create trace exporter: %v", err) + } + + // Setting up the metric exporter + metricExporter, err := otlpmetrichttp.New(ctx, otlpmetrichttp.WithInsecure(), otlpmetrichttp.WithEndpoint(otlpExporterEndpoint), ) if err != nil { - log.Fatalf("failed to create exporter: %v", err) + log.Fatalf("failed to create metric exporter: %v", err) } resource, err := resource.New(ctx, resource.WithAttributes( - semconv.ServiceNameKey.String("otel-metrics-gen"), + semconv.ServiceNameKey.String("otel-gen"), ), ) if err != nil { log.Fatalf("failed to create resource: %v", err) } - exponentialHistogramView := metric.NewView( - metric.Instrument{ + tp := trace.NewTracerProvider( + trace.WithBatcher(traceExporter), + trace.WithResource(resource), + ) + otel.SetTracerProvider(tp) + defer func() { + if err := tp.Shutdown(ctx); err != nil { + log.Fatalf("failed to shut down tracer provider: %v", err) + } + }() + + exponentialHistogramView := sdkmetric.NewView( + sdkmetric.Instrument{ Name: "example_exponential_*", }, - metric.Stream{ - Aggregation: metric.AggregationBase2ExponentialHistogram{ + sdkmetric.Stream{ + Aggregation: sdkmetric.AggregationBase2ExponentialHistogram{ MaxSize: 160, MaxScale: 20, }, @@ -52,9 +75,9 @@ func main() { ) provider := sdkmetric.NewMeterProvider( - sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exporter, sdkmetric.WithInterval(1*time.Second))), + sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExporter, sdkmetric.WithInterval(1*time.Second))), sdkmetric.WithResource(resource), - metric.WithView(exponentialHistogramView), + sdkmetric.WithView(exponentialHistogramView), ) otel.SetMeterProvider(provider) defer func() { @@ -66,6 +89,7 @@ func main() { } }() + tracer := otel.Tracer("example-tracer") meter := otel.Meter("example-meter") counter, _ := meter.Int64Counter("example_counter") floatCounter, _ := meter.Float64Counter("example_float_counter") @@ -77,6 +101,7 @@ func main() { exponentialFloatHistogram, _ := meter.Float64Histogram("example_exponential_float_histogram") for { + ctx, span := tracer.Start(ctx, "sample-trace") counter.Add(ctx, 10) floatCounter.Add(ctx, 2.5) upDownCounter.Add(ctx, -5) @@ -87,5 +112,6 @@ func main() { exponentialFloatHistogram.Record(ctx, 1.5) time.Sleep(200 * time.Millisecond) + span.End() } } diff --git a/internal/cmd/integration-tests/configs/tempo/tempo.yaml b/internal/cmd/integration-tests/configs/tempo/tempo.yaml new file mode 100644 index 0000000000..7c62f85db5 --- /dev/null +++ b/internal/cmd/integration-tests/configs/tempo/tempo.yaml @@ -0,0 +1,64 @@ +stream_over_http_enabled: true +server: + http_listen_port: 3200 + log_level: info + + +cache: + background: + writeback_goroutines: 5 + caches: + - roles: + - frontend-search + memcached: + host: memcached:11211 + +query_frontend: + search: + duration_slo: 5s + throughput_bytes_slo: 1.073741824e+09 + trace_by_id: + duration_slo: 5s + +distributor: + receivers: # this configuration will listen on all ports and protocols that tempo is capable of. + jaeger: # the receives all come from the OpenTelemetry collector. more configuration information can + protocols: # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver + thrift_http: # + grpc: # for a production deployment you should only enable the receivers you need! + thrift_binary: + thrift_compact: + zipkin: + otlp: + protocols: + http: + grpc: + opencensus: + +ingester: + max_block_duration: 5m # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally + +compactor: + compaction: + block_retention: 1h # overall Tempo trace retention. set for demo purposes + +metrics_generator: + registry: + external_labels: + source: tempo + cluster: docker-compose + storage: + path: /var/tempo/generator/wal + remote_write: + - url: http://prometheus:9090/api/v1/write + send_exemplars: true + traces_storage: + path: /var/tempo/generator/traces + +storage: + trace: + backend: local # backend configuration to use + wal: + path: /var/tempo/wal # where to store the wal locally + local: + path: /var/tempo/blocks \ No newline at end of file diff --git a/internal/cmd/integration-tests/docker-compose.yaml b/internal/cmd/integration-tests/docker-compose.yaml index a74285de7d..1632286bab 100644 --- a/internal/cmd/integration-tests/docker-compose.yaml +++ b/internal/cmd/integration-tests/docker-compose.yaml @@ -10,6 +10,19 @@ services: - -config.file=/etc/mimir-config/mimir.yaml ports: - "9009:9009" + + tempo: + image: grafana/tempo:latest + command: [ "-config.file=/etc/tempo.yaml" ] + volumes: + - ./configs/tempo/tempo.yaml:/etc/tempo.yaml + ports: + - "14268:14268" # jaeger ingest + - "3200:3200" # tempo + - "9095:9095" # tempo grpc + - "4319:4317" # otlp grpc + - "4320:4318" # otlp http + - "9411:9411" # zipkin zookeeper: image: confluentinc/cp-zookeeper:latest @@ -44,9 +57,9 @@ services: ports: - "3100:3100" - otel-metrics-gen: + otel-gen: build: - dockerfile: ./internal/cmd/integration-tests/configs/otel-metrics-gen/Dockerfile + dockerfile: ./internal/cmd/integration-tests/configs/otel-gen/Dockerfile context: ../../.. environment: - OTEL_EXPORTER_ENDPOINT=${OTEL_EXPORTER_ENDPOINT:-host.docker.internal:4318} diff --git a/internal/cmd/integration-tests/tests/otlp-metrics/config.alloy b/internal/cmd/integration-tests/tests/otlp-metrics/config.alloy index 859181f9cc..a4e4aac931 100644 --- a/internal/cmd/integration-tests/tests/otlp-metrics/config.alloy +++ b/internal/cmd/integration-tests/tests/otlp-metrics/config.alloy @@ -3,6 +3,7 @@ otelcol.receiver.otlp "otlp_metrics" { output { metrics = [otelcol.processor.attributes.otlp_metrics.input, otelcol.exporter.prometheus.otlp_to_prom_metrics.input] + traces = [otelcol.processor.attributes.otlp_metrics.input] } } @@ -13,14 +14,24 @@ otelcol.processor.attributes "otlp_metrics" { action = "insert" } + output { + metrics = [otelcol.processor.batch.otlp_metrics.input] + traces = [otelcol.processor.batch.otlp_metrics.input] + } +} + +otelcol.processor.batch "otlp_metrics" { output { metrics = [otelcol.exporter.otlphttp.otlp_metrics.input] + traces = [otelcol.exporter.otlphttp.otlp_metrics.input] } } otelcol.exporter.otlphttp "otlp_metrics" { + metrics_endpoint = "http://localhost:9009/otlp/v1/metrics" + traces_endpoint = "http://localhost:4320/v1/traces" client { - endpoint = "http://localhost:9009/otlp" + endpoint = "ignore" tls { insecure = true insecure_skip_verify = true @@ -47,3 +58,21 @@ prometheus.remote_write "otlp_to_prom_metrics" { test_name = "otlp_to_prom_metrics", } } + +prometheus.exporter.self "otlp_integration" {} + +prometheus.scrape "otlp_integration" { + targets = prometheus.exporter.self.otlp_integration.targets + forward_to = [prometheus.remote_write.otlp_integration.receiver] + scrape_interval = "1s" + scrape_timeout = "500ms" +} + +prometheus.remote_write "otlp_integration" { + endpoint { + url = "http://localhost:9009/api/v1/push" + } + external_labels = { + test_name = "otlp_integration", + } +} \ No newline at end of file diff --git a/internal/cmd/integration-tests/tests/otlp-metrics/otlp_alloy_integration_metrics_test.go b/internal/cmd/integration-tests/tests/otlp-metrics/otlp_alloy_integration_metrics_test.go new file mode 100644 index 0000000000..4fb5192593 --- /dev/null +++ b/internal/cmd/integration-tests/tests/otlp-metrics/otlp_alloy_integration_metrics_test.go @@ -0,0 +1,23 @@ +//go:build !windows + +package main + +import ( + "testing" + + "github.com/grafana/alloy/internal/cmd/integration-tests/common" +) + +func TestAlloyIntegrationMetrics(t *testing.T) { + // These otel metrics are needed in the k8s-monitoring helm chart (https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring-v1/default_allow_lists/alloy_integration.yaml) + var OTLPMetrics = []string{ + "otelcol_exporter_send_failed_spans_total", + "otelcol_exporter_sent_spans_total", + "otelcol_processor_batch_batch_send_size_bucket", + "otelcol_processor_batch_metadata_cardinality", + "otelcol_processor_batch_timeout_trigger_send_total", + "otelcol_receiver_accepted_spans_total", + "otelcol_receiver_refused_spans_total", + } + common.MimirMetricsTest(t, OTLPMetrics, []string{}, "otlp_integration") +} From 8b5c04f6fbde2773242b7b85512929ebd431aa38 Mon Sep 17 00:00:00 2001 From: mattdurham Date: Thu, 14 Nov 2024 16:25:24 -0500 Subject: [PATCH 15/32] True up the documentation with the code. (#2095) --- .../prometheus/prometheus.exporter.unix.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/sources/reference/components/prometheus/prometheus.exporter.unix.md b/docs/sources/reference/components/prometheus/prometheus.exporter.unix.md index b5e22f86c5..c7dc3e5c42 100644 --- a/docs/sources/reference/components/prometheus/prometheus.exporter.unix.md +++ b/docs/sources/reference/components/prometheus/prometheus.exporter.unix.md @@ -105,12 +105,12 @@ The following blocks are supported inside the definition of `prometheus.exporter ### cpu block -| Name | Type | Description | Default | Required | -| --------------- | --------- | --------------------------------------------------- | ------- | -------- | -| `guest` | `boolean` | Enable the `node_cpu_guest_seconds_total` metric. | true | no | -| `info` | `boolean` | Enable the `cpu_info metric` for the cpu collector. | true | no | -| `bugs_include` | `string` | Regexp of `bugs` field in cpu info to filter. | | no | -| `flags_include` | `string` | Regexp of `flags` field in cpu info to filter. | | no | +| Name | Type | Description | Default | Required | +| --------------- | --------- | -------------------------------------------------- |---------| -------- | +| `guest` | `boolean` | Enable the `node_cpu_guest_seconds_total` metric. | false | no | +| `info` | `boolean` | Enable the `cpu_info` metric for the cpu collector.| false | no | +| `bugs_include` | `string` | Regexp of `bugs` field in cpu info to filter. | | no | +| `flags_include` | `string` | Regexp of `flags` field in cpu info to filter. | | no | ### disk block From f24c2f9efeb5f323fd6ebc3bee13aa8d4fd411b8 Mon Sep 17 00:00:00 2001 From: Piotr <17101802+thampiotr@users.noreply.github.com> Date: Fri, 15 Nov 2024 11:50:56 +0000 Subject: [PATCH 16/32] Fix race condition in otelcol component wrappers (#2027) * Fix race condition in otelcol processors * fix issues * Test the pausing scheduler --------- Co-authored-by: William Dumont --- .../component/otelcol/connector/connector.go | 23 +-- .../component/otelcol/exporter/exporter.go | 23 +-- .../internal/lazyconsumer/lazyconsumer.go | 60 +++++++ .../lazyconsumer/lazyconsumer_test.go | 161 ++++++++++++++++++ .../otelcol/internal/scheduler/scheduler.go | 36 +++- .../internal/scheduler/scheduler_test.go | 60 ++++++- .../otelcol/processor/batch/batch_test.go | 77 ++++++++- .../component/otelcol/processor/processor.go | 25 +-- 8 files changed, 419 insertions(+), 46 deletions(-) create mode 100644 internal/component/otelcol/internal/lazyconsumer/lazyconsumer_test.go diff --git a/internal/component/otelcol/connector/connector.go b/internal/component/otelcol/connector/connector.go index a0185ae0f4..bb9b3a151e 100644 --- a/internal/component/otelcol/connector/connector.go +++ b/internal/component/otelcol/connector/connector.go @@ -7,15 +7,6 @@ import ( "errors" "os" - "github.com/grafana/alloy/internal/build" - "github.com/grafana/alloy/internal/component" - "github.com/grafana/alloy/internal/component/otelcol" - otelcolCfg "github.com/grafana/alloy/internal/component/otelcol/config" - "github.com/grafana/alloy/internal/component/otelcol/internal/fanoutconsumer" - "github.com/grafana/alloy/internal/component/otelcol/internal/lazycollector" - "github.com/grafana/alloy/internal/component/otelcol/internal/lazyconsumer" - "github.com/grafana/alloy/internal/component/otelcol/internal/scheduler" - "github.com/grafana/alloy/internal/util/zapadapter" "github.com/prometheus/client_golang/prometheus" otelcomponent "go.opentelemetry.io/collector/component" "go.opentelemetry.io/collector/config/configtelemetry" @@ -26,6 +17,16 @@ import ( otelmetric "go.opentelemetry.io/otel/metric" "go.opentelemetry.io/otel/metric/noop" "go.opentelemetry.io/otel/sdk/metric" + + "github.com/grafana/alloy/internal/build" + "github.com/grafana/alloy/internal/component" + "github.com/grafana/alloy/internal/component/otelcol" + otelcolCfg "github.com/grafana/alloy/internal/component/otelcol/config" + "github.com/grafana/alloy/internal/component/otelcol/internal/fanoutconsumer" + "github.com/grafana/alloy/internal/component/otelcol/internal/lazycollector" + "github.com/grafana/alloy/internal/component/otelcol/internal/lazyconsumer" + "github.com/grafana/alloy/internal/component/otelcol/internal/scheduler" + "github.com/grafana/alloy/internal/util/zapadapter" ) const ( @@ -94,7 +95,7 @@ var ( func New(opts component.Options, f otelconnector.Factory, args Arguments) (*Connector, error) { ctx, cancel := context.WithCancel(context.Background()) - consumer := lazyconsumer.New(ctx) + consumer := lazyconsumer.NewPaused(ctx) // Create a lazy collector where metrics from the upstream component will be // forwarded. @@ -116,7 +117,7 @@ func New(opts component.Options, f otelconnector.Factory, args Arguments) (*Conn factory: f, consumer: consumer, - sched: scheduler.New(opts.Logger), + sched: scheduler.NewWithPauseCallbacks(opts.Logger, consumer.Pause, consumer.Resume), collector: collector, } if err := p.Update(args); err != nil { diff --git a/internal/component/otelcol/exporter/exporter.go b/internal/component/otelcol/exporter/exporter.go index 90f1826966..22a2b56d3a 100644 --- a/internal/component/otelcol/exporter/exporter.go +++ b/internal/component/otelcol/exporter/exporter.go @@ -7,15 +7,6 @@ import ( "errors" "os" - "github.com/grafana/alloy/internal/build" - "github.com/grafana/alloy/internal/component" - "github.com/grafana/alloy/internal/component/otelcol" - otelcolCfg "github.com/grafana/alloy/internal/component/otelcol/config" - "github.com/grafana/alloy/internal/component/otelcol/internal/lazycollector" - "github.com/grafana/alloy/internal/component/otelcol/internal/lazyconsumer" - "github.com/grafana/alloy/internal/component/otelcol/internal/scheduler" - "github.com/grafana/alloy/internal/component/otelcol/internal/views" - "github.com/grafana/alloy/internal/util/zapadapter" "github.com/prometheus/client_golang/prometheus" otelcomponent "go.opentelemetry.io/collector/component" "go.opentelemetry.io/collector/config/configtelemetry" @@ -26,6 +17,16 @@ import ( otelmetric "go.opentelemetry.io/otel/metric" "go.opentelemetry.io/otel/metric/noop" "go.opentelemetry.io/otel/sdk/metric" + + "github.com/grafana/alloy/internal/build" + "github.com/grafana/alloy/internal/component" + "github.com/grafana/alloy/internal/component/otelcol" + otelcolCfg "github.com/grafana/alloy/internal/component/otelcol/config" + "github.com/grafana/alloy/internal/component/otelcol/internal/lazycollector" + "github.com/grafana/alloy/internal/component/otelcol/internal/lazyconsumer" + "github.com/grafana/alloy/internal/component/otelcol/internal/scheduler" + "github.com/grafana/alloy/internal/component/otelcol/internal/views" + "github.com/grafana/alloy/internal/util/zapadapter" ) // Arguments is an extension of component.Arguments which contains necessary @@ -108,7 +109,7 @@ var ( func New(opts component.Options, f otelexporter.Factory, args Arguments, supportedSignals TypeSignal) (*Exporter, error) { ctx, cancel := context.WithCancel(context.Background()) - consumer := lazyconsumer.New(ctx) + consumer := lazyconsumer.NewPaused(ctx) // Create a lazy collector where metrics from the upstream component will be // forwarded. @@ -130,7 +131,7 @@ func New(opts component.Options, f otelexporter.Factory, args Arguments, support factory: f, consumer: consumer, - sched: scheduler.New(opts.Logger), + sched: scheduler.NewWithPauseCallbacks(opts.Logger, consumer.Pause, consumer.Resume), collector: collector, supportedSignals: supportedSignals, diff --git a/internal/component/otelcol/internal/lazyconsumer/lazyconsumer.go b/internal/component/otelcol/internal/lazyconsumer/lazyconsumer.go index 3d2e653423..a5813e8469 100644 --- a/internal/component/otelcol/internal/lazyconsumer/lazyconsumer.go +++ b/internal/component/otelcol/internal/lazyconsumer/lazyconsumer.go @@ -17,6 +17,10 @@ import ( type Consumer struct { ctx context.Context + // pauseMut and pausedWg are used to implement Pause & Resume semantics. See Pause method for more info. + pauseMut sync.RWMutex + pausedWg *sync.WaitGroup + mut sync.RWMutex metricsConsumer otelconsumer.Metrics logsConsumer otelconsumer.Logs @@ -36,6 +40,13 @@ func New(ctx context.Context) *Consumer { return &Consumer{ctx: ctx} } +// NewPaused is like New, but returns a Consumer that is paused by calling Pause method. +func NewPaused(ctx context.Context) *Consumer { + c := New(ctx) + c.Pause() + return c +} + // Capabilities implements otelconsumer.baseConsumer. func (c *Consumer) Capabilities() otelconsumer.Capabilities { return otelconsumer.Capabilities{ @@ -52,6 +63,8 @@ func (c *Consumer) ConsumeTraces(ctx context.Context, td ptrace.Traces) error { return c.ctx.Err() } + c.waitUntilResumed() + c.mut.RLock() defer c.mut.RUnlock() @@ -73,6 +86,8 @@ func (c *Consumer) ConsumeMetrics(ctx context.Context, md pmetric.Metrics) error return c.ctx.Err() } + c.waitUntilResumed() + c.mut.RLock() defer c.mut.RUnlock() @@ -94,6 +109,8 @@ func (c *Consumer) ConsumeLogs(ctx context.Context, ld plog.Logs) error { return c.ctx.Err() } + c.waitUntilResumed() + c.mut.RLock() defer c.mut.RUnlock() @@ -109,6 +126,15 @@ func (c *Consumer) ConsumeLogs(ctx context.Context, ld plog.Logs) error { return c.logsConsumer.ConsumeLogs(ctx, ld) } +func (c *Consumer) waitUntilResumed() { + c.pauseMut.RLock() + pausedWg := c.pausedWg + c.pauseMut.RUnlock() + if pausedWg != nil { + pausedWg.Wait() + } +} + // SetConsumers updates the internal consumers that Consumer will forward data // to. It is valid for any combination of m, l, and t to be nil. func (c *Consumer) SetConsumers(t otelconsumer.Traces, m otelconsumer.Metrics, l otelconsumer.Logs) { @@ -119,3 +145,37 @@ func (c *Consumer) SetConsumers(t otelconsumer.Traces, m otelconsumer.Metrics, l c.logsConsumer = l c.tracesConsumer = t } + +// Pause will stop the consumer until Resume is called. While paused, the calls to Consume* methods will block. +// Pause can be called multiple times, but a single call to Resume will un-pause this consumer. Thread-safe. +func (c *Consumer) Pause() { + c.pauseMut.Lock() + defer c.pauseMut.Unlock() + + if c.pausedWg != nil { + return // already paused + } + + c.pausedWg = &sync.WaitGroup{} + c.pausedWg.Add(1) +} + +// Resume will revert the Pause call and the consumer will continue to work. See Pause for more details. +func (c *Consumer) Resume() { + c.pauseMut.Lock() + defer c.pauseMut.Unlock() + + if c.pausedWg == nil { + return // already resumed + } + + c.pausedWg.Done() // release all waiting + c.pausedWg = nil +} + +// IsPaused returns whether the consumer is currently paused. See Pause for details. +func (c *Consumer) IsPaused() bool { + c.pauseMut.RLock() + defer c.pauseMut.RUnlock() + return c.pausedWg != nil +} diff --git a/internal/component/otelcol/internal/lazyconsumer/lazyconsumer_test.go b/internal/component/otelcol/internal/lazyconsumer/lazyconsumer_test.go new file mode 100644 index 0000000000..1980029d5e --- /dev/null +++ b/internal/component/otelcol/internal/lazyconsumer/lazyconsumer_test.go @@ -0,0 +1,161 @@ +package lazyconsumer + +import ( + "context" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/require" + "go.opentelemetry.io/collector/pdata/plog" + "go.opentelemetry.io/collector/pdata/pmetric" + "go.opentelemetry.io/collector/pdata/ptrace" + "go.uber.org/goleak" + + "github.com/grafana/alloy/internal/runtime/componenttest" +) + +func Test_PauseAndResume(t *testing.T) { + c := New(componenttest.TestContext(t)) + require.False(t, c.IsPaused()) + c.Pause() + require.True(t, c.IsPaused()) + c.Resume() + require.False(t, c.IsPaused()) +} + +func Test_NewPaused(t *testing.T) { + c := NewPaused(componenttest.TestContext(t)) + require.True(t, c.IsPaused()) + c.Resume() + require.False(t, c.IsPaused()) +} + +func Test_PauseResume_MultipleCalls(t *testing.T) { + c := New(componenttest.TestContext(t)) + require.False(t, c.IsPaused()) + c.Pause() + c.Pause() + c.Pause() + require.True(t, c.IsPaused()) + c.Resume() + c.Resume() + c.Resume() + require.False(t, c.IsPaused()) +} + +func Test_ConsumeWaitsForResume(t *testing.T) { + goleak.VerifyNone(t, goleak.IgnoreCurrent()) + c := NewPaused(componenttest.TestContext(t)) + require.True(t, c.IsPaused()) + + method := map[string]func(){ + "ConsumeTraces": func() { + _ = c.ConsumeTraces(nil, ptrace.NewTraces()) + }, + "ConsumeMetrics": func() { + _ = c.ConsumeMetrics(nil, pmetric.NewMetrics()) + }, + "ConsumeLogs": func() { + _ = c.ConsumeLogs(nil, plog.NewLogs()) + }, + } + + for name, fn := range method { + t.Run(name, func(t *testing.T) { + c.Pause() + require.True(t, c.IsPaused()) + + started := make(chan struct{}) + finished := make(chan struct{}) + + // Start goroutine that attempts to run Consume* method + go func() { + started <- struct{}{} + fn() + finished <- struct{}{} + }() + + // Wait to be started + select { + case <-started: + case <-time.After(5 * time.Second): + t.Fatal("consumer goroutine never started") + } + + // Wait for a bit to ensure the consumer is blocking on Consume* function + select { + case <-finished: + t.Fatal("consumer should not have finished yet - it's paused") + case <-time.After(100 * time.Millisecond): + } + + // Resume the consumer and verify the Consume* function unblocked + c.Resume() + select { + case <-finished: + case <-time.After(5 * time.Second): + t.Fatal("consumer should have finished after resuming") + } + + }) + } +} + +func Test_PauseResume_Multithreaded(t *testing.T) { + goleak.VerifyNone(t, goleak.IgnoreCurrent()) + ctx, cancel := context.WithCancel(componenttest.TestContext(t)) + runs := 500 + routines := 5 + allDone := sync.WaitGroup{} + + c := NewPaused(componenttest.TestContext(t)) + require.True(t, c.IsPaused()) + + // Run goroutines that constantly try to call Consume* methods + for i := 0; i < routines; i++ { + allDone.Add(1) + go func() { + for { + select { + case <-ctx.Done(): + allDone.Done() + return + default: + _ = c.ConsumeLogs(ctx, plog.NewLogs()) + _ = c.ConsumeMetrics(ctx, pmetric.NewMetrics()) + _ = c.ConsumeTraces(ctx, ptrace.NewTraces()) + } + } + }() + } + + // Run goroutines that Pause and then Resume in parallel. + // In particular, this verifies we can call .Pause() and .Resume() on an already paused or already resumed consumer. + workChan := make(chan struct{}, routines) + for i := 0; i < routines; i++ { + allDone.Add(1) + go func() { + for { + select { + case <-workChan: + c.Pause() + c.Resume() + case <-ctx.Done(): + allDone.Done() + return + } + } + }() + } + + for i := 0; i < runs; i++ { + workChan <- struct{}{} + } + cancel() + + allDone.Wait() + + // Should not be paused as last call will always be c.Resume() + require.False(t, c.IsPaused()) +} diff --git a/internal/component/otelcol/internal/scheduler/scheduler.go b/internal/component/otelcol/internal/scheduler/scheduler.go index 022c09dadf..2c731616a5 100644 --- a/internal/component/otelcol/internal/scheduler/scheduler.go +++ b/internal/component/otelcol/internal/scheduler/scheduler.go @@ -9,10 +9,11 @@ import ( "time" "github.com/go-kit/log" - "github.com/grafana/alloy/internal/component" - "github.com/grafana/alloy/internal/runtime/logging/level" otelcomponent "go.opentelemetry.io/collector/component" "go.uber.org/multierr" + + "github.com/grafana/alloy/internal/component" + "github.com/grafana/alloy/internal/runtime/logging/level" ) // Scheduler implements manages a set of OpenTelemetry Collector components. @@ -39,6 +40,11 @@ type Scheduler struct { // newComponentsCh is written to when schedComponents gets updated. newComponentsCh chan struct{} + + // onPause is called when scheduler is making changes to running components. + onPause func() + // onResume is called when scheduler is done making changes to running components. + onResume func() } // New creates a new unstarted Scheduler. Call Run to start it, and call @@ -47,6 +53,21 @@ func New(l log.Logger) *Scheduler { return &Scheduler{ log: l, newComponentsCh: make(chan struct{}, 1), + onPause: func() {}, + onResume: func() {}, + } +} + +// NewWithPauseCallbacks is like New, but allows to specify onPause and onResume callbacks. The scheduler is assumed to +// start paused and only when its components are scheduled, it will call onResume. From then on, each update to running +// components via Schedule method will trigger a call to onPause and then onResume. When scheduler is shutting down, it +// will call onResume as a last step. +func NewWithPauseCallbacks(l log.Logger, onPause func(), onResume func()) *Scheduler { + return &Scheduler{ + log: l, + newComponentsCh: make(chan struct{}, 1), + onPause: onPause, + onResume: onResume, } } @@ -75,11 +96,16 @@ func (cs *Scheduler) Schedule(h otelcomponent.Host, cc ...otelcomponent.Componen // Run starts the Scheduler. Run will watch for schedule components to appear // and run them, terminating previously running components if they exist. func (cs *Scheduler) Run(ctx context.Context) error { + firstRun := true var components []otelcomponent.Component // Make sure we terminate all of our running components on shutdown. defer func() { + if !firstRun { // always handle the callbacks correctly + cs.onPause() + } cs.stopComponents(context.Background(), components...) + cs.onResume() }() // Wait for a write to cs.newComponentsCh. The initial list of components is @@ -90,6 +116,11 @@ func (cs *Scheduler) Run(ctx context.Context) error { case <-ctx.Done(): return nil case <-cs.newComponentsCh: + if !firstRun { + cs.onPause() // do not pause on first run + } else { + firstRun = false + } // Stop the old components before running new scheduled ones. cs.stopComponents(ctx, components...) @@ -100,6 +131,7 @@ func (cs *Scheduler) Run(ctx context.Context) error { level.Debug(cs.log).Log("msg", "scheduling components", "count", len(components)) components = cs.startComponents(ctx, host, components...) + cs.onResume() } } } diff --git a/internal/component/otelcol/internal/scheduler/scheduler_test.go b/internal/component/otelcol/internal/scheduler/scheduler_test.go index 5f238b33de..469d679b7f 100644 --- a/internal/component/otelcol/internal/scheduler/scheduler_test.go +++ b/internal/component/otelcol/internal/scheduler/scheduler_test.go @@ -5,11 +5,14 @@ import ( "testing" "time" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + otelcomponent "go.opentelemetry.io/collector/component" + "go.uber.org/atomic" + "github.com/grafana/alloy/internal/component/otelcol/internal/scheduler" "github.com/grafana/alloy/internal/runtime/componenttest" "github.com/grafana/alloy/internal/util" - "github.com/stretchr/testify/require" - otelcomponent "go.opentelemetry.io/collector/component" ) func TestScheduler(t *testing.T) { @@ -58,6 +61,59 @@ func TestScheduler(t *testing.T) { require.NoError(t, stopped.Wait(5*time.Second), "component did not shutdown") }) + t.Run("Pause callbacks are called", func(t *testing.T) { + var ( + pauseCalls = &atomic.Int32{} + resumeCalls = &atomic.Int32{} + l = util.TestLogger(t) + cs = scheduler.NewWithPauseCallbacks( + l, + func() { pauseCalls.Inc() }, + func() { resumeCalls.Inc() }, + ) + h = scheduler.NewHost(l) + ) + ctx, cancel := context.WithCancel(context.Background()) + + // Run our scheduler in the background. + go func() { + err := cs.Run(ctx) + require.NoError(t, err) + }() + + // Schedule our component, which should notify the started and stopped + // trigger once it starts and stops respectively. + component, started, stopped := newTriggerComponent() + cs.Schedule(h, component) + + toInt := func(a *atomic.Int32) int { return int(a.Load()) } + + require.EventuallyWithT(t, func(t *assert.CollectT) { + assert.Equal(t, 0, toInt(pauseCalls), "pause callbacks should not be called on first run") + assert.Equal(t, 1, toInt(resumeCalls), "resume callback should be called on first run") + }, 5*time.Second, 10*time.Millisecond, "pause/resume callbacks not called correctly") + + // Wait for the component to start, and then unschedule all components, which + // should cause our running component to terminate. + require.NoError(t, started.Wait(5*time.Second), "component did not start") + cs.Schedule(h) + + require.EventuallyWithT(t, func(t *assert.CollectT) { + assert.Equal(t, 1, toInt(pauseCalls), "pause callback should be called on second run") + assert.Equal(t, 2, toInt(resumeCalls), "resume callback should be called on second run") + }, 5*time.Second, 10*time.Millisecond, "pause/resume callbacks not called correctly") + + require.NoError(t, stopped.Wait(5*time.Second), "component did not shutdown") + + // Stop the scheduler + cancel() + + require.EventuallyWithT(t, func(t *assert.CollectT) { + assert.Equal(t, 2, toInt(pauseCalls), "pause callback should be called on shutdown") + assert.Equal(t, 3, toInt(resumeCalls), "resume callback should be called on shutdown") + }, 5*time.Second, 10*time.Millisecond, "pause/resume callbacks not called correctly") + }) + t.Run("Running components get stopped on shutdown", func(t *testing.T) { var ( l = util.TestLogger(t) diff --git a/internal/component/otelcol/processor/batch/batch_test.go b/internal/component/otelcol/processor/batch/batch_test.go index 12604eff83..b69c4802cc 100644 --- a/internal/component/otelcol/processor/batch/batch_test.go +++ b/internal/component/otelcol/processor/batch/batch_test.go @@ -5,6 +5,11 @@ import ( "testing" "time" + "github.com/grafana/dskit/backoff" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/collector/pdata/ptrace" + "go.opentelemetry.io/collector/processor/batchprocessor" + "github.com/grafana/alloy/internal/component/otelcol" "github.com/grafana/alloy/internal/component/otelcol/internal/fakeconsumer" "github.com/grafana/alloy/internal/component/otelcol/processor/batch" @@ -12,10 +17,6 @@ import ( "github.com/grafana/alloy/internal/runtime/logging/level" "github.com/grafana/alloy/internal/util" "github.com/grafana/alloy/syntax" - "github.com/grafana/dskit/backoff" - "github.com/stretchr/testify/require" - "go.opentelemetry.io/collector/pdata/ptrace" - "go.opentelemetry.io/collector/processor/batchprocessor" ) // Test performs a basic integration test which runs the @@ -53,10 +54,7 @@ func Test(t *testing.T) { go func() { exports := ctrl.Exports().(otelcol.ConsumerExports) - bo := backoff.New(ctx, backoff.Config{ - MinBackoff: 10 * time.Millisecond, - MaxBackoff: 100 * time.Millisecond, - }) + bo := backoff.New(ctx, testBackoffConfig()) for bo.Ongoing() { err := exports.Input.ConsumeTraces(ctx, createTestTraces()) if err != nil { @@ -78,6 +76,69 @@ func Test(t *testing.T) { } } +func Test_Update(t *testing.T) { + ctx := componenttest.TestContext(t) + + ctrl, err := componenttest.NewControllerFromID(util.TestLogger(t), "otelcol.processor.batch") + require.NoError(t, err) + + args := batch.Arguments{ + Timeout: 10 * time.Millisecond, + } + args.SetToDefault() + + // Override our arguments so traces get forwarded to traceCh. + traceCh := make(chan ptrace.Traces) + args.Output = makeTracesOutput(traceCh) + + go func() { + err := ctrl.Run(ctx, args) + require.NoError(t, err) + }() + + // Verify running and exported + require.NoError(t, ctrl.WaitRunning(time.Second), "component never started") + require.NoError(t, ctrl.WaitExports(time.Second), "component never exported anything") + + // Update the args + args.Timeout = 20 * time.Millisecond + require.NoError(t, ctrl.Update(args)) + + // Verify running + require.NoError(t, ctrl.WaitRunning(time.Second), "component never started") + + // Send traces in the background to our processor. + go func() { + exports := ctrl.Exports().(otelcol.ConsumerExports) + + bo := backoff.New(ctx, testBackoffConfig()) + for bo.Ongoing() { + err := exports.Input.ConsumeTraces(ctx, createTestTraces()) + if err != nil { + level.Error(util.TestLogger(t)).Log("msg", "failed to send traces", "err", err) + bo.Wait() + continue + } + return + } + }() + + // Wait for our processor to finish and forward data to traceCh. + select { + case <-time.After(time.Second): + require.FailNow(t, "failed waiting for traces") + case tr := <-traceCh: + require.Equal(t, 1, tr.SpanCount()) + } +} + +func testBackoffConfig() backoff.Config { + return backoff.Config{ + MinBackoff: 10 * time.Millisecond, + MaxBackoff: 100 * time.Millisecond, + } +} + // makeTracesOutput returns ConsumerArguments which will forward traces to the // provided channel. func makeTracesOutput(ch chan ptrace.Traces) *otelcol.ConsumerArguments { diff --git a/internal/component/otelcol/processor/processor.go b/internal/component/otelcol/processor/processor.go index 5aaa4750b3..5072d65233 100644 --- a/internal/component/otelcol/processor/processor.go +++ b/internal/component/otelcol/processor/processor.go @@ -7,6 +7,17 @@ import ( "errors" "os" + "github.com/prometheus/client_golang/prometheus" + otelcomponent "go.opentelemetry.io/collector/component" + "go.opentelemetry.io/collector/config/configtelemetry" + otelextension "go.opentelemetry.io/collector/extension" + "go.opentelemetry.io/collector/pipeline" + otelprocessor "go.opentelemetry.io/collector/processor" + sdkprometheus "go.opentelemetry.io/otel/exporters/prometheus" + otelmetric "go.opentelemetry.io/otel/metric" + "go.opentelemetry.io/otel/metric/noop" + "go.opentelemetry.io/otel/sdk/metric" + "github.com/grafana/alloy/internal/build" "github.com/grafana/alloy/internal/component" "github.com/grafana/alloy/internal/component/otelcol" @@ -18,16 +29,6 @@ import ( "github.com/grafana/alloy/internal/component/otelcol/internal/scheduler" "github.com/grafana/alloy/internal/service/livedebugging" "github.com/grafana/alloy/internal/util/zapadapter" - "github.com/prometheus/client_golang/prometheus" - otelcomponent "go.opentelemetry.io/collector/component" - "go.opentelemetry.io/collector/config/configtelemetry" - otelextension "go.opentelemetry.io/collector/extension" - "go.opentelemetry.io/collector/pipeline" - otelprocessor "go.opentelemetry.io/collector/processor" - sdkprometheus "go.opentelemetry.io/otel/exporters/prometheus" - otelmetric "go.opentelemetry.io/otel/metric" - "go.opentelemetry.io/otel/metric/noop" - "go.opentelemetry.io/otel/sdk/metric" ) // Arguments is an extension of component.Arguments which contains necessary @@ -94,7 +95,7 @@ func New(opts component.Options, f otelprocessor.Factory, args Arguments) (*Proc ctx, cancel := context.WithCancel(context.Background()) - consumer := lazyconsumer.New(ctx) + consumer := lazyconsumer.NewPaused(ctx) // Create a lazy collector where metrics from the upstream component will be // forwarded. @@ -116,7 +117,7 @@ func New(opts component.Options, f otelprocessor.Factory, args Arguments) (*Proc factory: f, consumer: consumer, - sched: scheduler.New(opts.Logger), + sched: scheduler.NewWithPauseCallbacks(opts.Logger, consumer.Pause, consumer.Resume), collector: collector, liveDebuggingConsumer: livedebuggingconsumer.New(debugDataPublisher.(livedebugging.DebugDataPublisher), opts.ID), From d3bea5797b3f6ed6933381e3828f7adb39a0b609 Mon Sep 17 00:00:00 2001 From: William Dumont Date: Fri, 15 Nov 2024 13:32:23 +0100 Subject: [PATCH 17/32] Change the stability of the livedebugging feature from experimental to generally available (#2104) --- CHANGELOG.md | 4 ++++ docs/sources/reference/config-blocks/livedebugging.md | 4 ---- internal/service/livedebugging/service.go | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 28fc106bc5..1f1b0832c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,10 @@ Main (unreleased) - Fixed an issue in the `prometheus.exporter.postgres` component that would leak goroutines when the target was not reachable (@dehaansa) +### Other changes + +- Change the stability of the `livedebugging` feature from "experimental" to "generally available". (@wildum) + v1.5.0 ----------------- diff --git a/docs/sources/reference/config-blocks/livedebugging.md b/docs/sources/reference/config-blocks/livedebugging.md index b1dee2763c..4587d95117 100644 --- a/docs/sources/reference/config-blocks/livedebugging.md +++ b/docs/sources/reference/config-blocks/livedebugging.md @@ -5,12 +5,8 @@ menuTitle: livedebugging title: livedebugging block --- -Experimental - # livedebugging block -{{< docs/shared lookup="stability/experimental.md" source="alloy" version="" >}} - `livedebugging` is an optional configuration block that enables the [live debugging feature][debug], which streams real-time data from your components directly to the {{< param "PRODUCT_NAME" >}} UI. By default, [live debugging][debug] is disabled and must be explicitly enabled through this configuration block to make the debugging data visible in the {{< param "PRODUCT_NAME" >}} UI. diff --git a/internal/service/livedebugging/service.go b/internal/service/livedebugging/service.go index 422b0230db..75e0c08cd7 100644 --- a/internal/service/livedebugging/service.go +++ b/internal/service/livedebugging/service.go @@ -39,7 +39,7 @@ func (*Service) Definition() service.Definition { Name: ServiceName, ConfigType: Arguments{}, DependsOn: []string{}, - Stability: featuregate.StabilityExperimental, + Stability: featuregate.StabilityGenerallyAvailable, } } From e9472bd9ad28e56c7392e34166260a89b118663f Mon Sep 17 00:00:00 2001 From: William Dumont Date: Fri, 15 Nov 2024 14:05:26 +0100 Subject: [PATCH 18/32] Add otelcol.receiver.solace (#2084) * Add otelcol.receiver.solace * fix test with ptr check to support empty structs * run doc generator * validate only one method of auth * Update docs/sources/reference/components/otelcol/otelcol.receiver.solace.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.receiver.solace.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.receiver.solace.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * doc nit * Update docs/sources/reference/components/otelcol/otelcol.receiver.solace.md Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> * Update internal/component/otelcol/receiver/solace/solace.go Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> * update error msg in test --------- Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> --- CHANGELOG.md | 2 + .../sources/reference/compatibility/_index.md | 1 + .../otelcol/otelcol.receiver.solace.md | 209 ++++++++++++++++ go.mod | 2 + go.sum | 4 + internal/component/all/all.go | 1 + internal/component/all/all_test.go | 4 + .../otelcol/receiver/solace/config_solace.go | 98 ++++++++ .../otelcol/receiver/solace/solace.go | 159 ++++++++++++ .../otelcol/receiver/solace/solace_test.go | 226 ++++++++++++++++++ .../converter_solacereceiver.go | 120 ++++++++++ .../otelcolconvert/testdata/solace.alloy | 31 +++ .../otelcolconvert/testdata/solace.yaml | 24 ++ 13 files changed, 881 insertions(+) create mode 100644 docs/sources/reference/components/otelcol/otelcol.receiver.solace.md create mode 100644 internal/component/otelcol/receiver/solace/config_solace.go create mode 100644 internal/component/otelcol/receiver/solace/solace.go create mode 100644 internal/component/otelcol/receiver/solace/solace_test.go create mode 100644 internal/converter/internal/otelcolconvert/converter_solacereceiver.go create mode 100644 internal/converter/internal/otelcolconvert/testdata/solace.alloy create mode 100644 internal/converter/internal/otelcolconvert/testdata/solace.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f1b0832c5..558a667d09 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,8 @@ Main (unreleased) - Add `otelcol.exporter.splunkhec` allowing to export otel data to Splunk HEC (@adlotsof) +- Add `otelcol.receiver.solace` component to receive traces from a Solace broker. (@wildum) + ### Bugfixes - Fixed an issue in the `prometheus.exporter.postgres` component that would leak goroutines when the target was not reachable (@dehaansa) diff --git a/docs/sources/reference/compatibility/_index.md b/docs/sources/reference/compatibility/_index.md index d55681be97..84830dabf3 100644 --- a/docs/sources/reference/compatibility/_index.md +++ b/docs/sources/reference/compatibility/_index.md @@ -359,6 +359,7 @@ The following components, grouped by namespace, _consume_ OpenTelemetry `otelcol - [otelcol.receiver.opencensus](../components/otelcol/otelcol.receiver.opencensus) - [otelcol.receiver.otlp](../components/otelcol/otelcol.receiver.otlp) - [otelcol.receiver.prometheus](../components/otelcol/otelcol.receiver.prometheus) +- [otelcol.receiver.solace](../components/otelcol/otelcol.receiver.solace) - [otelcol.receiver.vcenter](../components/otelcol/otelcol.receiver.vcenter) - [otelcol.receiver.zipkin](../components/otelcol/otelcol.receiver.zipkin) {{< /collapse >}} diff --git a/docs/sources/reference/components/otelcol/otelcol.receiver.solace.md b/docs/sources/reference/components/otelcol/otelcol.receiver.solace.md new file mode 100644 index 0000000000..2df0cd9133 --- /dev/null +++ b/docs/sources/reference/components/otelcol/otelcol.receiver.solace.md @@ -0,0 +1,209 @@ +--- +canonical: https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.receiver.solace/ +description: Learn about otelcol.receiver.solace +title: otelcol.receiver.solace +--- + +# otelcol.receiver.solace + +`otelcol.receiver.solace` accepts traces from a [Solace PubSub+ Event Broker](https://solace.com/products/event-broker/) and +forwards it to other `otelcol.*` components. + +{{< admonition type="note" >}} +`otelcol.receiver.solace` is a wrapper over the upstream OpenTelemetry Collector `solace` receiver from the `otelcol-contrib` distribution. +Bug reports or feature requests will be redirected to the upstream repository, if necessary. +{{< /admonition >}} + +You can specify multiple `otelcol.receiver.solace` components by giving them different labels. + +## Usage + +```alloy +otelcol.receiver.solace "LABEL" { + queue = "QUEUE" + auth { + // sasl_plain or sasl_xauth2 or sasl_external block + } + output { + traces = [...] + } +} +``` + +## Arguments + +The following arguments are supported: + +| Name | Type | Description | Default | Required | +| -------------------- | -------- | ------------------------------------------------------------------------- | ---------------- | -------- | +| `queue` | `string` | Name of the Solace telemetry queue to get span trace messages from. | | yes | +| `broker` | `string` | Name of the Solace broker using AMQP over TLS. | `localhost:5671` | no | +| `max_unacknowledged` | `int` | Maximum number of unacknowledged messages the Solace broker can transmit. | 10 | no | + +`queue` must have the format `queue://#telemetry-myTelemetryProfile`. + +## Blocks + +The following blocks are supported inside the definition of +`otelcol.receiver.solace`: + +| Hierarchy | Block | Description | Required | +| ------------------------------ | ------------------ | -------------------------------------------------------------------------------------------------------------------------------- | -------- | +| authentication | [authentication][] | Configures authentication for connecting to the Solace broker. | yes | +| authentication > sasl_plain | [sasl_plain][] | Authenticates against the Solace broker with SASL PLAIN. | no | +| authentication > sasl_xauth2 | [sasl_xauth2][] | Authenticates against the Solace broker with SASL XOauth2. | no | +| authentication > sasl_external | [sasl_external][] | Authenticates against the Solace broker with SASL External. | no | +| flow | [flow][] | Configures the behaviour to use when temporary errors are encountered from the next component. | no | +| flow > delayed_retry | [delayed_retry][] | Sets the flow control strategy to `delayed retry` which will wait before trying to push the message to the next component again. | no | +| tls | [tls][] | Configures TLS for connecting to the Solace broker. | no | +| debug_metrics | [debug_metrics][] | Configures the metrics which this component generates to monitor its state. | no | +| output | [output][] | Configures where to send received telemetry data. | yes | + +One SASL authentication block is required in the `authentication` block. + +`sasl_external` must be used together with the `tls` block. + +The `>` symbol indicates deeper levels of nesting. For example, +`authentication > tls` refers to a `tls` block defined inside an +`authentication` block. + +[authentication]: #authentication-block +[sasl_plain]: #sasl_plain-block +[sasl_xauth2]: #sasl_xauth2-block +[sasl_external]: #sasl_external-block +[tls]: #tls-block +[flow]: #flow-block +[delayed_retry]: #delayed_retry-block +[debug_metrics]: #debug_metrics-block +[output]: #output-block + +### authentication block + +The `authentication` block configures how to authenticate for connecting to the Solace broker. +It doesn't support any arguments and is configured fully through inner blocks. + +### sasl_plain block + +The `sasl_plain` block configures how to authenticate to the Solace broker with SASL PLAIN. + +The following arguments are supported: + +| Name | Type | Description | Default | Required | +| ---------- | -------- | -------------------- | ------- | -------- | +| `username` | `string` | The username to use. | | yes | +| `password` | `string` | The password to use. | | yes | + +### sasl_xauth2 block + +The `sasl_xauth2` block configures how to authenticate to the Solace broker with SASL XOauth2. + +The following arguments are supported: + +| Name | Type | Description | Default | Required | +| ---------- | -------- | ------------------------- | ------- | -------- | +| `username` | `string` | The username to use. | | yes | +| `bearer` | `string` | The bearer in plain text. | | yes | + +### sasl_external block + +The `sasl_xauth2` block configures how to authenticate to the Solace broker with SASL External. +It doesn't support any arguments or blocks. It must be used with the [tls][] block. + +### flow block + +The `flow` block configures the behaviour to use when temporary errors are encountered from the next component. +It doesn't support any arguments and is configured fully through inner blocks. + +### delayed_retry block + +The `delayed_retry` block sets the flow control strategy to `delayed retry` which will wait before trying to push the message to the next component again. + +The following arguments are supported: + +| Name | Type | Description | Default | Required | +| ------- | -------- | --------------------------------- | -------- | -------- | +| `delay` | `string` | The time to wait before retrying. | `"10ms"` | no | + +### tls block + +{{< docs/shared lookup="reference/components/otelcol-tls-client-block.md" source="alloy" version="" >}} + +### debug_metrics block + +{{< docs/shared lookup="reference/components/otelcol-debug-metrics-block.md" source="alloy" version="" >}} + +### output block + +{{< docs/shared lookup="reference/components/output-block.md" source="alloy" version="" >}} + +{{< admonition type="warning" >}} +Having multiple consumers may result in duplicated traces in case of errors because of the retry strategy. +It is recommended to only set one consumer for this component. +{{< /admonition >}} + +## Exported fields + +`otelcol.receiver.solace` does not export any fields. + +## Component health + +`otelcol.receiver.solace` is only reported as unhealthy if given an invalid +configuration. + +## Debug information + +`otelcol.receiver.solace` does not expose any component-specific debug +information. + +## Example + +This example forwards read telemetry data through a batch processor before +finally sending it to an OTLP-capable endpoint: + +```alloy +otelcol.receiver.solace "default" { + queue = "queue://#telemetry-testprofile" + broker = "localhost:5672" + auth { + sasl_plain { + username = "alloy" + password = "password" + } + } + tls { + insecure = true + insecure_skip_verify = true + } + output { + traces = [otelcol.processor.batch.default.input] + } +} + +otelcol.processor.batch "default" { + output { + traces = [otelcol.exporter.otlp.default.input] + } +} + +otelcol.exporter.otlp "default" { + client { + endpoint = sys.env("OTLP_ENDPOINT") + } +} +``` + + + +## Compatible components + +`otelcol.receiver.solace` can accept arguments from the following components: + +- Components that export [OpenTelemetry `otelcol.Consumer`](../../../compatibility/#opentelemetry-otelcolconsumer-exporters) + + +{{< admonition type="note" >}} +Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. +Refer to the linked documentation for more details. +{{< /admonition >}} + + diff --git a/go.mod b/go.mod index 1f1e516f04..41fa2a8f76 100644 --- a/go.mod +++ b/go.mod @@ -138,6 +138,7 @@ require ( github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jaegerreceiver v0.112.0 github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kafkareceiver v0.112.0 github.com/open-telemetry/opentelemetry-collector-contrib/receiver/opencensusreceiver v0.112.0 + github.com/open-telemetry/opentelemetry-collector-contrib/receiver/solacereceiver v0.112.0 github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver v0.112.0 github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver v0.112.0 github.com/ory/dockertest/v3 v3.8.1 @@ -821,6 +822,7 @@ require ( ) require ( + github.com/Azure/go-amqp v1.2.0 // indirect github.com/DataDog/datadog-agent/comp/core/log/def v0.57.1 // indirect github.com/antchfx/xmlquery v1.4.2 // indirect github.com/antchfx/xpath v1.3.2 // indirect diff --git a/go.sum b/go.sum index 5aa02756c4..6a139e700d 100644 --- a/go.sum +++ b/go.sum @@ -117,6 +117,8 @@ github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.2.0 h1:gggzg0SUMs6SQbEw+ github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.2.0/go.mod h1:+6KLcKIVgxoBDMqMO/Nvy7bZ9a0nbU3I1DtFQK3YvB4= github.com/Azure/azure-storage-queue-go v0.0.0-20181215014128-6ed74e755687/go.mod h1:K6am8mT+5iFXgingS9LUc7TmbsW6XBw3nxaRyaMyWc8= github.com/Azure/go-amqp v0.12.6/go.mod h1:qApuH6OFTSKZFmCOxccvAv5rLizBQf4v8pRmG138DPo= +github.com/Azure/go-amqp v1.2.0 h1:NNyfN3/cRszWzMvjmm64yaPZDHX/2DJkowv8Ub9y01I= +github.com/Azure/go-amqp v1.2.0/go.mod h1:vZAogwdrkbyK3Mla8m/CxSc/aKdnTZ4IbPxl51Y5WZE= github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8= github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 h1:L/gRVlceqvL25UVaW/CKtUDjefjrs0SPonmDGUVOYP0= github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E= @@ -2047,6 +2049,8 @@ github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kafkareceiver github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kafkareceiver v0.112.0/go.mod h1:v61OP6Zxu8Kx4WLuswY06uaUPiOWDt5ykW9vqNQJNXc= github.com/open-telemetry/opentelemetry-collector-contrib/receiver/opencensusreceiver v0.112.0 h1:X/eUpWLWBZg2fDT+jWZiIept45akvKbZXhktg1x86gE= github.com/open-telemetry/opentelemetry-collector-contrib/receiver/opencensusreceiver v0.112.0/go.mod h1:q2lFBHfnG+ar2DJJlIU6RviOFXDeFur9vJ083NvOMQs= +github.com/open-telemetry/opentelemetry-collector-contrib/receiver/solacereceiver v0.112.0 h1:cHk8vS/D1pjeZ0o4LJJAENP847HHWjTXFe4y1RJYlfo= +github.com/open-telemetry/opentelemetry-collector-contrib/receiver/solacereceiver v0.112.0/go.mod h1:2CK7Hh6UGLnBSGW7Y0nopvEhoo25D6t/395jFEephEs= github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver v0.112.0 h1:Vv1FDwd7pykzj8Wmuc7yj7bcN0qUv1mGBb/dcTMPfNE= github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver v0.112.0/go.mod h1:lklLK8ELD2Wk5z7ywjaf6XEbbViDtf7uK8jAExjRlls= github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver v0.112.0 h1:XhKHjEpQJQMaUuWVhWS1FEuaY4LJDwBgsGXE166j9SY= diff --git a/internal/component/all/all.go b/internal/component/all/all.go index 4f58d9fbe6..caa0107764 100644 --- a/internal/component/all/all.go +++ b/internal/component/all/all.go @@ -101,6 +101,7 @@ import ( _ "github.com/grafana/alloy/internal/component/otelcol/receiver/opencensus" // Import otelcol.receiver.opencensus _ "github.com/grafana/alloy/internal/component/otelcol/receiver/otlp" // Import otelcol.receiver.otlp _ "github.com/grafana/alloy/internal/component/otelcol/receiver/prometheus" // Import otelcol.receiver.prometheus + _ "github.com/grafana/alloy/internal/component/otelcol/receiver/solace" // Import otelcol.receiver.solace _ "github.com/grafana/alloy/internal/component/otelcol/receiver/vcenter" // Import otelcol.receiver.vcenter _ "github.com/grafana/alloy/internal/component/otelcol/receiver/zipkin" // Import otelcol.receiver.zipkin _ "github.com/grafana/alloy/internal/component/prometheus/exporter/apache" // Import prometheus.exporter.apache diff --git a/internal/component/all/all_test.go b/internal/component/all/all_test.go index fe28c7739a..a0cc0ef448 100644 --- a/internal/component/all/all_test.go +++ b/internal/component/all/all_test.go @@ -99,6 +99,10 @@ func sharePointer(a, b reflect.Value) (string, bool) { return "", false case reflect.Pointer: + // same edge case as above: skip if this is a struct ptr and that the structs are empty + if !a.IsNil() && a.Elem().Kind() == reflect.Struct && a.Elem().NumField() == 0 { + return "", false + } if pointersMatch(a, b) { return "", true } else { diff --git a/internal/component/otelcol/receiver/solace/config_solace.go b/internal/component/otelcol/receiver/solace/config_solace.go new file mode 100644 index 0000000000..b5ed417456 --- /dev/null +++ b/internal/component/otelcol/receiver/solace/config_solace.go @@ -0,0 +1,98 @@ +package solace + +import ( + "time" + + "github.com/grafana/alloy/syntax/alloytypes" + "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/solacereceiver" + "go.opentelemetry.io/collector/config/configopaque" +) + +// Authentication defines authentication strategies. +type Authentication struct { + PlainText *SaslPlainTextConfig `alloy:"sasl_plain,block,optional"` + XAuth2 *SaslXAuth2Config `alloy:"sasl_xauth2,block,optional"` + External *SaslExternalConfig `alloy:"sasl_external,block,optional"` +} + +// Convert converts args into the upstream type. +func (args Authentication) Convert() solacereceiver.Authentication { + auth := solacereceiver.Authentication{} + + if args.PlainText != nil { + auth.PlainText = args.PlainText.Convert() + } + if args.XAuth2 != nil { + auth.XAuth2 = args.XAuth2.Convert() + } + if args.External != nil { + auth.External = args.External.Convert() + } + + return auth +} + +// SaslPlainTextConfig defines SASL PLAIN authentication. +type SaslPlainTextConfig struct { + Username string `alloy:"username,attr"` + Password alloytypes.Secret `alloy:"password,attr"` +} + +func (args SaslPlainTextConfig) Convert() *solacereceiver.SaslPlainTextConfig { + return &solacereceiver.SaslPlainTextConfig{ + Username: args.Username, + Password: configopaque.String(args.Password), + } +} + +// SaslXAuth2Config defines the configuration for the SASL XAUTH2 authentication. +type SaslXAuth2Config struct { + Username string `alloy:"username,attr"` + Bearer string `alloy:"bearer,attr"` +} + +func (args SaslXAuth2Config) Convert() *solacereceiver.SaslXAuth2Config { + return &solacereceiver.SaslXAuth2Config{ + Username: args.Username, + Bearer: args.Bearer, + } +} + +// SaslExternalConfig defines the configuration for the SASL External used in conjunction with TLS client authentication. +type SaslExternalConfig struct{} + +func (args SaslExternalConfig) Convert() *solacereceiver.SaslExternalConfig { + return &solacereceiver.SaslExternalConfig{} +} + +// FlowControl defines the configuration for what to do in backpressure scenarios, e.g. memorylimiter errors +type FlowControl struct { + DelayedRetry *FlowControlDelayedRetry `alloy:"delayed_retry,block"` +} + +func (args FlowControl) Convert() solacereceiver.FlowControl { + flowControl := solacereceiver.FlowControl{} + if args.DelayedRetry != nil { + flowControl.DelayedRetry = args.DelayedRetry.Convert() + } + return flowControl +} + +func (args *FlowControl) SetToDefault() { + *args = FlowControl{ + DelayedRetry: &FlowControlDelayedRetry{ + Delay: 10 * time.Millisecond, + }, + } +} + +// FlowControlDelayedRetry represents the strategy of waiting for a defined amount of time (in time.Duration) and attempt redelivery +type FlowControlDelayedRetry struct { + Delay time.Duration `alloy:"delay,attr,optional"` +} + +func (args FlowControlDelayedRetry) Convert() *solacereceiver.FlowControlDelayedRetry { + return &solacereceiver.FlowControlDelayedRetry{ + Delay: args.Delay, + } +} diff --git a/internal/component/otelcol/receiver/solace/solace.go b/internal/component/otelcol/receiver/solace/solace.go new file mode 100644 index 0000000000..5895d1c9bd --- /dev/null +++ b/internal/component/otelcol/receiver/solace/solace.go @@ -0,0 +1,159 @@ +// Package solace provides an otelcol.receiver.solace component. +package solace + +/* +How to test solace manually: +- Use the docker compose template here: https://github.com/SolaceLabs/solace-single-docker-compose/blob/master/template/PubSubStandard_singleNode.yml +- Log in to http://localhost:8080/ to configure solace and select default +- Go to Telemetry, enable it, create a profile "testprofile" and enable both trace and receiver +- Telemetry > Receiver connect ACLs > Client Connect Default Action: set it to "Allow" via the edit button +- Telemetry > Trace Filters: create a trace filter "testfilter" and enable it +- Click on the test filter, go to Subscriptions and create a new Subscription ">" +- In Queues, create a new queue "testqueue" +- Click on "testqueue" and create a subscription "solace/tracing" +- Access Control > Client Authentication: set the Type to "Internal database" +- Access Control > Client Username: create two clients: + - "alloy", with all toggle enabled, the password set to "alloy" and the client profile and acl profile set to #telemetry-testprofile + - "bob", with all toggle and the password set to "bob". Keep the client profile and acl profile set to default +- Connect Alloy with the following config: + otelcol.receiver.solace "solace" { + queue = "queue://#telemetry-testprofile" + broker = "localhost:5672" + auth { + sasl_plain { + username = "alloy" + password = "alloy" + } + } + tls { + insecure = true + insecure_skip_verify = true + } + output { + traces = [otelcol.exporter.debug.solace.input] + } + } + + otelcol.exporter.debug "solace" { + verbosity = "detailed" + } + + logging { + level = "debug" + } +- In "Try Me!", connect as "bob" (username and password) +- You should be able to see the span in the terminal +*/ + +import ( + "fmt" + "strings" + + "github.com/grafana/alloy/internal/component" + "github.com/grafana/alloy/internal/component/otelcol" + otelcolCfg "github.com/grafana/alloy/internal/component/otelcol/config" + "github.com/grafana/alloy/internal/component/otelcol/receiver" + "github.com/grafana/alloy/internal/featuregate" + "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/solacereceiver" + otelcomponent "go.opentelemetry.io/collector/component" + otelextension "go.opentelemetry.io/collector/extension" + "go.opentelemetry.io/collector/pipeline" +) + +func init() { + component.Register(component.Registration{ + Name: "otelcol.receiver.solace", + Stability: featuregate.StabilityGenerallyAvailable, + Args: Arguments{}, + + Build: func(opts component.Options, args component.Arguments) (component.Component, error) { + fact := solacereceiver.NewFactory() + return receiver.New(opts, fact, args.(Arguments)) + }, + }) +} + +// Arguments configures the otelcol.receiver.solace component. +type Arguments struct { + // The upstream component uses a list for the broker but they only use the first element in the list so I decided to use + // a simple string in Alloy to avoid confusing the users. + Broker string `alloy:"broker,attr,optional"` + Queue string `alloy:"queue,attr"` + MaxUnacked int32 `alloy:"max_unacknowledged,attr,optional"` + + TLS otelcol.TLSClientArguments `alloy:"tls,block,optional"` + Flow FlowControl `alloy:"flow_control,block,optional"` + DebugMetrics otelcolCfg.DebugMetricsArguments `alloy:"debug_metrics,block,optional"` + Auth Authentication `alloy:"auth,block"` + + // Output configures where to send received data. Required. + Output *otelcol.ConsumerArguments `alloy:"output,block"` +} + +var _ receiver.Arguments = Arguments{} + +// SetToDefault implements syntax.Defaulter. +func (args *Arguments) SetToDefault() { + *args = Arguments{ + Broker: "localhost:5671", + MaxUnacked: 1000, + } + args.Flow.SetToDefault() + args.DebugMetrics.SetToDefault() +} + +// Validate implements syntax.Validator. +func (args *Arguments) Validate() error { + authMethod := 0 + if args.Auth.PlainText != nil { + authMethod++ + } + if args.Auth.External != nil { + authMethod++ + } + if args.Auth.XAuth2 != nil { + authMethod++ + } + if authMethod != 1 { + return fmt.Errorf("the auth block must contain exactly one of sasl_plain block, sasl_xauth2 block or sasl_external block") + } + if len(strings.TrimSpace(args.Queue)) == 0 { + return fmt.Errorf("queue must not be empty, queue definition has format queue://") + } + if args.Flow.DelayedRetry != nil && args.Flow.DelayedRetry.Delay <= 0 { + return fmt.Errorf("the delay attribute in the delayed_retry block must be > 0, got %d", args.Flow.DelayedRetry.Delay) + } + return nil +} + +// Convert implements receiver.Arguments. +func (args Arguments) Convert() (otelcomponent.Config, error) { + return &solacereceiver.Config{ + Broker: []string{args.Broker}, + Queue: args.Queue, + MaxUnacked: args.MaxUnacked, + TLS: *args.TLS.Convert(), + Auth: args.Auth.Convert(), + Flow: args.Flow.Convert(), + }, nil +} + +// Extensions implements receiver.Arguments. +func (args Arguments) Extensions() map[otelcomponent.ID]otelextension.Extension { + return nil +} + +// Exporters implements receiver.Arguments. +func (args Arguments) Exporters() map[pipeline.Signal]map[otelcomponent.ID]otelcomponent.Component { + return nil +} + +// NextConsumers implements receiver.Arguments. +func (args Arguments) NextConsumers() *otelcol.ConsumerArguments { + return args.Output +} + +// DebugMetricsConfig implements receiver.Arguments. +func (args Arguments) DebugMetricsConfig() otelcolCfg.DebugMetricsArguments { + return args.DebugMetrics +} diff --git a/internal/component/otelcol/receiver/solace/solace_test.go b/internal/component/otelcol/receiver/solace/solace_test.go new file mode 100644 index 0000000000..fc1fb6f708 --- /dev/null +++ b/internal/component/otelcol/receiver/solace/solace_test.go @@ -0,0 +1,226 @@ +package solace_test + +import ( + "testing" + "time" + + "github.com/grafana/alloy/internal/component/otelcol/receiver/solace" + "github.com/grafana/alloy/syntax" + "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/solacereceiver" + "github.com/stretchr/testify/require" + "go.opentelemetry.io/collector/config/configtls" +) + +func TestArguments_UnmarshalAlloy(t *testing.T) { + tests := []struct { + testName string + cfg string + expected solacereceiver.Config + }{ + { + testName: "Defaults", + cfg: ` + queue = "queue://#telemetry_testprofile" + auth { + sasl_plain { + username = "alloy" + password = "password" + } + } + output {} + `, + expected: solacereceiver.Config{ + Broker: []string{"localhost:5671"}, + Queue: "queue://#telemetry_testprofile", + MaxUnacked: 1000, + Flow: solacereceiver.FlowControl{ + DelayedRetry: &solacereceiver.FlowControlDelayedRetry{ + Delay: 10 * time.Millisecond, + }, + }, + Auth: solacereceiver.Authentication{ + PlainText: &solacereceiver.SaslPlainTextConfig{ + Username: "alloy", + Password: "password", + }, + }, + }, + }, + { + testName: "Explicit Values - External / TLS", + cfg: ` + broker = "localhost:5672" + max_unacknowledged = 500 + queue = "queue://#telemetry_testprofile" + auth { + sasl_external {} + } + tls { + cert_file = "testdata/test-cert.crt" + key_file = "testdata/test-key.key" + } + flow_control { + delayed_retry { + delay = "50ms" + } + } + output {} + `, + expected: solacereceiver.Config{ + Broker: []string{"localhost:5672"}, + Queue: "queue://#telemetry_testprofile", + MaxUnacked: 500, + Flow: solacereceiver.FlowControl{ + DelayedRetry: &solacereceiver.FlowControlDelayedRetry{ + Delay: 50 * time.Millisecond, + }, + }, + Auth: solacereceiver.Authentication{ + External: &solacereceiver.SaslExternalConfig{}, + }, + TLS: configtls.ClientConfig{ + Config: configtls.Config{ + CertFile: "testdata/test-cert.crt", + KeyFile: "testdata/test-key.key", + }, + }, + }, + }, + { + testName: "Explicit Values - XAuth2 / TLS", + cfg: ` + broker = "localhost:5672" + max_unacknowledged = 500 + queue = "queue://#telemetry_testprofile" + auth { + sasl_xauth2 { + username = "alloy" + bearer = "bearer" + } + } + tls { + cert_file = "testdata/test-cert.crt" + key_file = "testdata/test-key.key" + } + flow_control { + delayed_retry { + delay = "50ms" + } + } + output {} + `, + expected: solacereceiver.Config{ + Broker: []string{"localhost:5672"}, + Queue: "queue://#telemetry_testprofile", + MaxUnacked: 500, + Flow: solacereceiver.FlowControl{ + DelayedRetry: &solacereceiver.FlowControlDelayedRetry{ + Delay: 50 * time.Millisecond, + }, + }, + Auth: solacereceiver.Authentication{ + XAuth2: &solacereceiver.SaslXAuth2Config{ + Username: "alloy", + Bearer: "bearer", + }, + }, + TLS: configtls.ClientConfig{ + Config: configtls.Config{ + CertFile: "testdata/test-cert.crt", + KeyFile: "testdata/test-key.key", + }, + }, + }, + }, + } + + for _, tc := range tests { + t.Run(tc.testName, func(t *testing.T) { + var args solace.Arguments + err := syntax.Unmarshal([]byte(tc.cfg), &args) + require.NoError(t, err) + + actualPtr, err := args.Convert() + require.NoError(t, err) + + actual := actualPtr.(*solacereceiver.Config) + + require.Equal(t, tc.expected, *actual) + }) + } +} + +func TestArguments_Validate(t *testing.T) { + tests := []struct { + testName string + cfg string + expectedError string + }{ + { + testName: "Missing Auth", + cfg: ` + queue = "queue://#telemetry_testprofile" + auth {} + output {} + `, + expectedError: "the auth block must contain exactly one of sasl_plain block, sasl_xauth2 block or sasl_external block", + }, + { + testName: "Multiple Auth", + cfg: ` + queue = "queue://#telemetry_testprofile" + auth { + sasl_plain { + username = "alloy" + password = "password" + } + sasl_xauth2 { + username = "alloy" + bearer = "bearer" + } + } + output {} + `, + expectedError: "the auth block must contain exactly one of sasl_plain block, sasl_xauth2 block or sasl_external block", + }, + { + testName: "Empty Queue", + cfg: ` + queue = "" + auth { + sasl_plain { + username = "alloy" + password = "password" + } + } + output {} + `, + expectedError: "queue must not be empty, queue definition has format queue://", + }, + { + testName: "Wrong value for delay in delayed_retry block", + cfg: ` + queue = "queue://#telemetry_testprofile" + auth { + sasl_plain { + username = "alloy" + password = "password" + } + } + flow_control { + delayed_retry { + delay = "0ms" + } + } + output {} + `, + expectedError: "the delay attribute in the delayed_retry block must be > 0, got 0", + }, + } + for _, tc := range tests { + t.Run(tc.testName, func(t *testing.T) { + var args solace.Arguments + require.ErrorContains(t, syntax.Unmarshal([]byte(tc.cfg), &args), tc.expectedError) + }) + } +} diff --git a/internal/converter/internal/otelcolconvert/converter_solacereceiver.go b/internal/converter/internal/otelcolconvert/converter_solacereceiver.go new file mode 100644 index 0000000000..435d95b4dd --- /dev/null +++ b/internal/converter/internal/otelcolconvert/converter_solacereceiver.go @@ -0,0 +1,120 @@ +package otelcolconvert + +import ( + "fmt" + + "github.com/grafana/alloy/internal/component/otelcol" + "github.com/grafana/alloy/internal/component/otelcol/receiver/solace" + "github.com/grafana/alloy/internal/converter/diag" + "github.com/grafana/alloy/internal/converter/internal/common" + "github.com/grafana/alloy/syntax/alloytypes" + "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/solacereceiver" + "go.opentelemetry.io/collector/component" + "go.opentelemetry.io/collector/component/componentstatus" + "go.opentelemetry.io/collector/pipeline" +) + +func init() { + converters = append(converters, solaceReceiverConverter{}) +} + +type solaceReceiverConverter struct{} + +func (solaceReceiverConverter) Factory() component.Factory { return solacereceiver.NewFactory() } + +func (solaceReceiverConverter) InputComponentName() string { return "" } + +func (solaceReceiverConverter) ConvertAndAppend(state *State, id componentstatus.InstanceID, cfg component.Config) diag.Diagnostics { + var diags diag.Diagnostics + + label := state.AlloyComponentLabel() + + args := toSolaceReceiver(state, id, cfg.(*solacereceiver.Config)) + block := common.NewBlockWithOverride([]string{"otelcol", "receiver", "solace"}, label, args) + + diags.Add( + diag.SeverityLevelInfo, + fmt.Sprintf("Converted %s into %s", StringifyInstanceID(id), StringifyBlock(block)), + ) + + state.Body().AppendBlock(block) + return diags +} + +func toSolaceReceiver(state *State, id componentstatus.InstanceID, cfg *solacereceiver.Config) *solace.Arguments { + nextTraces := state.Next(id, pipeline.SignalTraces) + + var broker string + if len(cfg.Broker) == 0 { + broker = "" + } else { + broker = cfg.Broker[0] + } + + return &solace.Arguments{ + Broker: broker, + Queue: cfg.Queue, + MaxUnacked: cfg.MaxUnacked, + + Auth: toSolaceAuthentication(cfg.Auth), + TLS: toTLSClientArguments(cfg.TLS), + Flow: toSolaceFlow(cfg.Flow), + DebugMetrics: common.DefaultValue[solace.Arguments]().DebugMetrics, + Output: &otelcol.ConsumerArguments{ + Traces: ToTokenizedConsumers(nextTraces), + }, + } +} + +func toSolaceAuthentication(cfg solacereceiver.Authentication) solace.Authentication { + return solace.Authentication{ + PlainText: toSaslPlaintext(cfg.PlainText), + XAuth2: toSaslXAuth2(cfg.XAuth2), + External: toSaslExternal(cfg.External), + } +} + +func toSaslPlaintext(cfg *solacereceiver.SaslPlainTextConfig) *solace.SaslPlainTextConfig { + if cfg == nil { + return nil + } + + return &solace.SaslPlainTextConfig{ + Username: cfg.Username, + Password: alloytypes.Secret(cfg.Password), + } +} + +func toSaslXAuth2(cfg *solacereceiver.SaslXAuth2Config) *solace.SaslXAuth2Config { + if cfg == nil { + return nil + } + + return &solace.SaslXAuth2Config{ + Username: cfg.Username, + Bearer: cfg.Bearer, + } +} + +func toSaslExternal(cfg *solacereceiver.SaslExternalConfig) *solace.SaslExternalConfig { + if cfg == nil { + return nil + } + + return &solace.SaslExternalConfig{} +} + +func toSolaceFlow(cfg solacereceiver.FlowControl) solace.FlowControl { + return solace.FlowControl{ + DelayedRetry: toFlowControlDelayedRetry(cfg.DelayedRetry), + } +} + +func toFlowControlDelayedRetry(cfg *solacereceiver.FlowControlDelayedRetry) *solace.FlowControlDelayedRetry { + if cfg == nil { + return nil + } + return &solace.FlowControlDelayedRetry{ + Delay: cfg.Delay, + } +} diff --git a/internal/converter/internal/otelcolconvert/testdata/solace.alloy b/internal/converter/internal/otelcolconvert/testdata/solace.alloy new file mode 100644 index 0000000000..7e088ea149 --- /dev/null +++ b/internal/converter/internal/otelcolconvert/testdata/solace.alloy @@ -0,0 +1,31 @@ +otelcol.receiver.solace "default" { + broker = "localhost:5672" + queue = "queue://#telemetry-profile123" + + tls { + insecure = true + } + + flow_control { + delayed_retry { + delay = "20ms" + } + } + + auth { + sasl_plain { + username = "otel" + password = "otel01$" + } + } + + output { + traces = [otelcol.exporter.otlp.default.input] + } +} + +otelcol.exporter.otlp "default" { + client { + endpoint = "database:4317" + } +} diff --git a/internal/converter/internal/otelcolconvert/testdata/solace.yaml b/internal/converter/internal/otelcolconvert/testdata/solace.yaml new file mode 100644 index 0000000000..dfaa5dc0ab --- /dev/null +++ b/internal/converter/internal/otelcolconvert/testdata/solace.yaml @@ -0,0 +1,24 @@ +receivers: + solace: + broker: [localhost:5672] + auth: + sasl_plain: + username: otel + password: otel01$ + flow_control: + delayed_retry: + delay: 20ms + tls: + insecure: true + queue: queue://#telemetry-profile123 + +exporters: + otlp: + endpoint: database:4317 + +service: + pipelines: + traces: + receivers: [solace] + processors: [] + exporters: [otlp] From a37e481f5acc2196a9f037d4e305cd8ba1b16514 Mon Sep 17 00:00:00 2001 From: mattdurham Date: Fri, 15 Nov 2024 09:27:52 -0500 Subject: [PATCH 19/32] Move wal queue to its own repository and minor bug fixes. (#1994) * Working on wal cleanup * use wrapped mailboxes * more safely use mailboxes. * Fix check that is no longer needed. * Cleanup * Add test for metrics. * remove check * fix check * lower threshold * The go func was bogging down tests. * Adjusting times. * Fix issue with items not being put back into the timeseries pool. * Add comment. * Remove unneeded test. * Use the same concepts. * Use the same concepts. * Switch to using the walqueue repo. * add changelog * Fix go.mod and empty file. * update go.mod * remove unneeded file and update to cleaner code for walqueue. * Remove race exclusion * Remove race exclusion * Update CHANGELOG.md Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> --------- Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com> --- CHANGELOG.md | 2 + Makefile | 2 +- go.mod | 14 +- go.sum | 10 +- .../datadog/config/config_datadog_test.go | 1 - .../prometheus/write/queue/component.go | 78 +- .../prometheus/write/queue/e2e_stats_test.go | 9 +- .../prometheus/write/queue/e2e_test.go | 7 +- .../prometheus/write/queue/endpoint.go | 133 - .../write/queue/filequeue/filequeue.go | 191 - .../write/queue/filequeue/filequeue_test.go | 253 -- .../write/queue/filequeue/record.go | 11 - .../write/queue/filequeue/record_gen.go | 206 -- .../write/queue/filequeue/record_gen_test.go | 123 - .../write/queue/network/benchmark_test.go | 32 - .../prometheus/write/queue/network/loop.go | 371 -- .../prometheus/write/queue/network/manager.go | 199 - .../write/queue/network/manager_test.go | 311 -- .../prometheus/write/queue/network/stats.go | 126 - .../write/queue/serialization/appender.go | 130 - .../queue/serialization/appender_test.go | 55 - .../write/queue/serialization/serializer.go | 222 -- .../serialization/serializer_bench_test.go | 117 - .../queue/serialization/serializer_test.go | 113 - .../component/prometheus/write/queue/types.go | 4 +- .../prometheus/write/queue/types/messages.go | 12 - .../prometheus/write/queue/types/network.go | 42 - .../write/queue/types/serialization.go | 296 -- .../write/queue/types/serialization_gen.go | 3294 ----------------- .../queue/types/serialization_gen_test.go | 914 ----- .../write/queue/types/serialization_test.go | 59 - .../write/queue/types/serializer.go | 24 - .../prometheus/write/queue/types/stats.go | 289 -- .../prometheus/write/queue/types/storage.go | 11 - .../write/queue/types/storage_test.go | 24 - 35 files changed, 63 insertions(+), 7622 deletions(-) delete mode 100644 internal/component/prometheus/write/queue/endpoint.go delete mode 100644 internal/component/prometheus/write/queue/filequeue/filequeue.go delete mode 100644 internal/component/prometheus/write/queue/filequeue/filequeue_test.go delete mode 100644 internal/component/prometheus/write/queue/filequeue/record.go delete mode 100644 internal/component/prometheus/write/queue/filequeue/record_gen.go delete mode 100644 internal/component/prometheus/write/queue/filequeue/record_gen_test.go delete mode 100644 internal/component/prometheus/write/queue/network/benchmark_test.go delete mode 100644 internal/component/prometheus/write/queue/network/loop.go delete mode 100644 internal/component/prometheus/write/queue/network/manager.go delete mode 100644 internal/component/prometheus/write/queue/network/manager_test.go delete mode 100644 internal/component/prometheus/write/queue/network/stats.go delete mode 100644 internal/component/prometheus/write/queue/serialization/appender.go delete mode 100644 internal/component/prometheus/write/queue/serialization/appender_test.go delete mode 100644 internal/component/prometheus/write/queue/serialization/serializer.go delete mode 100644 internal/component/prometheus/write/queue/serialization/serializer_bench_test.go delete mode 100644 internal/component/prometheus/write/queue/serialization/serializer_test.go delete mode 100644 internal/component/prometheus/write/queue/types/messages.go delete mode 100644 internal/component/prometheus/write/queue/types/network.go delete mode 100644 internal/component/prometheus/write/queue/types/serialization.go delete mode 100644 internal/component/prometheus/write/queue/types/serialization_gen.go delete mode 100644 internal/component/prometheus/write/queue/types/serialization_gen_test.go delete mode 100644 internal/component/prometheus/write/queue/types/serialization_test.go delete mode 100644 internal/component/prometheus/write/queue/types/serializer.go delete mode 100644 internal/component/prometheus/write/queue/types/stats.go delete mode 100644 internal/component/prometheus/write/queue/types/storage.go delete mode 100644 internal/component/prometheus/write/queue/types/storage_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 558a667d09..99a49d0eb2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,8 @@ Main (unreleased) - Fixed an issue in the `prometheus.exporter.postgres` component that would leak goroutines when the target was not reachable (@dehaansa) +- Fixed issue with reloading configuration and prometheus metrics duplication in `prometheus.write.queue`. (@mattdurham) + ### Other changes - Change the stability of the `livedebugging` feature from "experimental" to "generally available". (@wildum) diff --git a/Makefile b/Makefile index 579eea9a7d..b2a6146b55 100644 --- a/Makefile +++ b/Makefile @@ -141,7 +141,7 @@ lint: alloylint # final command runs tests for all other submodules. test: $(GO_ENV) go test $(GO_FLAGS) -race $(shell go list ./... | grep -v /integration-tests/) - $(GO_ENV) go test $(GO_FLAGS) ./internal/static/integrations/node_exporter ./internal/static/logs ./internal/component/otelcol/processor/tail_sampling ./internal/component/loki/source/file ./internal/component/loki/source/docker ./internal/component/prometheus/write/queue/serialization ./internal/component/prometheus/write/queue/network + $(GO_ENV) go test $(GO_FLAGS) ./internal/static/integrations/node_exporter ./internal/static/logs ./internal/component/otelcol/processor/tail_sampling ./internal/component/loki/source/file ./internal/component/loki/source/docker $(GO_ENV) find . -name go.mod -not -path "./go.mod" -execdir go test -race ./... \; test-packages: diff --git a/go.mod b/go.mod index 41fa2a8f76..5ab95d477b 100644 --- a/go.mod +++ b/go.mod @@ -72,6 +72,7 @@ require ( github.com/grafana/regexp v0.0.0-20240518133315-a468a5bfb3bc github.com/grafana/tail v0.0.0-20230510142333-77b18831edf0 github.com/grafana/vmware_exporter v0.0.5-beta + github.com/grafana/walqueue v0.0.0-20241114193920-da8174120940 github.com/hashicorp/consul/api v1.29.5 github.com/hashicorp/go-discover v0.0.0-20230724184603-e89ebd1b2f65 github.com/hashicorp/go-multierror v1.1.1 @@ -165,7 +166,7 @@ require ( github.com/prometheus/mysqld_exporter v0.14.0 github.com/prometheus/node_exporter v1.6.0 github.com/prometheus/procfs v0.15.1 - github.com/prometheus/prometheus v0.54.1 // a.k.a. v2.51.2 + github.com/prometheus/prometheus v0.55.1 // a.k.a. v2.51.2 github.com/prometheus/snmp_exporter v0.26.0 // if you update the snmp_exporter version, make sure to update the SNMP_VERSION in _index github.com/prometheus/statsd_exporter v0.22.8 github.com/richardartoul/molecule v1.0.1-0.20221107223329-32cfee06a052 @@ -468,7 +469,7 @@ require ( github.com/dustin/go-humanize v1.0.1 // indirect github.com/dvsekhvalnov/jose2go v1.6.0 // indirect github.com/eapache/go-resiliency v1.7.0 // indirect - github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 + github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 // indirect github.com/eapache/queue v1.1.0 // indirect github.com/edsrzf/mmap-go v1.1.0 // indirect github.com/efficientgo/core v1.0.0-rc.2 // indirect @@ -744,7 +745,7 @@ require ( github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635 // indirect github.com/tencentcloud/tencentcloud-sdk-go v1.0.162 // indirect github.com/tg123/go-htpasswd v1.2.2 // indirect - github.com/tinylib/msgp v1.2.2 + github.com/tinylib/msgp v1.2.4 // indirect github.com/tklauser/go-sysconf v0.3.13 // indirect github.com/tklauser/numcpus v0.7.0 // indirect github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80 // indirect @@ -752,7 +753,7 @@ require ( github.com/vertica/vertica-sql-go v1.3.3 // indirect github.com/vishvananda/netlink v1.2.1-beta.2 // indirect github.com/vishvananda/netns v0.0.4 // indirect - github.com/vladopajic/go-actor v0.9.0 + github.com/vladopajic/go-actor v0.9.0 // indirect github.com/vmware/govmomi v0.44.1 // indirect github.com/vultr/govultr/v2 v2.17.2 // indirect github.com/willf/bitset v1.1.11 // indirect @@ -789,7 +790,7 @@ require ( go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.7.0 // indirect go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.31.0 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.31.0 // indirect - go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.31.0 // indirect + go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.31.0 go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.31.0 // indirect go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.31.0 // indirect go.opentelemetry.io/otel/log v0.7.0 // indirect @@ -942,3 +943,6 @@ exclude ( ) replace github.com/prometheus/procfs => github.com/prometheus/procfs v0.12.0 + +// This is to handle issues witn synchronous mailbox and closing channels. +replace github.com/vladopajic/go-actor => github.com/grafana/go-actor v0.0.0-20241113133736-e18c4a5c12f4 diff --git a/go.sum b/go.sum index 6a139e700d..409391ecb1 100644 --- a/go.sum +++ b/go.sum @@ -1223,6 +1223,8 @@ github.com/grafana/cloudflare-go v0.0.0-20230110200409-c627cf6792f2 h1:qhugDMdQ4 github.com/grafana/cloudflare-go v0.0.0-20230110200409-c627cf6792f2/go.mod h1:w/aiO1POVIeXUQyl0VQSZjl5OAGDTL5aX+4v0RA1tcw= github.com/grafana/dskit v0.0.0-20240104111617-ea101a3b86eb h1:AWE6+kvtE18HP+lRWNUCyvymyrFSXs6TcS2vXIXGIuw= github.com/grafana/dskit v0.0.0-20240104111617-ea101a3b86eb/go.mod h1:kkWM4WUV230bNG3urVRWPBnSJHs64y/0RmWjftnnn0c= +github.com/grafana/go-actor v0.0.0-20241113133736-e18c4a5c12f4 h1:jid0h8vbKxOfHbVu/5exi6fz2y9/vKmtcKtTfuXElMY= +github.com/grafana/go-actor v0.0.0-20241113133736-e18c4a5c12f4/go.mod h1:b4thGZ60fnjC3TaJ4XeCN+uZXM+ec27t3ibqFfd8iAk= github.com/grafana/go-gelf/v2 v2.0.1 h1:BOChP0h/jLeD+7F9mL7tq10xVkDG15he3T1zHuQaWak= github.com/grafana/go-gelf/v2 v2.0.1/go.mod h1:lexHie0xzYGwCgiRGcvZ723bSNyNI8ZRD4s0CLobh90= github.com/grafana/go-offsets-tracker v0.1.7 h1:2zBQ7iiGzvyXY7LA8kaaSiEqH/Yx82UcfRabbY5aOG4= @@ -1276,6 +1278,8 @@ github.com/grafana/tail v0.0.0-20230510142333-77b18831edf0 h1:bjh0PVYSVVFxzINqPF github.com/grafana/tail v0.0.0-20230510142333-77b18831edf0/go.mod h1:7t5XR+2IA8P2qggOAHTj/GCZfoLBle3OvNSYh1VkRBU= github.com/grafana/vmware_exporter v0.0.5-beta h1:2JCqzIWJzns8FN78wPsueC9rT3e3kZo2OUoL5kGMjdM= github.com/grafana/vmware_exporter v0.0.5-beta/go.mod h1:1CecUZII0zVsVcHtNfNeTTcxK7EksqAsAn/TCCB0Mh4= +github.com/grafana/walqueue v0.0.0-20241114193920-da8174120940 h1:g086EMuMz94kliAaT5RanZ+R/wp5JdD4MZdoCWg0oDQ= +github.com/grafana/walqueue v0.0.0-20241114193920-da8174120940/go.mod h1:vaxO1V0q1dptHEiTIMW1krRy+aehkYyC2YGrKPyGxHY= github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= github.com/grobie/gomemcache v0.0.0-20230213081705-239240bbc445 h1:FlKQKUYPZ5yDCN248M3R7x8yu2E3yEZ0H7aLomE4EoE= github.com/grobie/gomemcache v0.0.0-20230213081705-239240bbc445/go.mod h1:L69/dBlPQlWkcnU76WgcppK5e4rrxzQdi6LhLnK/ytA= @@ -2446,8 +2450,8 @@ github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhV github.com/tilinna/clock v1.1.0 h1:6IQQQCo6KoBxVudv6gwtY8o4eDfhHo8ojA5dP0MfhSs= github.com/tilinna/clock v1.1.0/go.mod h1:ZsP7BcY7sEEz7ktc0IVy8Us6boDrK8VradlKRUGfOao= github.com/tinylib/msgp v1.1.5/go.mod h1:eQsjooMTnV42mHu917E26IogZ2930nFyBQdofk10Udg= -github.com/tinylib/msgp v1.2.2 h1:iHiBE1tJQwFI740SPEPkGE8cfhNfrqOYRlH450BnC/4= -github.com/tinylib/msgp v1.2.2/go.mod h1:ykjzy2wzgrlvpDCRc4LA8UXy6D8bzMSuAF3WD57Gok0= +github.com/tinylib/msgp v1.2.4 h1:yLFeUGostXXSGW5vxfT5dXG/qzkn4schv2I7at5+hVU= +github.com/tinylib/msgp v1.2.4/go.mod h1:ykjzy2wzgrlvpDCRc4LA8UXy6D8bzMSuAF3WD57Gok0= github.com/tklauser/go-sysconf v0.3.11/go.mod h1:GqXfhXY3kiPa0nAXPDIQIWzJbMCB7AmcWpGR8lSZfqI= github.com/tklauser/go-sysconf v0.3.13 h1:GBUpcahXSpR2xN01jhkNAbTLRk2Yzgggk8IM08lq3r4= github.com/tklauser/go-sysconf v0.3.13/go.mod h1:zwleP4Q4OehZHGn4CYZDipCgg9usW5IJePewFCGVEa0= @@ -2488,8 +2492,6 @@ github.com/vishvananda/netns v0.0.0-20200728191858-db3c7e526aae/go.mod h1:DD4vA1 github.com/vishvananda/netns v0.0.4 h1:Oeaw1EM2JMxD51g9uhtC0D7erkIjgmj8+JZc26m1YX8= github.com/vishvananda/netns v0.0.4/go.mod h1:SpkAiCQRtJ6TvvxPnOSyH3BMl6unz3xZlaprSwhNNJM= github.com/vjeantet/grok v1.0.0/go.mod h1:/FWYEVYekkm+2VjcFmO9PufDU5FgXHUz9oy2EGqmQBo= -github.com/vladopajic/go-actor v0.9.0 h1:fFj5RDGo4YZ6XCx2BWCThx/efOGRwokTpsc3CWHVEIU= -github.com/vladopajic/go-actor v0.9.0/go.mod h1:CKVRXStfjEIi7K74SyFQv/KfM8a/Po57bxmbBGv9YwE= github.com/vmihailenco/msgpack/v4 v4.3.13 h1:A2wsiTbvp63ilDaWmsk2wjx6xZdxQOvpiNlKBGKKXKI= github.com/vmihailenco/msgpack/v4 v4.3.13/go.mod h1:gborTTJjAo/GWTqqRjrLCn9pgNN+NXzzngzBKDPIqw4= github.com/vmihailenco/msgpack/v5 v5.4.1 h1:cQriyiUvjTwOHg8QZaPihLWeRAAVoCpE00IUPn0Bjt8= diff --git a/internal/component/otelcol/exporter/datadog/config/config_datadog_test.go b/internal/component/otelcol/exporter/datadog/config/config_datadog_test.go index bb4c654bb1..a6b640a351 100644 --- a/internal/component/otelcol/exporter/datadog/config/config_datadog_test.go +++ b/internal/component/otelcol/exporter/datadog/config/config_datadog_test.go @@ -221,4 +221,3 @@ func TestUnmarshalDatadogLogsConfig(t *testing.T) { }) } } - diff --git a/internal/component/prometheus/write/queue/component.go b/internal/component/prometheus/write/queue/component.go index 29c22253b4..9518389f2c 100644 --- a/internal/component/prometheus/write/queue/component.go +++ b/internal/component/prometheus/write/queue/component.go @@ -8,12 +8,8 @@ import ( "github.com/go-kit/log" "github.com/grafana/alloy/internal/component" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/filequeue" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/network" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/serialization" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" "github.com/grafana/alloy/internal/featuregate" - "github.com/prometheus/client_golang/prometheus" + promqueue "github.com/grafana/walqueue/implementations/prometheus" "github.com/prometheus/prometheus/storage" ) @@ -34,7 +30,7 @@ func NewComponent(opts component.Options, args Arguments) (*Queue, error) { opts: opts, args: args, log: opts.Logger, - endpoints: map[string]*endpoint{}, + endpoints: map[string]promqueue.Queue{}, } err := s.createEndpoints() @@ -58,7 +54,7 @@ type Queue struct { args Arguments opts component.Options log log.Logger - endpoints map[string]*endpoint + endpoints map[string]promqueue.Queue } // Run starts the component, blocking until ctx is canceled or the component @@ -90,60 +86,52 @@ func (s *Queue) Update(args component.Arguments) error { defer s.mut.Unlock() newArgs := args.(Arguments) - sync.OnceFunc(func() { - s.opts.OnStateChange(Exports{Receiver: s}) - }) // If they are the same do nothing. if reflect.DeepEqual(newArgs, s.args) { return nil } s.args = newArgs - // TODO @mattdurham need to cycle through the endpoints figuring out what changed instead of this global stop and start. - // This will cause data in the endpoints and their children to be lost. - if len(s.endpoints) > 0 { - for _, ep := range s.endpoints { + // Figure out which endpoint is new, which is updated, and which needs to be gone. + // So add all the endpoints and then if they are in the new config then remove them from deletable. + deletableEndpoints := make(map[string]struct{}) + for k := range s.endpoints { + deletableEndpoints[k] = struct{}{} + } + + for _, epCfg := range s.args.Endpoints { + delete(deletableEndpoints, epCfg.Name) + ep, found := s.endpoints[epCfg.Name] + // If found stop and recreate. + if found { + // Stop and loose all the signals in the queue. + // TODO drain the signals and re-add them ep.Stop() } - s.endpoints = map[string]*endpoint{} - } - err := s.createEndpoints() - if err != nil { - return err + nativeCfg := epCfg.ToNativeType() + // Create + end, err := promqueue.NewQueue(epCfg.Name, nativeCfg, filepath.Join(s.opts.DataPath, epCfg.Name, "wal"), uint32(s.args.Persistence.MaxSignalsToBatch), s.args.Persistence.BatchInterval, s.args.TTL, s.opts.Registerer, "alloy", s.opts.Logger) + if err != nil { + return err + } + end.Start() + s.endpoints[epCfg.Name] = end + } - for _, ep := range s.endpoints { - ep.Start() + // Now we need to figure out the endpoints that were not touched and able to be deleted. + for name := range deletableEndpoints { + s.endpoints[name].Stop() + delete(s.endpoints, name) } return nil } func (s *Queue) createEndpoints() error { - // @mattdurham not in love with this code. for _, ep := range s.args.Endpoints { - reg := prometheus.WrapRegistererWith(prometheus.Labels{"endpoint": ep.Name}, s.opts.Registerer) - stats := types.NewStats("alloy", "queue_series", reg) - stats.SeriesBackwardsCompatibility(reg) - meta := types.NewStats("alloy", "queue_metadata", reg) - meta.MetaBackwardsCompatibility(reg) - cfg := ep.ToNativeType() - client, err := network.New(cfg, s.log, stats.UpdateNetwork, meta.UpdateNetwork) - if err != nil { - return err - } - end := NewEndpoint(client, nil, s.args.TTL, s.opts.Logger) - fq, err := filequeue.NewQueue(filepath.Join(s.opts.DataPath, ep.Name, "wal"), func(ctx context.Context, dh types.DataHandle) { - _ = end.incoming.Send(ctx, dh) - }, s.opts.Logger) - if err != nil { - return err - } - serial, err := serialization.NewSerializer(types.SerializerConfig{ - MaxSignalsInBatch: uint32(s.args.Persistence.MaxSignalsToBatch), - FlushFrequency: s.args.Persistence.BatchInterval, - }, fq, stats.UpdateSerializer, s.opts.Logger) + nativeCfg := ep.ToNativeType() + end, err := promqueue.NewQueue(ep.Name, nativeCfg, filepath.Join(s.opts.DataPath, ep.Name, "wal"), uint32(s.args.Persistence.MaxSignalsToBatch), s.args.Persistence.BatchInterval, s.args.TTL, s.opts.Registerer, "alloy", s.opts.Logger) if err != nil { return err } - end.serializer = serial s.endpoints[ep.Name] = end } return nil @@ -158,7 +146,7 @@ func (c *Queue) Appender(ctx context.Context) storage.Appender { children := make([]storage.Appender, 0) for _, ep := range c.endpoints { - children = append(children, serialization.NewAppender(ctx, c.args.TTL, ep.serializer, c.opts.Logger)) + children = append(children, ep.Appender(ctx)) } return &fanout{children: children} } diff --git a/internal/component/prometheus/write/queue/e2e_stats_test.go b/internal/component/prometheus/write/queue/e2e_stats_test.go index 16c1e33fe0..bb4c059d1c 100644 --- a/internal/component/prometheus/write/queue/e2e_stats_test.go +++ b/internal/component/prometheus/write/queue/e2e_stats_test.go @@ -615,10 +615,7 @@ func runE2eStats(t *testing.T, test statsTest) { } require.NoError(t, app.Commit()) }() - tm := time.NewTimer(8 * time.Second) - <-tm.C - cancel() - + time.Sleep(5 * time.Second) require.Eventually(t, func() bool { dtos, gatherErr := reg.Gather() require.NoError(t, gatherErr) @@ -632,9 +629,13 @@ func runE2eStats(t *testing.T, test statsTest) { // Make sure we have a few metrics. return found > 1 }, 10*time.Second, 1*time.Second) + metrics := make(map[string]float64) dtos, err := reg.Gather() require.NoError(t, err) + // Cancel needs to be here since it will unregister the metrics. + cancel() + // Get the value of metrics. for _, d := range dtos { metrics[*d.Name] = getValue(d) diff --git a/internal/component/prometheus/write/queue/e2e_test.go b/internal/component/prometheus/write/queue/e2e_test.go index 5593a6cbdc..dfd3963d47 100644 --- a/internal/component/prometheus/write/queue/e2e_test.go +++ b/internal/component/prometheus/write/queue/e2e_test.go @@ -14,9 +14,9 @@ import ( "github.com/golang/snappy" "github.com/grafana/alloy/internal/component" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" "github.com/grafana/alloy/internal/runtime/logging" "github.com/grafana/alloy/internal/util" + "github.com/grafana/walqueue/types" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/prometheus/model/exemplar" "github.com/prometheus/prometheus/model/histogram" @@ -155,6 +155,7 @@ func runTest(t *testing.T, add func(index int, appendable storage.Appender) (flo require.NoError(t, err) ctx := context.Background() ctx, cancel := context.WithCancel(ctx) + go func() { runErr := c.Run(ctx) require.NoError(t, runErr) @@ -178,6 +179,7 @@ func runTest(t *testing.T, add func(index int, appendable storage.Appender) (flo require.NoError(t, app.Commit()) }() } + // This is a weird use case to handle eventually. // With race turned on this can take a long time. tm := time.NewTimer(20 * time.Second) @@ -186,6 +188,7 @@ func runTest(t *testing.T, add func(index int, appendable storage.Appender) (flo case <-tm.C: require.Truef(t, false, "failed to collect signals in the appropriate time") } + cancel() for i := 0; i < samples.Len(); i++ { @@ -213,7 +216,7 @@ func runTest(t *testing.T, add func(index int, appendable storage.Appender) (flo } require.Eventuallyf(t, func() bool { return types.OutStandingTimeSeriesBinary.Load() == 0 - }, 2*time.Second, 100*time.Millisecond, "there are %d time series not collected", types.OutStandingTimeSeriesBinary.Load()) + }, 20*time.Second, 1*time.Second, "there are %d time series not collected", types.OutStandingTimeSeriesBinary.Load()) } func handlePost(t *testing.T, _ http.ResponseWriter, r *http.Request) ([]prompb.TimeSeries, []prompb.MetricMetadata) { diff --git a/internal/component/prometheus/write/queue/endpoint.go b/internal/component/prometheus/write/queue/endpoint.go deleted file mode 100644 index de446ae808..0000000000 --- a/internal/component/prometheus/write/queue/endpoint.go +++ /dev/null @@ -1,133 +0,0 @@ -package queue - -import ( - "context" - "strconv" - "time" - - snappy "github.com/eapache/go-xerial-snappy" - "github.com/go-kit/log" - "github.com/go-kit/log/level" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" - "github.com/vladopajic/go-actor/actor" -) - -var _ actor.Worker = (*endpoint)(nil) - -// endpoint handles communication between the serializer, filequeue and network. -type endpoint struct { - network types.NetworkClient - serializer types.Serializer - log log.Logger - ttl time.Duration - incoming actor.Mailbox[types.DataHandle] - buf []byte - self actor.Actor -} - -func NewEndpoint(client types.NetworkClient, serializer types.Serializer, ttl time.Duration, logger log.Logger) *endpoint { - return &endpoint{ - network: client, - serializer: serializer, - log: logger, - ttl: ttl, - incoming: actor.NewMailbox[types.DataHandle](), - buf: make([]byte, 0, 1024), - } -} - -func (ep *endpoint) Start() { - ep.self = actor.Combine(actor.New(ep), ep.incoming).Build() - ep.self.Start() - ep.serializer.Start() - ep.network.Start() -} - -func (ep *endpoint) Stop() { - // Stop in order of data flow. This prevents errors around stopped mailboxes that can pop up. - ep.serializer.Stop() - ep.network.Stop() - ep.self.Stop() -} - -func (ep *endpoint) DoWork(ctx actor.Context) actor.WorkerStatus { - select { - case <-ctx.Done(): - return actor.WorkerEnd - case file, ok := <-ep.incoming.ReceiveC(): - if !ok { - return actor.WorkerEnd - } - meta, buf, err := file.Pop() - if err != nil { - level.Error(ep.log).Log("msg", "unable to get file contents", "name", file.Name, "err", err) - return actor.WorkerContinue - } - ep.deserializeAndSend(ctx, meta, buf) - return actor.WorkerContinue - } -} - -func (ep *endpoint) deserializeAndSend(ctx context.Context, meta map[string]string, buf []byte) { - var err error - ep.buf, err = snappy.DecodeInto(ep.buf, buf) - if err != nil { - level.Debug(ep.log).Log("msg", "error snappy decoding", "err", err) - return - } - // The version of each file is in the metadata. Right now there is only one version - // supported but in the future the ability to support more. Along with different - // compression. - version, ok := meta["version"] - if !ok { - level.Error(ep.log).Log("msg", "version not found for deserialization") - return - } - if version != types.AlloyFileVersion { - level.Error(ep.log).Log("msg", "invalid version found for deserialization", "version", version) - return - } - // Grab the amounts of each type and we can go ahead and alloc the space. - seriesCount, _ := strconv.Atoi(meta["series_count"]) - metaCount, _ := strconv.Atoi(meta["meta_count"]) - stringsCount, _ := strconv.Atoi(meta["strings_count"]) - sg := &types.SeriesGroup{ - Series: make([]*types.TimeSeriesBinary, seriesCount), - Metadata: make([]*types.TimeSeriesBinary, metaCount), - Strings: make([]string, stringsCount), - } - // Prefill our series with items from the pool to limit allocs. - for i := 0; i < seriesCount; i++ { - sg.Series[i] = types.GetTimeSeriesFromPool() - } - for i := 0; i < metaCount; i++ { - sg.Metadata[i] = types.GetTimeSeriesFromPool() - } - sg, ep.buf, err = types.DeserializeToSeriesGroup(sg, ep.buf) - if err != nil { - level.Debug(ep.log).Log("msg", "error deserializing", "err", err) - return - } - - for _, series := range sg.Series { - // One last chance to check the TTL. Writing to the filequeue will check it but - // in a situation where the network is down and writing backs up we dont want to send - // data that will get rejected. - seriesAge := time.Since(time.Unix(series.TS, 0)) - if seriesAge > ep.ttl { - // TODO @mattdurham add metric here for ttl expired. - continue - } - sendErr := ep.network.SendSeries(ctx, series) - if sendErr != nil { - level.Error(ep.log).Log("msg", "error sending to write client", "err", sendErr) - } - } - - for _, md := range sg.Metadata { - sendErr := ep.network.SendMetadata(ctx, md) - if sendErr != nil { - level.Error(ep.log).Log("msg", "error sending metadata to write client", "err", sendErr) - } - } -} diff --git a/internal/component/prometheus/write/queue/filequeue/filequeue.go b/internal/component/prometheus/write/queue/filequeue/filequeue.go deleted file mode 100644 index 7088d3df4b..0000000000 --- a/internal/component/prometheus/write/queue/filequeue/filequeue.go +++ /dev/null @@ -1,191 +0,0 @@ -package filequeue - -import ( - "context" - "fmt" - "os" - "path/filepath" - "sort" - "strconv" - "strings" - - "github.com/go-kit/log" - "github.com/vladopajic/go-actor/actor" - - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" - "github.com/grafana/alloy/internal/runtime/logging/level" -) - -var _ actor.Worker = (*queue)(nil) -var _ types.FileStorage = (*queue)(nil) - -// queue represents an on-disk queue. This is a list implemented as files ordered by id with a name pattern: .committed -// Each file contains a byte buffer and an optional metatdata map. -type queue struct { - self actor.Actor - directory string - maxID int - logger log.Logger - dataQueue actor.Mailbox[types.Data] - // Out is where to send data when pulled from queue, it is assumed that it will - // block until ready for another record. - out func(ctx context.Context, dh types.DataHandle) - // existingFiles is the list of files found initially. - existingFiles []string -} - -// NewQueue returns a implementation of FileStorage. -func NewQueue(directory string, out func(ctx context.Context, dh types.DataHandle), logger log.Logger) (types.FileStorage, error) { - err := os.MkdirAll(directory, 0777) - if err != nil { - return nil, err - } - - // We dont actually support uncommitted but I think its good to at least have some naming to avoid parsing random files - // that get installed into the system. - matches, _ := filepath.Glob(filepath.Join(directory, "*.committed")) - ids := make([]int, len(matches)) - - // Try and grab the id from each file. - // e.g. grab 1 from `1.committed` - for i, fileName := range matches { - id, err := strconv.Atoi(strings.ReplaceAll(filepath.Base(fileName), ".committed", "")) - if err != nil { - level.Error(logger).Log("msg", "unable to convert numeric prefix for committed file", "err", err, "file", fileName) - continue - } - ids[i] = id - } - sort.Ints(ids) - var currentMaxID int - if len(ids) > 0 { - currentMaxID = ids[len(ids)-1] - } - q := &queue{ - directory: directory, - maxID: currentMaxID, - logger: logger, - out: out, - dataQueue: actor.NewMailbox[types.Data](), - existingFiles: make([]string, 0), - } - - // Save the existing files in `q.existingFiles`, which will have their data pushed to `out` when actor starts. - for _, id := range ids { - name := filepath.Join(directory, fmt.Sprintf("%d.committed", id)) - q.existingFiles = append(q.existingFiles, name) - } - return q, nil -} - -func (q *queue) Start() { - // Actors and mailboxes have to be started. It makes sense to combine them into one unit since they - // have the same lifespan. - q.self = actor.Combine(actor.New(q), q.dataQueue).Build() - q.self.Start() -} - -func (q *queue) Stop() { - q.self.Stop() -} - -// Store will add records to the dataQueue that will add the data to the filesystem. This is an unbuffered channel. -// Its possible in the future we would want to make it a buffer of 1, but so far it hasnt been an issue in testing. -func (q *queue) Store(ctx context.Context, meta map[string]string, data []byte) error { - return q.dataQueue.Send(ctx, types.Data{ - Meta: meta, - Data: data, - }) -} - -// get returns the data of the file or an error if something wrong went on. -func get(logger log.Logger, name string) (map[string]string, []byte, error) { - defer deleteFile(logger, name) - buf, err := readFile(name) - if err != nil { - return nil, nil, err - } - r := &Record{} - _, err = r.UnmarshalMsg(buf) - if err != nil { - return nil, nil, err - } - return r.Meta, r.Data, nil -} - -// DoWork allows most of the queue to be single threaded with work only coming in and going out via mailboxes(channels). -func (q *queue) DoWork(ctx actor.Context) actor.WorkerStatus { - // Queue up our existing items. - for _, name := range q.existingFiles { - q.out(ctx, types.DataHandle{ - Name: name, - Pop: func() (map[string]string, []byte, error) { - return get(q.logger, name) - }, - }) - } - // We only want to process existing files once. - q.existingFiles = nil - select { - case <-ctx.Done(): - return actor.WorkerEnd - case item, ok := <-q.dataQueue.ReceiveC(): - if !ok { - return actor.WorkerEnd - } - name, err := q.add(item.Meta, item.Data) - if err != nil { - level.Error(q.logger).Log("msg", "error adding item - dropping data", "err", err) - return actor.WorkerContinue - } - // The idea is that this will callee will block/process until the callee is ready for another file. - q.out(ctx, types.DataHandle{ - Name: name, - Pop: func() (map[string]string, []byte, error) { - return get(q.logger, name) - }, - }) - return actor.WorkerContinue - } -} - -// Add a file to the queue (as committed). -func (q *queue) add(meta map[string]string, data []byte) (string, error) { - if meta == nil { - meta = make(map[string]string) - } - q.maxID++ - name := filepath.Join(q.directory, fmt.Sprintf("%d.committed", q.maxID)) - r := &Record{ - Meta: meta, - Data: data, - } - // Not reusing a buffer here since allocs are not bad here and we are trying to reduce memory. - rBuf, err := r.MarshalMsg(nil) - if err != nil { - return "", err - } - err = q.writeFile(name, rBuf) - if err != nil { - return "", err - } - return name, nil -} - -func (q *queue) writeFile(name string, data []byte) error { - return os.WriteFile(name, data, 0644) -} - -func deleteFile(logger log.Logger, name string) { - err := os.Remove(name) - if err != nil { - level.Error(logger).Log("msg", "unable to delete file", "err", err, "file", name) - } -} -func readFile(name string) ([]byte, error) { - bb, err := os.ReadFile(name) - if err != nil { - return nil, err - } - return bb, err -} diff --git a/internal/component/prometheus/write/queue/filequeue/filequeue_test.go b/internal/component/prometheus/write/queue/filequeue/filequeue_test.go deleted file mode 100644 index d36e0b71ed..0000000000 --- a/internal/component/prometheus/write/queue/filequeue/filequeue_test.go +++ /dev/null @@ -1,253 +0,0 @@ -package filequeue - -import ( - "context" - "os" - "path/filepath" - "runtime" - "strconv" - "testing" - "time" - - "github.com/vladopajic/go-actor/actor" - "go.uber.org/goleak" - - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" - - "github.com/go-kit/log" - "github.com/stretchr/testify/require" -) - -func TestFileQueue(t *testing.T) { - defer goleak.VerifyNone(t) - dir := t.TempDir() - log := log.NewNopLogger() - mbx := actor.NewMailbox[types.DataHandle]() - mbx.Start() - defer mbx.Stop() - q, err := NewQueue(dir, func(ctx context.Context, dh types.DataHandle) { - _ = mbx.Send(ctx, dh) - }, log) - require.NoError(t, err) - q.Start() - defer q.Stop() - err = q.Store(context.Background(), nil, []byte("test")) - - require.NoError(t, err) - - meta, buf, err := getHandle(t, mbx) - require.NoError(t, err) - require.True(t, string(buf) == "test") - require.Len(t, meta, 0) - - // Ensure nothing new comes through. - timer := time.NewTicker(100 * time.Millisecond) - select { - case <-timer.C: - return - case <-mbx.ReceiveC(): - require.True(t, false) - } -} - -func TestMetaFileQueue(t *testing.T) { - defer goleak.VerifyNone(t) - - dir := t.TempDir() - log := log.NewNopLogger() - mbx := actor.NewMailbox[types.DataHandle]() - mbx.Start() - defer mbx.Stop() - q, err := NewQueue(dir, func(ctx context.Context, dh types.DataHandle) { - _ = mbx.Send(ctx, dh) - }, log) - q.Start() - defer q.Stop() - require.NoError(t, err) - err = q.Store(context.Background(), map[string]string{"name": "bob"}, []byte("test")) - require.NoError(t, err) - - meta, buf, err := getHandle(t, mbx) - require.NoError(t, err) - require.True(t, string(buf) == "test") - require.Len(t, meta, 1) - require.True(t, meta["name"] == "bob") -} - -func TestCorruption(t *testing.T) { - defer goleak.VerifyNone(t) - - dir := t.TempDir() - log := log.NewNopLogger() - mbx := actor.NewMailbox[types.DataHandle]() - mbx.Start() - defer mbx.Stop() - q, err := NewQueue(dir, func(ctx context.Context, dh types.DataHandle) { - _ = mbx.Send(ctx, dh) - }, log) - q.Start() - defer q.Stop() - require.NoError(t, err) - - err = q.Store(context.Background(), map[string]string{"name": "bob"}, []byte("first")) - require.NoError(t, err) - err = q.Store(context.Background(), map[string]string{"name": "bob"}, []byte("second")) - - require.NoError(t, err) - - // Send is async so may need to wait a bit for it happen. - require.Eventually(t, func() bool { - // First should be 1.committed - _, errStat := os.Stat(filepath.Join(dir, "1.committed")) - return errStat == nil - }, 2*time.Second, 100*time.Millisecond) - - fi, err := os.Stat(filepath.Join(dir, "1.committed")) - - require.NoError(t, err) - err = os.WriteFile(filepath.Join(dir, fi.Name()), []byte("bad"), 0644) - require.NoError(t, err) - - _, _, err = getHandle(t, mbx) - require.Error(t, err) - - meta, buf, err := getHandle(t, mbx) - require.NoError(t, err) - require.True(t, string(buf) == "second") - require.Len(t, meta, 1) -} - -func TestFileDeleted(t *testing.T) { - defer goleak.VerifyNone(t) - - dir := t.TempDir() - log := log.NewNopLogger() - mbx := actor.NewMailbox[types.DataHandle]() - mbx.Start() - defer mbx.Stop() - q, err := NewQueue(dir, func(ctx context.Context, dh types.DataHandle) { - _ = mbx.Send(ctx, dh) - }, log) - q.Start() - defer q.Stop() - require.NoError(t, err) - - evenHandles := make([]string, 0) - for i := 0; i < 10; i++ { - err = q.Store(context.Background(), map[string]string{"name": "bob"}, []byte(strconv.Itoa(i))) - - require.NoError(t, err) - if i%2 == 0 { - evenHandles = append(evenHandles, filepath.Join(dir, strconv.Itoa(i+1)+".committed")) - } - } - - // Send is async so may need to wait a bit for it happen, check for the last file written. - require.Eventually(t, func() bool { - _, errStat := os.Stat(filepath.Join(dir, "10.committed")) - return errStat == nil - }, 2*time.Second, 100*time.Millisecond) - - for _, h := range evenHandles { - _ = os.Remove(h) - } - // Every even file was deleted and should have an error. - for i := 0; i < 10; i++ { - _, buf2, err := getHandle(t, mbx) - if i%2 == 0 { - require.Error(t, err) - } else { - require.NoError(t, err) - require.True(t, string(buf2) == strconv.Itoa(i)) - } - } -} - -func TestOtherFiles(t *testing.T) { - if runtime.GOOS == "windows" { - // TODO: Fix this test as we mature the file queue - t.Skip("This test is very flaky on Windows. Will need to fix it as we mature the filequeue.") - } - defer goleak.VerifyNone(t) - - dir := t.TempDir() - log := log.NewNopLogger() - mbx := actor.NewMailbox[types.DataHandle]() - mbx.Start() - defer mbx.Stop() - q, err := NewQueue(dir, func(ctx context.Context, dh types.DataHandle) { - _ = mbx.Send(ctx, dh) - }, log) - q.Start() - defer q.Stop() - require.NoError(t, err) - - err = q.Store(context.Background(), nil, []byte("first")) - require.NoError(t, err) - os.Create(filepath.Join(dir, "otherfile")) - _, buf, err := getHandle(t, mbx) - require.NoError(t, err) - require.True(t, string(buf) == "first") -} - -func TestResuming(t *testing.T) { - defer goleak.VerifyNone(t) - - dir := t.TempDir() - log := log.NewNopLogger() - mbx := actor.NewMailbox[types.DataHandle]() - mbx.Start() - q, err := NewQueue(dir, func(ctx context.Context, dh types.DataHandle) { - _ = mbx.Send(ctx, dh) - }, log) - q.Start() - require.NoError(t, err) - - err = q.Store(context.Background(), nil, []byte("first")) - - require.NoError(t, err) - - err = q.Store(context.Background(), nil, []byte("second")) - - require.NoError(t, err) - time.Sleep(1 * time.Second) - mbx.Stop() - q.Stop() - - mbx2 := actor.NewMailbox[types.DataHandle]() - mbx2.Start() - defer mbx2.Stop() - q2, err := NewQueue(dir, func(ctx context.Context, dh types.DataHandle) { - _ = mbx2.Send(ctx, dh) - }, log) - require.NoError(t, err) - q2.Start() - defer q2.Stop() - err = q2.Store(context.Background(), nil, []byte("third")) - - require.NoError(t, err) - _, buf, err := getHandle(t, mbx2) - require.NoError(t, err) - require.True(t, string(buf) == "first") - - _, buf, err = getHandle(t, mbx2) - require.NoError(t, err) - require.True(t, string(buf) == "second") - - _, buf, err = getHandle(t, mbx2) - require.NoError(t, err) - require.True(t, string(buf) == "third") -} - -func getHandle(t *testing.T, mbx actor.MailboxReceiver[types.DataHandle]) (map[string]string, []byte, error) { - timer := time.NewTicker(5 * time.Second) - select { - case <-timer.C: - require.True(t, false) - // This is only here to satisfy the linting. - return nil, nil, nil - case item, ok := <-mbx.ReceiveC(): - require.True(t, ok) - return item.Pop() - } -} diff --git a/internal/component/prometheus/write/queue/filequeue/record.go b/internal/component/prometheus/write/queue/filequeue/record.go deleted file mode 100644 index 2d6b12a034..0000000000 --- a/internal/component/prometheus/write/queue/filequeue/record.go +++ /dev/null @@ -1,11 +0,0 @@ -package filequeue - -// Record wraps the input data and combines it with the metadata. -// -//go:generate msgp -type Record struct { - // Meta holds a key value pair that can include information about the data. - // Such as compression used, file format version and other important bits of data. - Meta map[string]string - Data []byte -} diff --git a/internal/component/prometheus/write/queue/filequeue/record_gen.go b/internal/component/prometheus/write/queue/filequeue/record_gen.go deleted file mode 100644 index 285940eb88..0000000000 --- a/internal/component/prometheus/write/queue/filequeue/record_gen.go +++ /dev/null @@ -1,206 +0,0 @@ -package filequeue - -// Code generated by github.com/tinylib/msgp DO NOT EDIT. - -import ( - "github.com/tinylib/msgp/msgp" -) - -// DecodeMsg implements msgp.Decodable -func (z *Record) DecodeMsg(dc *msgp.Reader) (err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "Meta": - var zb0002 uint32 - zb0002, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err, "Meta") - return - } - if z.Meta == nil { - z.Meta = make(map[string]string, zb0002) - } else if len(z.Meta) > 0 { - for key := range z.Meta { - delete(z.Meta, key) - } - } - for zb0002 > 0 { - zb0002-- - var za0001 string - var za0002 string - za0001, err = dc.ReadString() - if err != nil { - err = msgp.WrapError(err, "Meta") - return - } - za0002, err = dc.ReadString() - if err != nil { - err = msgp.WrapError(err, "Meta", za0001) - return - } - z.Meta[za0001] = za0002 - } - case "Data": - z.Data, err = dc.ReadBytes(z.Data) - if err != nil { - err = msgp.WrapError(err, "Data") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - return -} - -// EncodeMsg implements msgp.Encodable -func (z *Record) EncodeMsg(en *msgp.Writer) (err error) { - // map header, size 2 - // write "Meta" - err = en.Append(0x82, 0xa4, 0x4d, 0x65, 0x74, 0x61) - if err != nil { - return - } - err = en.WriteMapHeader(uint32(len(z.Meta))) - if err != nil { - err = msgp.WrapError(err, "Meta") - return - } - for za0001, za0002 := range z.Meta { - err = en.WriteString(za0001) - if err != nil { - err = msgp.WrapError(err, "Meta") - return - } - err = en.WriteString(za0002) - if err != nil { - err = msgp.WrapError(err, "Meta", za0001) - return - } - } - // write "Data" - err = en.Append(0xa4, 0x44, 0x61, 0x74, 0x61) - if err != nil { - return - } - err = en.WriteBytes(z.Data) - if err != nil { - err = msgp.WrapError(err, "Data") - return - } - return -} - -// MarshalMsg implements msgp.Marshaler -func (z *Record) MarshalMsg(b []byte) (o []byte, err error) { - o = msgp.Require(b, z.Msgsize()) - // map header, size 2 - // string "Meta" - o = append(o, 0x82, 0xa4, 0x4d, 0x65, 0x74, 0x61) - o = msgp.AppendMapHeader(o, uint32(len(z.Meta))) - for za0001, za0002 := range z.Meta { - o = msgp.AppendString(o, za0001) - o = msgp.AppendString(o, za0002) - } - // string "Data" - o = append(o, 0xa4, 0x44, 0x61, 0x74, 0x61) - o = msgp.AppendBytes(o, z.Data) - return -} - -// UnmarshalMsg implements msgp.Unmarshaler -func (z *Record) UnmarshalMsg(bts []byte) (o []byte, err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "Meta": - var zb0002 uint32 - zb0002, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "Meta") - return - } - if z.Meta == nil { - z.Meta = make(map[string]string, zb0002) - } else if len(z.Meta) > 0 { - for key := range z.Meta { - delete(z.Meta, key) - } - } - for zb0002 > 0 { - var za0001 string - var za0002 string - zb0002-- - za0001, bts, err = msgp.ReadStringBytes(bts) - if err != nil { - err = msgp.WrapError(err, "Meta") - return - } - za0002, bts, err = msgp.ReadStringBytes(bts) - if err != nil { - err = msgp.WrapError(err, "Meta", za0001) - return - } - z.Meta[za0001] = za0002 - } - case "Data": - z.Data, bts, err = msgp.ReadBytesBytes(bts, z.Data) - if err != nil { - err = msgp.WrapError(err, "Data") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - o = bts - return -} - -// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message -func (z *Record) Msgsize() (s int) { - s = 1 + 5 + msgp.MapHeaderSize - if z.Meta != nil { - for za0001, za0002 := range z.Meta { - _ = za0002 - s += msgp.StringPrefixSize + len(za0001) + msgp.StringPrefixSize + len(za0002) - } - } - s += 5 + msgp.BytesPrefixSize + len(z.Data) - return -} diff --git a/internal/component/prometheus/write/queue/filequeue/record_gen_test.go b/internal/component/prometheus/write/queue/filequeue/record_gen_test.go deleted file mode 100644 index 6206b5f93c..0000000000 --- a/internal/component/prometheus/write/queue/filequeue/record_gen_test.go +++ /dev/null @@ -1,123 +0,0 @@ -package filequeue - -// Code generated by github.com/tinylib/msgp DO NOT EDIT. - -import ( - "bytes" - "testing" - - "github.com/tinylib/msgp/msgp" -) - -func TestMarshalUnmarshalRecord(t *testing.T) { - v := Record{} - bts, err := v.MarshalMsg(nil) - if err != nil { - t.Fatal(err) - } - left, err := v.UnmarshalMsg(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left) - } - - left, err = msgp.Skip(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after Skip(): %q", len(left), left) - } -} - -func BenchmarkMarshalMsgRecord(b *testing.B) { - v := Record{} - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.MarshalMsg(nil) - } -} - -func BenchmarkAppendMsgRecord(b *testing.B) { - v := Record{} - bts := make([]byte, 0, v.Msgsize()) - bts, _ = v.MarshalMsg(bts[0:0]) - b.SetBytes(int64(len(bts))) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - bts, _ = v.MarshalMsg(bts[0:0]) - } -} - -func BenchmarkUnmarshalRecord(b *testing.B) { - v := Record{} - bts, _ := v.MarshalMsg(nil) - b.ReportAllocs() - b.SetBytes(int64(len(bts))) - b.ResetTimer() - for i := 0; i < b.N; i++ { - _, err := v.UnmarshalMsg(bts) - if err != nil { - b.Fatal(err) - } - } -} - -func TestEncodeDecodeRecord(t *testing.T) { - v := Record{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - - m := v.Msgsize() - if buf.Len() > m { - t.Log("WARNING: TestEncodeDecodeRecord Msgsize() is inaccurate") - } - - vn := Record{} - err := msgp.Decode(&buf, &vn) - if err != nil { - t.Error(err) - } - - buf.Reset() - msgp.Encode(&buf, &v) - err = msgp.NewReader(&buf).Skip() - if err != nil { - t.Error(err) - } -} - -func BenchmarkEncodeRecord(b *testing.B) { - v := Record{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - en := msgp.NewWriter(msgp.Nowhere) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.EncodeMsg(en) - } - en.Flush() -} - -func BenchmarkDecodeRecord(b *testing.B) { - v := Record{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - rd := msgp.NewEndlessReader(buf.Bytes(), b) - dc := msgp.NewReader(rd) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - err := v.DecodeMsg(dc) - if err != nil { - b.Fatal(err) - } - } -} diff --git a/internal/component/prometheus/write/queue/network/benchmark_test.go b/internal/component/prometheus/write/queue/network/benchmark_test.go deleted file mode 100644 index f8bc6a3f5f..0000000000 --- a/internal/component/prometheus/write/queue/network/benchmark_test.go +++ /dev/null @@ -1,32 +0,0 @@ -package network - -import ( - "context" - "testing" - - "github.com/vladopajic/go-actor/actor" -) - -func BenchmarkMailbox(b *testing.B) { - // This should be 260 ns roughly or 3m messages a second. - mbx := actor.NewMailbox[struct{}]() - mbx.Start() - defer mbx.Stop() - - doneC := make(chan any) - - go func() { - for range b.N { - <-mbx.ReceiveC() - } - - close(doneC) - }() - - ctx := context.Background() - for range b.N { - mbx.Send(ctx, struct{}{}) - } - - <-doneC -} diff --git a/internal/component/prometheus/write/queue/network/loop.go b/internal/component/prometheus/write/queue/network/loop.go deleted file mode 100644 index 4654b190e3..0000000000 --- a/internal/component/prometheus/write/queue/network/loop.go +++ /dev/null @@ -1,371 +0,0 @@ -package network - -import ( - "bufio" - "bytes" - "context" - "fmt" - "io" - "net/http" - "strconv" - "strings" - "time" - - "github.com/go-kit/log" - "github.com/go-kit/log/level" - "github.com/golang/protobuf/proto" - "github.com/golang/snappy" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" - "github.com/prometheus/prometheus/prompb" - "github.com/vladopajic/go-actor/actor" - "go.uber.org/atomic" -) - -var _ actor.Worker = (*loop)(nil) - -// loop handles the low level sending of data. It's conceptually a queue. -// loop makes no attempt to save or restore signals in the queue. -// loop config cannot be updated, it is easier to recreate. This does mean we lose any signals in the queue. -type loop struct { - isMeta bool - seriesMbx actor.Mailbox[*types.TimeSeriesBinary] - client *http.Client - cfg types.ConnectionConfig - log log.Logger - lastSend time.Time - statsFunc func(s types.NetworkStats) - stopCalled atomic.Bool - externalLabels map[string]string - series []*types.TimeSeriesBinary - self actor.Actor - ticker *time.Ticker - req *prompb.WriteRequest - buf *proto.Buffer - sendBuffer []byte -} - -func newLoop(cc types.ConnectionConfig, isMetaData bool, l log.Logger, stats func(s types.NetworkStats)) *loop { - // TODO @mattdurham add TLS support afer the initial push. - return &loop{ - isMeta: isMetaData, - // In general we want a healthy queue of items, in this case we want to have 2x our maximum send sized ready. - seriesMbx: actor.NewMailbox[*types.TimeSeriesBinary](actor.OptCapacity(2*cc.BatchCount), actor.OptAsChan()), - client: &http.Client{}, - cfg: cc, - log: log.With(l, "name", "loop", "url", cc.URL), - statsFunc: stats, - externalLabels: cc.ExternalLabels, - ticker: time.NewTicker(1 * time.Second), - buf: proto.NewBuffer(nil), - sendBuffer: make([]byte, 0), - req: &prompb.WriteRequest{ - // We know BatchCount is the most we will ever send. - Timeseries: make([]prompb.TimeSeries, 0, cc.BatchCount), - }, - } -} - -func (l *loop) Start() { - l.self = actor.Combine(l.actors()...).Build() - l.self.Start() -} - -func (l *loop) Stop() { - l.stopCalled.Store(true) - l.self.Stop() -} - -func (l *loop) actors() []actor.Actor { - return []actor.Actor{ - actor.New(l), - l.seriesMbx, - } -} - -func (l *loop) DoWork(ctx actor.Context) actor.WorkerStatus { - // Main select loop - select { - case <-ctx.Done(): - return actor.WorkerEnd - // Ticker is to ensure the flush timer is called. - case <-l.ticker.C: - if len(l.series) == 0 { - return actor.WorkerContinue - } - if time.Since(l.lastSend) > l.cfg.FlushInterval { - l.trySend(ctx) - } - return actor.WorkerContinue - case series, ok := <-l.seriesMbx.ReceiveC(): - if !ok { - return actor.WorkerEnd - } - l.series = append(l.series, series) - if len(l.series) >= l.cfg.BatchCount { - l.trySend(ctx) - } - return actor.WorkerContinue - } -} - -// trySend is the core functionality for sending data to a endpoint. It will attempt retries as defined in MaxRetryAttempts. -func (l *loop) trySend(ctx context.Context) { - attempts := 0 - for { - start := time.Now() - result := l.send(ctx, attempts) - duration := time.Since(start) - l.statsFunc(types.NetworkStats{ - SendDuration: duration, - }) - if result.err != nil { - level.Error(l.log).Log("msg", "error in sending telemetry", "err", result.err.Error()) - } - if result.successful { - l.sendingCleanup() - return - } - if !result.recoverableError { - l.sendingCleanup() - return - } - attempts++ - if attempts > int(l.cfg.MaxRetryAttempts) && l.cfg.MaxRetryAttempts > 0 { - level.Debug(l.log).Log("msg", "max retry attempts reached", "attempts", attempts) - l.sendingCleanup() - return - } - // This helps us short circuit the loop if we are stopping. - if l.stopCalled.Load() { - return - } - // Sleep between attempts. - time.Sleep(result.retryAfter) - } -} - -type sendResult struct { - err error - successful bool - recoverableError bool - retryAfter time.Duration - statusCode int - networkError bool -} - -func (l *loop) sendingCleanup() { - types.PutTimeSeriesSliceIntoPool(l.series) - l.sendBuffer = l.sendBuffer[:0] - l.series = make([]*types.TimeSeriesBinary, 0, l.cfg.BatchCount) - l.lastSend = time.Now() -} - -// send is the main work loop of the loop. -func (l *loop) send(ctx context.Context, retryCount int) sendResult { - result := sendResult{} - defer func() { - recordStats(l.series, l.isMeta, l.statsFunc, result, len(l.sendBuffer)) - }() - // Check to see if this is a retry and we can reuse the buffer. - // I wonder if we should do this, its possible we are sending things that have exceeded the TTL. - if len(l.sendBuffer) == 0 { - var data []byte - var wrErr error - if l.isMeta { - data, wrErr = createWriteRequestMetadata(l.log, l.req, l.series, l.buf) - } else { - data, wrErr = createWriteRequest(l.req, l.series, l.externalLabels, l.buf) - } - if wrErr != nil { - result.err = wrErr - result.recoverableError = false - return result - } - l.sendBuffer = snappy.Encode(l.sendBuffer, data) - } - - httpReq, err := http.NewRequest("POST", l.cfg.URL, bytes.NewReader(l.sendBuffer)) - if err != nil { - result.err = err - result.recoverableError = true - result.networkError = true - return result - } - httpReq.Header.Add("Content-Encoding", "snappy") - httpReq.Header.Set("Content-Type", "application/x-protobuf") - httpReq.Header.Set("User-Agent", l.cfg.UserAgent) - httpReq.Header.Set("X-Prometheus-Remote-Write-Version", "0.1.0") - if l.cfg.BasicAuth != nil { - httpReq.SetBasicAuth(l.cfg.BasicAuth.Username, l.cfg.BasicAuth.Password) - } else if l.cfg.BearerToken != "" { - httpReq.Header.Set("Authorization", "Bearer "+string(l.cfg.BearerToken)) - } - - if retryCount > 0 { - httpReq.Header.Set("Retry-Attempt", strconv.Itoa(retryCount)) - } - ctx, cncl := context.WithTimeout(ctx, l.cfg.Timeout) - defer cncl() - resp, err := l.client.Do(httpReq.WithContext(ctx)) - // Network errors are recoverable. - if err != nil { - result.err = err - result.networkError = true - result.recoverableError = true - result.retryAfter = l.cfg.RetryBackoff - return result - } - result.statusCode = resp.StatusCode - defer resp.Body.Close() - // 500 errors are considered recoverable. - if resp.StatusCode/100 == 5 || resp.StatusCode == http.StatusTooManyRequests { - result.err = fmt.Errorf("server responded with status code %d", resp.StatusCode) - result.retryAfter = retryAfterDuration(l.cfg.RetryBackoff, resp.Header.Get("Retry-After")) - result.recoverableError = true - return result - } - // Status Codes that are not 500 or 200 are not recoverable and dropped. - if resp.StatusCode/100 != 2 { - scanner := bufio.NewScanner(io.LimitReader(resp.Body, 1_000)) - line := "" - if scanner.Scan() { - line = scanner.Text() - } - result.err = fmt.Errorf("server returned HTTP status %s: %s", resp.Status, line) - return result - } - - result.successful = true - return result -} - -func createWriteRequest(wr *prompb.WriteRequest, series []*types.TimeSeriesBinary, externalLabels map[string]string, data *proto.Buffer) ([]byte, error) { - if cap(wr.Timeseries) < len(series) { - wr.Timeseries = make([]prompb.TimeSeries, len(series)) - } - wr.Timeseries = wr.Timeseries[:len(series)] - - for i, tsBuf := range series { - ts := wr.Timeseries[i] - if cap(ts.Labels) < len(tsBuf.Labels) { - ts.Labels = make([]prompb.Label, 0, len(tsBuf.Labels)) - } - ts.Labels = ts.Labels[:len(tsBuf.Labels)] - for k, v := range tsBuf.Labels { - ts.Labels[k].Name = v.Name - ts.Labels[k].Value = v.Value - } - - // By default each sample only has a histogram, float histogram or sample. - if cap(ts.Histograms) == 0 { - ts.Histograms = make([]prompb.Histogram, 1) - } else { - ts.Histograms = ts.Histograms[:0] - } - if tsBuf.Histograms.Histogram != nil { - ts.Histograms = ts.Histograms[:1] - ts.Histograms[0] = tsBuf.Histograms.Histogram.ToPromHistogram() - } - if tsBuf.Histograms.FloatHistogram != nil { - ts.Histograms = ts.Histograms[:1] - ts.Histograms[0] = tsBuf.Histograms.FloatHistogram.ToPromFloatHistogram() - } - - if tsBuf.Histograms.Histogram == nil && tsBuf.Histograms.FloatHistogram == nil { - ts.Histograms = ts.Histograms[:0] - } - - // Encode the external labels inside if needed. - for k, v := range externalLabels { - found := false - for j, lbl := range ts.Labels { - if lbl.Name == k { - ts.Labels[j].Value = v - found = true - break - } - } - if !found { - ts.Labels = append(ts.Labels, prompb.Label{ - Name: k, - Value: v, - }) - } - } - // By default each TimeSeries only has one sample. - if len(ts.Samples) == 0 { - ts.Samples = make([]prompb.Sample, 1) - } - ts.Samples[0].Value = tsBuf.Value - ts.Samples[0].Timestamp = tsBuf.TS - wr.Timeseries[i] = ts - } - defer func() { - for i := 0; i < len(wr.Timeseries); i++ { - wr.Timeseries[i].Histograms = wr.Timeseries[i].Histograms[:0] - wr.Timeseries[i].Labels = wr.Timeseries[i].Labels[:0] - wr.Timeseries[i].Exemplars = wr.Timeseries[i].Exemplars[:0] - } - }() - // Reset the buffer for reuse. - data.Reset() - err := data.Marshal(wr) - return data.Bytes(), err -} - -func createWriteRequestMetadata(l log.Logger, wr *prompb.WriteRequest, series []*types.TimeSeriesBinary, data *proto.Buffer) ([]byte, error) { - // Metadata is rarely sent so having this being less than optimal is fine. - wr.Metadata = make([]prompb.MetricMetadata, 0) - for _, ts := range series { - mt, valid := toMetadata(ts) - // TODO @mattdurham somewhere there is a bug where metadata with no labels are being passed through. - if !valid { - level.Error(l).Log("msg", "invalid metadata was found", "labels", ts.Labels.String()) - continue - } - wr.Metadata = append(wr.Metadata, mt) - } - data.Reset() - err := data.Marshal(wr) - return data.Bytes(), err -} - -func getMetadataCount(tss []*types.TimeSeriesBinary) int { - var cnt int - for _, ts := range tss { - if isMetadata(ts) { - cnt++ - } - } - return cnt -} - -func isMetadata(ts *types.TimeSeriesBinary) bool { - return ts.Labels.Has(types.MetaType) && - ts.Labels.Has(types.MetaUnit) && - ts.Labels.Has(types.MetaHelp) -} - -func toMetadata(ts *types.TimeSeriesBinary) (prompb.MetricMetadata, bool) { - if !isMetadata(ts) { - return prompb.MetricMetadata{}, false - } - return prompb.MetricMetadata{ - Type: prompb.MetricMetadata_MetricType(prompb.MetricMetadata_MetricType_value[strings.ToUpper(ts.Labels.Get(types.MetaType))]), - Help: ts.Labels.Get(types.MetaHelp), - Unit: ts.Labels.Get(types.MetaUnit), - MetricFamilyName: ts.Labels.Get("__name__"), - }, true -} - -func retryAfterDuration(defaultDuration time.Duration, t string) time.Duration { - if parsedTime, err := time.Parse(http.TimeFormat, t); err == nil { - return time.Until(parsedTime) - } - // The duration can be in seconds. - d, err := strconv.Atoi(t) - if err != nil { - return defaultDuration - } - return time.Duration(d) * time.Second -} diff --git a/internal/component/prometheus/write/queue/network/manager.go b/internal/component/prometheus/write/queue/network/manager.go deleted file mode 100644 index 0244569403..0000000000 --- a/internal/component/prometheus/write/queue/network/manager.go +++ /dev/null @@ -1,199 +0,0 @@ -package network - -import ( - "context" - "github.com/go-kit/log" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" - "github.com/grafana/alloy/internal/runtime/logging/level" - "github.com/vladopajic/go-actor/actor" -) - -// manager manages loops. Mostly it exists to control their lifecycle and send work to them. -type manager struct { - loops []*loop - metadata *loop - logger log.Logger - inbox actor.Mailbox[*types.TimeSeriesBinary] - metaInbox actor.Mailbox[*types.TimeSeriesBinary] - configInbox actor.Mailbox[configCallback] - self actor.Actor - cfg types.ConnectionConfig - stats func(types.NetworkStats) - metaStats func(types.NetworkStats) -} - -// configCallback allows actors to notify via `done` channel when they're done processing the config `cc`. Useful when synchronous processing is required. -type configCallback struct { - cc types.ConnectionConfig - done chan struct{} -} - -var _ types.NetworkClient = (*manager)(nil) - -var _ actor.Worker = (*manager)(nil) - -func New(cc types.ConnectionConfig, logger log.Logger, seriesStats, metadataStats func(types.NetworkStats)) (types.NetworkClient, error) { - s := &manager{ - loops: make([]*loop, 0, cc.Connections), - logger: logger, - // This provides blocking to only handle one at a time, so that if a queue blocks - // it will stop the filequeue from feeding more. Without OptAsChan the minimum capacity is actually a 64-item buffer. - inbox: actor.NewMailbox[*types.TimeSeriesBinary](actor.OptCapacity(1), actor.OptAsChan()), - metaInbox: actor.NewMailbox[*types.TimeSeriesBinary](actor.OptCapacity(1), actor.OptAsChan()), - configInbox: actor.NewMailbox[configCallback](), - stats: seriesStats, - metaStats: metadataStats, - cfg: cc, - } - - // start kicks off a number of concurrent connections. - for i := uint(0); i < s.cfg.Connections; i++ { - l := newLoop(cc, false, logger, seriesStats) - l.self = actor.New(l) - s.loops = append(s.loops, l) - } - - s.metadata = newLoop(cc, true, logger, metadataStats) - s.metadata.self = actor.New(s.metadata) - return s, nil -} - -func (s *manager) Start() { - s.startLoops() - s.configInbox.Start() - s.metaInbox.Start() - s.inbox.Start() - s.self = actor.New(s) - s.self.Start() -} - -func (s *manager) SendSeries(ctx context.Context, data *types.TimeSeriesBinary) error { - return s.inbox.Send(ctx, data) -} - -func (s *manager) SendMetadata(ctx context.Context, data *types.TimeSeriesBinary) error { - return s.metaInbox.Send(ctx, data) -} - -func (s *manager) UpdateConfig(ctx context.Context, cc types.ConnectionConfig) error { - done := make(chan struct{}) - defer close(done) - err := s.configInbox.Send(ctx, configCallback{ - cc: cc, - done: done, - }) - if err != nil { - return err - } - <-done - return nil -} - -func (s *manager) DoWork(ctx actor.Context) actor.WorkerStatus { - // This acts as a priority queue, always check for configuration changes first. - select { - case cfg, ok := <-s.configInbox.ReceiveC(): - if !ok { - level.Debug(s.logger).Log("msg", "config inbox closed") - return actor.WorkerEnd - } - s.updateConfig(cfg.cc) - // Notify the caller we have applied the config. - cfg.done <- struct{}{} - return actor.WorkerContinue - default: - } - - // main work queue. - select { - case <-ctx.Done(): - s.Stop() - return actor.WorkerEnd - case ts, ok := <-s.inbox.ReceiveC(): - if !ok { - level.Debug(s.logger).Log("msg", "series inbox closed") - return actor.WorkerEnd - } - s.queue(ctx, ts) - return actor.WorkerContinue - case ts, ok := <-s.metaInbox.ReceiveC(): - if !ok { - level.Debug(s.logger).Log("msg", "meta inbox closed") - return actor.WorkerEnd - } - err := s.metadata.seriesMbx.Send(ctx, ts) - if err != nil { - level.Error(s.logger).Log("msg", "failed to send to metadata loop", "err", err) - } - return actor.WorkerContinue - case cfg, ok := <-s.configInbox.ReceiveC(): - // We need to also check the config here, else its possible this will deadlock. - if !ok { - level.Debug(s.logger).Log("msg", "config inbox closed") - return actor.WorkerEnd - } - s.updateConfig(cfg.cc) - // Notify the caller we have applied the config. - cfg.done <- struct{}{} - return actor.WorkerContinue - } -} - -func (s *manager) updateConfig(cc types.ConnectionConfig) { - // No need to do anything if the configuration is the same. - if s.cfg.Equals(cc) { - return - } - s.cfg = cc - // TODO @mattdurham make this smarter, at the moment any samples in the loops are lost. - // Ideally we would drain the queues and re add them but that is a future need. - // In practice this shouldn't change often so data loss should be minimal. - // For the moment we will stop all the items and recreate them. - level.Debug(s.logger).Log("msg", "dropping all series in loops and creating queue due to config change") - s.stopLoops() - s.loops = make([]*loop, 0, s.cfg.Connections) - for i := uint(0); i < s.cfg.Connections; i++ { - l := newLoop(cc, false, s.logger, s.stats) - l.self = actor.New(l) - s.loops = append(s.loops, l) - } - - s.metadata = newLoop(cc, true, s.logger, s.metaStats) - s.metadata.self = actor.New(s.metadata) - level.Debug(s.logger).Log("msg", "starting loops") - s.startLoops() - level.Debug(s.logger).Log("msg", "loops started") -} - -func (s *manager) Stop() { - s.stopLoops() - s.configInbox.Stop() - s.metaInbox.Stop() - s.inbox.Stop() - s.self.Stop() -} - -func (s *manager) stopLoops() { - for _, l := range s.loops { - l.Stop() - } - s.metadata.Stop() -} - -func (s *manager) startLoops() { - for _, l := range s.loops { - l.Start() - } - s.metadata.Start() -} - -// Queue adds anything thats not metadata to the queue. -func (s *manager) queue(ctx context.Context, ts *types.TimeSeriesBinary) { - // Based on a hash which is the label hash add to the queue. - queueNum := ts.Hash % uint64(s.cfg.Connections) - // This will block if the queue is full. - err := s.loops[queueNum].seriesMbx.Send(ctx, ts) - if err != nil { - level.Error(s.logger).Log("msg", "failed to send to loop", "err", err) - } -} diff --git a/internal/component/prometheus/write/queue/network/manager_test.go b/internal/component/prometheus/write/queue/network/manager_test.go deleted file mode 100644 index 947608021a..0000000000 --- a/internal/component/prometheus/write/queue/network/manager_test.go +++ /dev/null @@ -1,311 +0,0 @@ -package network - -import ( - "context" - "github.com/grafana/alloy/internal/util" - "io" - "math/rand" - "net/http" - "net/http/httptest" - "testing" - "time" - - "github.com/go-kit/log" - "github.com/golang/snappy" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" - "github.com/prometheus/prometheus/model/labels" - "github.com/prometheus/prometheus/prompb" - "github.com/stretchr/testify/require" - "go.uber.org/atomic" - "go.uber.org/goleak" -) - -func TestSending(t *testing.T) { - defer goleak.VerifyNone(t) - - recordsFound := atomic.Uint32{} - svr := httptest.NewServer(handler(t, http.StatusOK, func(wr *prompb.WriteRequest) { - recordsFound.Add(uint32(len(wr.Timeseries))) - })) - - defer svr.Close() - ctx := context.Background() - ctx, cncl := context.WithCancel(ctx) - defer cncl() - - cc := types.ConnectionConfig{ - URL: svr.URL, - Timeout: 1 * time.Second, - BatchCount: 10, - FlushInterval: 1 * time.Second, - Connections: 4, - } - - logger := log.NewNopLogger() - wr, err := New(cc, logger, func(s types.NetworkStats) {}, func(s types.NetworkStats) {}) - wr.Start() - defer wr.Stop() - require.NoError(t, err) - for i := 0; i < 1_000; i++ { - send(t, wr, ctx) - } - require.Eventually(t, func() bool { - return recordsFound.Load() == 1_000 - }, 10*time.Second, 100*time.Millisecond) -} - -func TestUpdatingConfig(t *testing.T) { - defer goleak.VerifyNone(t) - - recordsFound := atomic.Uint32{} - lastBatchSize := atomic.Uint32{} - svr := httptest.NewServer(handler(t, http.StatusOK, func(wr *prompb.WriteRequest) { - lastBatchSize.Store(uint32(len(wr.Timeseries))) - recordsFound.Add(uint32(len(wr.Timeseries))) - })) - - defer svr.Close() - - cc := types.ConnectionConfig{ - URL: svr.URL, - Timeout: 1 * time.Second, - BatchCount: 10, - FlushInterval: 5 * time.Second, - Connections: 1, - } - - logger := util.TestAlloyLogger(t) - - wr, err := New(cc, logger, func(s types.NetworkStats) {}, func(s types.NetworkStats) {}) - require.NoError(t, err) - wr.Start() - defer wr.Stop() - - cc2 := types.ConnectionConfig{ - URL: svr.URL, - Timeout: 1 * time.Second, - BatchCount: 20, - FlushInterval: 5 * time.Second, - Connections: 1, - } - ctx := context.Background() - err = wr.UpdateConfig(ctx, cc2) - require.NoError(t, err) - time.Sleep(1 * time.Second) - for i := 0; i < 100; i++ { - send(t, wr, ctx) - } - require.Eventuallyf(t, func() bool { - return recordsFound.Load() == 100 - }, 20*time.Second, 1*time.Second, "record count should be 100 but is %d", recordsFound.Load()) - - require.Truef(t, lastBatchSize.Load() == 20, "batch_count should be 20 but is %d", lastBatchSize.Load()) -} - -func TestRetry(t *testing.T) { - defer goleak.VerifyNone(t) - - retries := atomic.Uint32{} - var previous *prompb.WriteRequest - svr := httptest.NewServer(handler(t, http.StatusTooManyRequests, func(wr *prompb.WriteRequest) { - retries.Add(1) - // Check that we are getting the same sample back. - if previous == nil { - previous = wr - } else { - require.True(t, previous.Timeseries[0].Labels[0].Value == wr.Timeseries[0].Labels[0].Value) - } - })) - defer svr.Close() - ctx := context.Background() - ctx, cncl := context.WithCancel(ctx) - defer cncl() - - cc := types.ConnectionConfig{ - URL: svr.URL, - Timeout: 1 * time.Second, - BatchCount: 1, - FlushInterval: 1 * time.Second, - RetryBackoff: 100 * time.Millisecond, - Connections: 1, - } - - logger := log.NewNopLogger() - wr, err := New(cc, logger, func(s types.NetworkStats) {}, func(s types.NetworkStats) {}) - require.NoError(t, err) - wr.Start() - defer wr.Stop() - send(t, wr, ctx) - - require.Eventually(t, func() bool { - done := retries.Load() > 5 - return done - }, 10*time.Second, 1*time.Second) -} - -func TestRetryBounded(t *testing.T) { - defer goleak.VerifyNone(t) - - sends := atomic.Uint32{} - svr := httptest.NewServer(handler(t, http.StatusTooManyRequests, func(wr *prompb.WriteRequest) { - sends.Add(1) - })) - - defer svr.Close() - ctx := context.Background() - ctx, cncl := context.WithCancel(ctx) - defer cncl() - - cc := types.ConnectionConfig{ - URL: svr.URL, - Timeout: 1 * time.Second, - BatchCount: 1, - FlushInterval: 1 * time.Second, - RetryBackoff: 100 * time.Millisecond, - MaxRetryAttempts: 1, - Connections: 1, - } - - logger := log.NewNopLogger() - wr, err := New(cc, logger, func(s types.NetworkStats) {}, func(s types.NetworkStats) {}) - wr.Start() - defer wr.Stop() - require.NoError(t, err) - for i := 0; i < 10; i++ { - send(t, wr, ctx) - } - require.Eventually(t, func() bool { - // We send 10 but each one gets retried once so 20 total. - return sends.Load() == 10*2 - }, 2*time.Second, 100*time.Millisecond) - time.Sleep(2 * time.Second) - // Ensure we dont get any more. - require.True(t, sends.Load() == 10*2) -} - -func TestRecoverable(t *testing.T) { - defer goleak.VerifyNone(t) - - recoverable := atomic.Uint32{} - svr := httptest.NewServer(handler(t, http.StatusInternalServerError, func(wr *prompb.WriteRequest) { - })) - defer svr.Close() - ctx := context.Background() - ctx, cncl := context.WithCancel(ctx) - defer cncl() - - cc := types.ConnectionConfig{ - URL: svr.URL, - Timeout: 1 * time.Second, - BatchCount: 1, - FlushInterval: 1 * time.Second, - RetryBackoff: 100 * time.Millisecond, - MaxRetryAttempts: 1, - Connections: 1, - } - - logger := log.NewNopLogger() - wr, err := New(cc, logger, func(s types.NetworkStats) { - recoverable.Add(uint32(s.Total5XX())) - }, func(s types.NetworkStats) {}) - require.NoError(t, err) - wr.Start() - defer wr.Stop() - for i := 0; i < 10; i++ { - send(t, wr, ctx) - } - require.Eventually(t, func() bool { - // We send 10 but each one gets retried once so 20 total. - return recoverable.Load() == 10*2 - }, 2*time.Second, 100*time.Millisecond) - time.Sleep(2 * time.Second) - // Ensure we dont get any more. - require.True(t, recoverable.Load() == 10*2) -} - -func TestNonRecoverable(t *testing.T) { - defer goleak.VerifyNone(t) - - nonRecoverable := atomic.Uint32{} - svr := httptest.NewServer(handler(t, http.StatusBadRequest, func(wr *prompb.WriteRequest) { - })) - - defer svr.Close() - ctx := context.Background() - ctx, cncl := context.WithCancel(ctx) - defer cncl() - - cc := types.ConnectionConfig{ - URL: svr.URL, - Timeout: 1 * time.Second, - BatchCount: 1, - FlushInterval: 1 * time.Second, - RetryBackoff: 100 * time.Millisecond, - MaxRetryAttempts: 1, - Connections: 1, - } - - logger := log.NewNopLogger() - wr, err := New(cc, logger, func(s types.NetworkStats) { - nonRecoverable.Add(uint32(s.TotalFailed())) - }, func(s types.NetworkStats) {}) - wr.Start() - defer wr.Stop() - require.NoError(t, err) - for i := 0; i < 10; i++ { - send(t, wr, ctx) - } - require.Eventually(t, func() bool { - return nonRecoverable.Load() == 10 - }, 2*time.Second, 100*time.Millisecond) - time.Sleep(2 * time.Second) - // Ensure we dont get any more. - require.True(t, nonRecoverable.Load() == 10) -} - -func send(t *testing.T, wr types.NetworkClient, ctx context.Context) { - ts := createSeries(t) - // The actual hash is only used for queueing into different buckets. - err := wr.SendSeries(ctx, ts) - require.NoError(t, err) -} - -func handler(t *testing.T, code int, callback func(wr *prompb.WriteRequest)) http.HandlerFunc { - return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - buf, err := io.ReadAll(r.Body) - require.NoError(t, err) - defer r.Body.Close() - decoded, err := snappy.Decode(nil, buf) - require.NoError(t, err) - - wr := &prompb.WriteRequest{} - err = wr.Unmarshal(decoded) - require.NoError(t, err) - callback(wr) - w.WriteHeader(code) - }) -} - -func createSeries(_ *testing.T) *types.TimeSeriesBinary { - ts := &types.TimeSeriesBinary{ - TS: time.Now().Unix(), - Value: 1, - Labels: []labels.Label{ - { - Name: "__name__", - Value: randSeq(10), - }, - }, - } - return ts -} - -var letters = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ") - -func randSeq(n int) string { - b := make([]rune, n) - for i := range b { - b[i] = letters[rand.Intn(len(letters))] - } - return string(b) -} diff --git a/internal/component/prometheus/write/queue/network/stats.go b/internal/component/prometheus/write/queue/network/stats.go deleted file mode 100644 index 345069e1cb..0000000000 --- a/internal/component/prometheus/write/queue/network/stats.go +++ /dev/null @@ -1,126 +0,0 @@ -package network - -import ( - "net/http" - - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" -) - -// recordStats determines what values to send to the stats function. This allows for any -// number of metrics/signals libraries to be used. Prometheus, OTel, and any other. -func recordStats(series []*types.TimeSeriesBinary, isMeta bool, stats func(s types.NetworkStats), r sendResult, bytesSent int) { - seriesCount := getSeriesCount(series) - histogramCount := getHistogramCount(series) - metadataCount := getMetadataCount(series) - switch { - case r.networkError: - stats(types.NetworkStats{ - Series: types.CategoryStats{ - NetworkSamplesFailed: seriesCount, - }, - Histogram: types.CategoryStats{ - NetworkSamplesFailed: histogramCount, - }, - Metadata: types.CategoryStats{ - NetworkSamplesFailed: metadataCount, - }, - }) - case r.successful: - // Need to grab the newest series. - var newestTS int64 - for _, ts := range series { - if ts.TS > newestTS { - newestTS = ts.TS - } - } - var sampleBytesSent int - var metaBytesSent int - // Each loop is explicitly a normal signal or metadata sender. - if isMeta { - metaBytesSent = bytesSent - } else { - sampleBytesSent = bytesSent - } - stats(types.NetworkStats{ - Series: types.CategoryStats{ - SeriesSent: seriesCount, - }, - Histogram: types.CategoryStats{ - SeriesSent: histogramCount, - }, - Metadata: types.CategoryStats{ - SeriesSent: metadataCount, - }, - MetadataBytes: metaBytesSent, - SeriesBytes: sampleBytesSent, - NewestTimestamp: newestTS, - }) - case r.statusCode == http.StatusTooManyRequests: - stats(types.NetworkStats{ - Series: types.CategoryStats{ - RetriedSamples: seriesCount, - RetriedSamples429: seriesCount, - }, - Histogram: types.CategoryStats{ - RetriedSamples: histogramCount, - RetriedSamples429: histogramCount, - }, - Metadata: types.CategoryStats{ - RetriedSamples: metadataCount, - RetriedSamples429: metadataCount, - }, - }) - case r.statusCode/100 == 5: - stats(types.NetworkStats{ - Series: types.CategoryStats{ - RetriedSamples5XX: seriesCount, - }, - Histogram: types.CategoryStats{ - RetriedSamples5XX: histogramCount, - }, - Metadata: types.CategoryStats{ - RetriedSamples: metadataCount, - }, - }) - case r.statusCode != 200: - stats(types.NetworkStats{ - Series: types.CategoryStats{ - FailedSamples: seriesCount, - }, - Histogram: types.CategoryStats{ - FailedSamples: histogramCount, - }, - Metadata: types.CategoryStats{ - FailedSamples: metadataCount, - }, - }) - } - -} - -func getSeriesCount(tss []*types.TimeSeriesBinary) int { - cnt := 0 - for _, ts := range tss { - // This is metadata - if isMetadata(ts) { - continue - } - if ts.Histograms.Histogram == nil && ts.Histograms.FloatHistogram == nil { - cnt++ - } - } - return cnt -} - -func getHistogramCount(tss []*types.TimeSeriesBinary) int { - cnt := 0 - for _, ts := range tss { - if isMetadata(ts) { - continue - } - if ts.Histograms.Histogram != nil || ts.Histograms.FloatHistogram != nil { - cnt++ - } - } - return cnt -} diff --git a/internal/component/prometheus/write/queue/serialization/appender.go b/internal/component/prometheus/write/queue/serialization/appender.go deleted file mode 100644 index 3e8515a19b..0000000000 --- a/internal/component/prometheus/write/queue/serialization/appender.go +++ /dev/null @@ -1,130 +0,0 @@ -package serialization - -import ( - "context" - "fmt" - "time" - - "github.com/go-kit/log" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" - "github.com/prometheus/prometheus/model/exemplar" - "github.com/prometheus/prometheus/model/histogram" - "github.com/prometheus/prometheus/model/labels" - "github.com/prometheus/prometheus/model/metadata" - "github.com/prometheus/prometheus/storage" -) - -type appender struct { - ctx context.Context - ttl time.Duration - s types.Serializer - logger log.Logger -} - -func (a *appender) AppendCTZeroSample(ref storage.SeriesRef, l labels.Labels, t, ct int64) (storage.SeriesRef, error) { - // TODO @mattdurham figure out what to do here later. This mirrors what we do elsewhere. - return ref, nil -} - -// NewAppender returns an Appender that writes to a given serializer. NOTE the returned Appender writes -// data immediately, discards data older than `ttl` and does not honor commit or rollback. -func NewAppender(ctx context.Context, ttl time.Duration, s types.Serializer, logger log.Logger) storage.Appender { - app := &appender{ - ttl: ttl, - s: s, - logger: logger, - ctx: ctx, - } - return app -} - -// Append metric -func (a *appender) Append(ref storage.SeriesRef, l labels.Labels, t int64, v float64) (storage.SeriesRef, error) { - // Check to see if the TTL has expired for this record. - endTime := time.Now().Unix() - int64(a.ttl.Seconds()) - if t < endTime { - return ref, nil - } - ts := types.GetTimeSeriesFromPool() - ts.Labels = l - ts.TS = t - ts.Value = v - ts.Hash = l.Hash() - err := a.s.SendSeries(a.ctx, ts) - return ref, err -} - -// Commit is a no op since we always write. -func (a *appender) Commit() (_ error) { - return nil -} - -// Rollback is a no op since we write all the data. -func (a *appender) Rollback() error { - return nil -} - -// AppendExemplar appends exemplar to cache. The passed in labels is unused, instead use the labels on the exemplar. -func (a *appender) AppendExemplar(ref storage.SeriesRef, _ labels.Labels, e exemplar.Exemplar) (_ storage.SeriesRef, _ error) { - endTime := time.Now().Unix() - int64(a.ttl.Seconds()) - if e.HasTs && e.Ts < endTime { - return ref, nil - } - ts := types.GetTimeSeriesFromPool() - ts.Hash = e.Labels.Hash() - ts.TS = e.Ts - ts.Labels = e.Labels - ts.Hash = e.Labels.Hash() - err := a.s.SendSeries(a.ctx, ts) - return ref, err -} - -// AppendHistogram appends histogram -func (a *appender) AppendHistogram(ref storage.SeriesRef, l labels.Labels, t int64, h *histogram.Histogram, fh *histogram.FloatHistogram) (_ storage.SeriesRef, _ error) { - endTime := time.Now().Unix() - int64(a.ttl.Seconds()) - if t < endTime { - return ref, nil - } - ts := types.GetTimeSeriesFromPool() - ts.Labels = l - ts.TS = t - if h != nil { - ts.FromHistogram(t, h) - } else { - ts.FromFloatHistogram(t, fh) - } - ts.Hash = l.Hash() - err := a.s.SendSeries(a.ctx, ts) - return ref, err -} - -// UpdateMetadata updates metadata. -func (a *appender) UpdateMetadata(ref storage.SeriesRef, l labels.Labels, m metadata.Metadata) (_ storage.SeriesRef, _ error) { - if !l.Has("__name__") { - return ref, fmt.Errorf("missing __name__ label for metadata") - } - ts := types.GetTimeSeriesFromPool() - // We are going to handle converting some strings to hopefully not reused label names. TimeSeriesBinary has a lot of work - // to ensure its efficient it makes sense to encode metadata into it. - combinedLabels := labels.EmptyLabels() - combinedLabels = append(combinedLabels, labels.Label{ - Name: types.MetaType, - Value: string(m.Type), - }) - combinedLabels = append(combinedLabels, labels.Label{ - Name: types.MetaHelp, - Value: m.Help, - }) - combinedLabels = append(combinedLabels, labels.Label{ - Name: types.MetaUnit, - Value: m.Unit, - }) - // We ONLY want __name__ from labels - combinedLabels = append(combinedLabels, labels.Label{ - Name: "__name__", - Value: l.Get("__name__"), - }) - ts.Labels = combinedLabels - err := a.s.SendMetadata(a.ctx, ts) - return ref, err -} diff --git a/internal/component/prometheus/write/queue/serialization/appender_test.go b/internal/component/prometheus/write/queue/serialization/appender_test.go deleted file mode 100644 index 0215eeee6e..0000000000 --- a/internal/component/prometheus/write/queue/serialization/appender_test.go +++ /dev/null @@ -1,55 +0,0 @@ -package serialization - -import ( - "context" - log2 "github.com/go-kit/log" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" - "github.com/prometheus/prometheus/model/labels" - "github.com/stretchr/testify/require" - "testing" - "time" -) - -func TestAppenderTTL(t *testing.T) { - fake := &counterSerializer{} - l := log2.NewNopLogger() - - app := NewAppender(context.Background(), 1*time.Minute, fake, l) - _, err := app.Append(0, labels.FromStrings("one", "two"), time.Now().Unix(), 0) - require.NoError(t, err) - - for i := 0; i < 10; i++ { - _, err = app.Append(0, labels.FromStrings("one", "two"), time.Now().Add(-5*time.Minute).Unix(), 0) - require.NoError(t, err) - } - // Only one record should make it through. - require.True(t, fake.received == 1) -} - -var _ types.Serializer = (*fakeSerializer)(nil) - -type counterSerializer struct { - received int -} - -func (f *counterSerializer) Start() { - -} - -func (f *counterSerializer) Stop() { - -} - -func (f *counterSerializer) SendSeries(ctx context.Context, data *types.TimeSeriesBinary) error { - f.received++ - return nil - -} - -func (f *counterSerializer) SendMetadata(ctx context.Context, data *types.TimeSeriesBinary) error { - return nil -} - -func (f *counterSerializer) UpdateConfig(ctx context.Context, data types.SerializerConfig) error { - return nil -} diff --git a/internal/component/prometheus/write/queue/serialization/serializer.go b/internal/component/prometheus/write/queue/serialization/serializer.go deleted file mode 100644 index 71c96cb3f9..0000000000 --- a/internal/component/prometheus/write/queue/serialization/serializer.go +++ /dev/null @@ -1,222 +0,0 @@ -package serialization - -import ( - "context" - "fmt" - "strconv" - "time" - - snappy "github.com/eapache/go-xerial-snappy" - "github.com/go-kit/log" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" - "github.com/grafana/alloy/internal/runtime/logging/level" - "github.com/vladopajic/go-actor/actor" - "go.uber.org/atomic" -) - -// serializer collects data from multiple appenders in-memory and will periodically flush the data to file.Storage. -// serializer will flush based on configured time duration OR if it hits a certain number of items. -type serializer struct { - inbox actor.Mailbox[*types.TimeSeriesBinary] - metaInbox actor.Mailbox[*types.TimeSeriesBinary] - cfgInbox actor.Mailbox[types.SerializerConfig] - maxItemsBeforeFlush int - flushFrequency time.Duration - queue types.FileStorage - lastFlush time.Time - logger log.Logger - self actor.Actor - // Every 1 second we should check if we need to flush. - flushTestTimer *time.Ticker - series []*types.TimeSeriesBinary - meta []*types.TimeSeriesBinary - msgpBuffer []byte - stats func(stats types.SerializerStats) - stopped *atomic.Bool -} - -func NewSerializer(cfg types.SerializerConfig, q types.FileStorage, stats func(stats types.SerializerStats), l log.Logger) (types.Serializer, error) { - s := &serializer{ - maxItemsBeforeFlush: int(cfg.MaxSignalsInBatch), - flushFrequency: cfg.FlushFrequency, - queue: q, - series: make([]*types.TimeSeriesBinary, 0), - logger: l, - inbox: actor.NewMailbox[*types.TimeSeriesBinary](), - metaInbox: actor.NewMailbox[*types.TimeSeriesBinary](), - cfgInbox: actor.NewMailbox[types.SerializerConfig](), - flushTestTimer: time.NewTicker(1 * time.Second), - msgpBuffer: make([]byte, 0), - lastFlush: time.Now(), - stats: stats, - stopped: atomic.NewBool(false), - } - - return s, nil -} -func (s *serializer) Start() { - // All the actors and mailboxes need to start. - s.queue.Start() - s.self = actor.Combine(actor.New(s), s.inbox, s.metaInbox, s.cfgInbox).Build() - s.self.Start() -} - -func (s *serializer) Stop() { - s.stopped.Store(true) - s.queue.Stop() - s.self.Stop() -} - -func (s *serializer) SendSeries(ctx context.Context, data *types.TimeSeriesBinary) error { - if s.stopped.Load() { - return fmt.Errorf("serializer is stopped") - } - return s.inbox.Send(ctx, data) -} - -func (s *serializer) SendMetadata(ctx context.Context, data *types.TimeSeriesBinary) error { - if s.stopped.Load() { - return fmt.Errorf("serializer is stopped") - } - return s.metaInbox.Send(ctx, data) -} - -func (s *serializer) UpdateConfig(ctx context.Context, cfg types.SerializerConfig) error { - if s.stopped.Load() { - return fmt.Errorf("serializer is stopped") - } - return s.cfgInbox.Send(ctx, cfg) -} - -func (s *serializer) DoWork(ctx actor.Context) actor.WorkerStatus { - // Check for config which should have priority. Selector is random but since incoming - // series will always have a queue by explicitly checking the config here we always give it a chance. - // By pulling the config from the mailbox we ensure it does NOT need a mutex around access. - select { - case <-ctx.Done(): - return actor.WorkerEnd - case cfg, ok := <-s.cfgInbox.ReceiveC(): - if !ok { - return actor.WorkerEnd - } - s.maxItemsBeforeFlush = int(cfg.MaxSignalsInBatch) - s.flushFrequency = cfg.FlushFrequency - return actor.WorkerContinue - default: - } - - select { - case <-ctx.Done(): - return actor.WorkerEnd - case item, ok := <-s.inbox.ReceiveC(): - if !ok { - return actor.WorkerEnd - } - s.series = append(s.series, item) - // If we would go over the max size then send, or if we have hit the flush duration then send. - if len(s.meta)+len(s.series) >= s.maxItemsBeforeFlush { - err := s.flushToDisk(ctx) - if err != nil { - level.Error(s.logger).Log("msg", "unable to append to serializer", "err", err) - } - } - - return actor.WorkerContinue - case item, ok := <-s.metaInbox.ReceiveC(): - if !ok { - return actor.WorkerEnd - } - s.meta = append(s.meta, item) - if len(s.meta)+len(s.series) >= s.maxItemsBeforeFlush { - err := s.flushToDisk(ctx) - if err != nil { - level.Error(s.logger).Log("msg", "unable to append metadata to serializer", "err", err) - } - } - return actor.WorkerContinue - case <-s.flushTestTimer.C: - if time.Since(s.lastFlush) > s.flushFrequency { - err := s.flushToDisk(ctx) - if err != nil { - level.Error(s.logger).Log("msg", "unable to store data", "err", err) - } - } - return actor.WorkerContinue - } -} - -func (s *serializer) flushToDisk(ctx actor.Context) error { - var err error - defer func() { - s.lastFlush = time.Now() - }() - // Do nothing if there is nothing. - if len(s.series) == 0 && len(s.meta) == 0 { - return nil - } - group := &types.SeriesGroup{ - Series: make([]*types.TimeSeriesBinary, len(s.series)), - Metadata: make([]*types.TimeSeriesBinary, len(s.meta)), - } - defer func() { - s.storeStats(err) - // Return series to the pool, this is key to reducing allocs. - types.PutTimeSeriesSliceIntoPool(s.series) - types.PutTimeSeriesSliceIntoPool(s.meta) - s.series = s.series[:0] - s.meta = s.meta[:0] - }() - - // This maps strings to index position in a slice. This is doing to reduce the file size of the data. - strMapToIndex := make(map[string]uint32) - for i, ts := range s.series { - ts.FillLabelMapping(strMapToIndex) - group.Series[i] = ts - } - for i, ts := range s.meta { - ts.FillLabelMapping(strMapToIndex) - group.Metadata[i] = ts - } - - stringsSlice := make([]string, len(strMapToIndex)) - for stringValue, index := range strMapToIndex { - stringsSlice[index] = stringValue - } - group.Strings = stringsSlice - - buf, err := group.MarshalMsg(s.msgpBuffer) - if err != nil { - return err - } - - out := snappy.Encode(buf) - meta := map[string]string{ - // product.signal_type.schema.version - "version": types.AlloyFileVersion, - "compression": "snappy", - "series_count": strconv.Itoa(len(group.Series)), - "meta_count": strconv.Itoa(len(group.Metadata)), - "strings_count": strconv.Itoa(len(group.Strings)), - } - err = s.queue.Store(ctx, meta, out) - return err -} - -func (s *serializer) storeStats(err error) { - hasError := 0 - if err != nil { - hasError = 1 - } - newestTS := int64(0) - for _, ts := range s.series { - if ts.TS > newestTS { - newestTS = ts.TS - } - } - s.stats(types.SerializerStats{ - SeriesStored: len(s.series), - MetadataStored: len(s.meta), - Errors: hasError, - NewestTimestamp: newestTS, - }) -} diff --git a/internal/component/prometheus/write/queue/serialization/serializer_bench_test.go b/internal/component/prometheus/write/queue/serialization/serializer_bench_test.go deleted file mode 100644 index 8d30591159..0000000000 --- a/internal/component/prometheus/write/queue/serialization/serializer_bench_test.go +++ /dev/null @@ -1,117 +0,0 @@ -package serialization - -import ( - "context" - "fmt" - "math/rand" - "testing" - "time" - - "github.com/go-kit/log" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" - "github.com/prometheus/prometheus/model/labels" -) - -var lbls = labels.FromStrings("one", "two", "three", "four") - -func BenchmarkAppender(b *testing.B) { - // This should be 0 allocs - b.ReportAllocs() - logger := log.NewNopLogger() - for i := 0; i < b.N; i++ { - app := NewAppender(context.Background(), 1*time.Hour, &fakeSerializer{}, logger) - for j := 0; j < 10_000; j++ { - _, _ = app.Append(0, lbls, time.Now().Unix(), 1.1) - } - _ = app.Commit() - } -} - -func BenchmarkSerializer(b *testing.B) { - b.ResetTimer() - b.ReportAllocs() - // This should be ~11 allocs and 1400-1800 ns/op. - logger := log.NewNopLogger() - serial, _ := NewSerializer(types.SerializerConfig{ - MaxSignalsInBatch: 1_000, - FlushFrequency: 1 * time.Second, - }, &fakeFileQueue{}, func(stats types.SerializerStats) {}, logger) - serial.Start() - for i := 0; i < b.N; i++ { - _ = serial.SendSeries(context.Background(), getSingleTimeSeries(b)) - } - serial.Stop() -} - -func getTimeSeries(b *testing.B) []*types.TimeSeriesBinary { - b.Helper() - series := make([]*types.TimeSeriesBinary, 0) - for j := 0; j < 10_000; j++ { - timeseries := types.GetTimeSeriesFromPool() - timeseries.TS = time.Now().Unix() - timeseries.Value = rand.Float64() - timeseries.Labels = getLabels() - series = append(series, timeseries) - } - return series -} - -func getSingleTimeSeries(b *testing.B) *types.TimeSeriesBinary { - b.Helper() - timeseries := types.GetTimeSeriesFromPool() - timeseries.TS = time.Now().Unix() - timeseries.Value = rand.Float64() - timeseries.Labels = getLabels() - return timeseries - -} - -func getLabels() labels.Labels { - retLbls := make(labels.Labels, 0) - for i := 0; i < rand.Intn(20); i++ { - l := labels.Label{ - Name: fmt.Sprintf("label_%d", i), - Value: fmt.Sprintf("value_%d", i), - } - retLbls = append(retLbls, l) - } - return retLbls -} - -var _ types.Serializer = (*fakeSerializer)(nil) - -type fakeSerializer struct{} - -func (f *fakeSerializer) UpdateConfig(ctx context.Context, cfg types.SerializerConfig) error { - return nil -} - -func (f *fakeSerializer) Start() {} - -func (f *fakeSerializer) Stop() {} - -func (f *fakeSerializer) SendSeries(ctx context.Context, data *types.TimeSeriesBinary) error { - types.PutTimeSeriesIntoPool(data) - return nil -} - -func (f *fakeSerializer) SendMetadata(ctx context.Context, data *types.TimeSeriesBinary) error { - types.PutTimeSeriesIntoPool(data) - return nil -} - -var _ types.FileStorage = (*fakeFileQueue)(nil) - -type fakeFileQueue struct{} - -func (f fakeFileQueue) Start() { - -} - -func (f fakeFileQueue) Stop() { - -} - -func (f fakeFileQueue) Store(ctx context.Context, meta map[string]string, value []byte) error { - return nil -} diff --git a/internal/component/prometheus/write/queue/serialization/serializer_test.go b/internal/component/prometheus/write/queue/serialization/serializer_test.go deleted file mode 100644 index 80054a24a0..0000000000 --- a/internal/component/prometheus/write/queue/serialization/serializer_test.go +++ /dev/null @@ -1,113 +0,0 @@ -//go:build !race - -package serialization - -import ( - "context" - "fmt" - "sync/atomic" - "testing" - "time" - - "github.com/go-kit/log" - "github.com/golang/snappy" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" - "github.com/prometheus/prometheus/model/labels" - "github.com/stretchr/testify/require" -) - -func TestRoundTripSerialization(t *testing.T) { - totalSeries := atomic.Int64{} - f := &fqq{t: t} - l := log.NewNopLogger() - start := time.Now().Add(-1 * time.Second).Unix() - - s, err := NewSerializer(types.SerializerConfig{ - MaxSignalsInBatch: 10, - FlushFrequency: 5 * time.Second, - }, f, func(stats types.SerializerStats) { - totalSeries.Add(int64(stats.SeriesStored)) - require.True(t, stats.SeriesStored == 10) - require.True(t, stats.Errors == 0) - require.True(t, stats.MetadataStored == 0) - require.True(t, stats.NewestTimestamp > start) - }, l) - require.NoError(t, err) - - s.Start() - defer s.Stop() - for i := 0; i < 100; i++ { - tss := types.GetTimeSeriesFromPool() - tss.Labels = make(labels.Labels, 10) - for j := 0; j < 10; j++ { - tss.Labels[j] = labels.Label{ - Name: fmt.Sprintf("name_%d_%d", i, j), - Value: fmt.Sprintf("value_%d_%d", i, j), - } - tss.Value = float64(i) - tss.TS = time.Now().Unix() - } - sendErr := s.SendSeries(context.Background(), tss) - require.NoError(t, sendErr) - } - require.Eventually(t, func() bool { - return f.total.Load() == 100 - }, 5*time.Second, 100*time.Millisecond) - // 100 series send from the above for loop - require.True(t, totalSeries.Load() == 100) -} - -func TestUpdateConfig(t *testing.T) { - f := &fqq{t: t} - l := log.NewNopLogger() - s, err := NewSerializer(types.SerializerConfig{ - MaxSignalsInBatch: 10, - FlushFrequency: 5 * time.Second, - }, f, func(stats types.SerializerStats) {}, l) - require.NoError(t, err) - s.Start() - defer s.Stop() - err = s.UpdateConfig(context.Background(), types.SerializerConfig{ - MaxSignalsInBatch: 1, - FlushFrequency: 1 * time.Second, - }) - require.NoError(t, err) - require.Eventually(t, func() bool { - return s.(*serializer).maxItemsBeforeFlush == 1 && s.(*serializer).flushFrequency == 1*time.Second - }, 5*time.Second, 100*time.Millisecond) -} - -var _ types.FileStorage = (*fqq)(nil) - -type fqq struct { - t *testing.T - buf []byte - total atomic.Int64 -} - -func (f *fqq) Start() { - -} - -func (f *fqq) Stop() { - -} - -func (f *fqq) Store(ctx context.Context, meta map[string]string, value []byte) error { - f.buf, _ = snappy.Decode(nil, value) - sg := &types.SeriesGroup{} - sg, _, err := types.DeserializeToSeriesGroup(sg, f.buf) - require.NoError(f.t, err) - require.Len(f.t, sg.Series, 10) - for _, series := range sg.Series { - require.Len(f.t, series.LabelsNames, 0) - require.Len(f.t, series.LabelsValues, 0) - require.Len(f.t, series.Labels, 10) - for j := 0; j < 10; j++ { - series.Labels[j].Name = fmt.Sprintf("name_%d_%d", int(series.Value), j) - series.Labels[j].Value = fmt.Sprintf("value_%d_%d", int(series.Value), j) - } - } - f.total.Add(int64(len(sg.Series))) - return nil -} diff --git a/internal/component/prometheus/write/queue/types.go b/internal/component/prometheus/write/queue/types.go index b56e391d3c..ffffe287ea 100644 --- a/internal/component/prometheus/write/queue/types.go +++ b/internal/component/prometheus/write/queue/types.go @@ -4,8 +4,8 @@ import ( "fmt" "time" - "github.com/grafana/alloy/internal/component/prometheus/write/queue/types" "github.com/grafana/alloy/syntax/alloytypes" + "github.com/grafana/walqueue/types" "github.com/prometheus/common/version" "github.com/prometheus/prometheus/storage" ) @@ -96,7 +96,7 @@ var UserAgent = fmt.Sprintf("Alloy/%s", version.Version) func (cc EndpointConfig) ToNativeType() types.ConnectionConfig { tcc := types.ConnectionConfig{ URL: cc.URL, - BearerToken: cc.BearerToken, + BearerToken: string(cc.BearerToken), UserAgent: UserAgent, Timeout: cc.Timeout, RetryBackoff: cc.RetryBackoff, diff --git a/internal/component/prometheus/write/queue/types/messages.go b/internal/component/prometheus/write/queue/types/messages.go deleted file mode 100644 index 30c37961c7..0000000000 --- a/internal/component/prometheus/write/queue/types/messages.go +++ /dev/null @@ -1,12 +0,0 @@ -package types - -type Data struct { - Meta map[string]string - Data []byte -} - -type DataHandle struct { - Name string - // Pop will get the data and delete the source of the data. - Pop func() (map[string]string, []byte, error) -} diff --git a/internal/component/prometheus/write/queue/types/network.go b/internal/component/prometheus/write/queue/types/network.go deleted file mode 100644 index 23bbe4d2e7..0000000000 --- a/internal/component/prometheus/write/queue/types/network.go +++ /dev/null @@ -1,42 +0,0 @@ -package types - -import ( - "context" - "github.com/grafana/alloy/syntax/alloytypes" - "reflect" - "time" -) - -type NetworkClient interface { - Start() - Stop() - // SendSeries will block if the network caches are full. - SendSeries(ctx context.Context, d *TimeSeriesBinary) error - // SendMetadata will block if the network caches are full. - SendMetadata(ctx context.Context, d *TimeSeriesBinary) error - // UpdateConfig is a synchronous call and will only return once the config - // is applied or an error occurs. - UpdateConfig(ctx context.Context, cfg ConnectionConfig) error -} -type ConnectionConfig struct { - URL string - BasicAuth *BasicAuth - BearerToken alloytypes.Secret - UserAgent string - Timeout time.Duration - RetryBackoff time.Duration - MaxRetryAttempts uint - BatchCount int - FlushInterval time.Duration - ExternalLabels map[string]string - Connections uint -} - -type BasicAuth struct { - Username string - Password string -} - -func (cc ConnectionConfig) Equals(bb ConnectionConfig) bool { - return reflect.DeepEqual(cc, bb) -} diff --git a/internal/component/prometheus/write/queue/types/serialization.go b/internal/component/prometheus/write/queue/types/serialization.go deleted file mode 100644 index 80b2282f7d..0000000000 --- a/internal/component/prometheus/write/queue/types/serialization.go +++ /dev/null @@ -1,296 +0,0 @@ -//go:generate msgp -package types - -import ( - "sync" - - "github.com/prometheus/prometheus/model/histogram" - "github.com/prometheus/prometheus/model/labels" - "github.com/prometheus/prometheus/prompb" - "go.uber.org/atomic" -) - -const MetaType = "__alloy_metadata_type__" -const MetaUnit = "__alloy_metadata_unit__" -const MetaHelp = "__alloy_metadata_help__" - -// SeriesGroup is the holder for TimeSeries, Metadata, and the strings array. -// When serialized the Labels Key,Value array will be transformed into -// LabelNames and LabelsValues that point to the index in Strings. -// This deduplicates the strings and decreases the size on disk. -type SeriesGroup struct { - Strings []string - Series []*TimeSeriesBinary - Metadata []*TimeSeriesBinary -} - -// TimeSeriesBinary is an optimized format for handling metrics and metadata. It should never be instantiated directly -// but instead use GetTimeSeriesFromPool and PutTimeSeriesSliceIntoPool. This allows us to reuse these objects and avoid -// allocations. -type TimeSeriesBinary struct { - // Labels are not serialized to msgp, instead we store separately a dictionary of strings and use `LabelNames` and `LabelValues` to refer to the dictionary by ID. - Labels labels.Labels `msg:"-"` - LabelsNames []uint32 - LabelsValues []uint32 - TS int64 - Value float64 - Hash uint64 - Histograms Histograms -} - -type Histograms struct { - Histogram *Histogram - FloatHistogram *FloatHistogram -} - -type Histogram struct { - Count HistogramCount - Sum float64 - Schema int32 - ZeroThreshold float64 - ZeroCount HistogramZeroCount - NegativeSpans []BucketSpan - NegativeBuckets []int64 - NegativeCounts []float64 - PositiveSpans []BucketSpan - PositiveBuckets []int64 - PositiveCounts []float64 - ResetHint int32 - TimestampMillisecond int64 -} - -type FloatHistogram struct { - Count HistogramCount - Sum float64 - Schema int32 - ZeroThreshold float64 - ZeroCount HistogramZeroCount - NegativeSpans []BucketSpan - NegativeDeltas []int64 - NegativeCounts []float64 - PositiveSpans []BucketSpan - PositiveDeltas []int64 - PositiveCounts []float64 - ResetHint int32 - TimestampMillisecond int64 -} - -type HistogramCount struct { - IsInt bool - IntValue uint64 - FloatValue float64 -} - -type HistogramZeroCount struct { - IsInt bool - IntValue uint64 - FloatValue float64 -} - -type BucketSpan struct { - Offset int32 - Length uint32 -} - -// IsMetadata is used because it's easier to store metadata as a set of labels. -func (ts TimeSeriesBinary) IsMetadata() bool { - return ts.Labels.Has("__alloy_metadata_type__") -} - -func (h *Histogram) ToPromHistogram() prompb.Histogram { - return prompb.Histogram{ - Count: &prompb.Histogram_CountInt{CountInt: h.Count.IntValue}, - Sum: h.Sum, - Schema: h.Schema, - ZeroThreshold: h.ZeroThreshold, - ZeroCount: &prompb.Histogram_ZeroCountInt{ZeroCountInt: h.ZeroCount.IntValue}, - NegativeSpans: ToPromBucketSpans(h.NegativeSpans), - NegativeDeltas: h.NegativeBuckets, - PositiveSpans: ToPromBucketSpans(h.PositiveSpans), - PositiveDeltas: h.PositiveBuckets, - ResetHint: prompb.Histogram_ResetHint(h.ResetHint), - Timestamp: h.TimestampMillisecond, - } -} - -func (h *FloatHistogram) ToPromFloatHistogram() prompb.Histogram { - return prompb.Histogram{ - Count: &prompb.Histogram_CountFloat{CountFloat: h.Count.FloatValue}, - Sum: h.Sum, - Schema: h.Schema, - ZeroThreshold: h.ZeroThreshold, - ZeroCount: &prompb.Histogram_ZeroCountFloat{ZeroCountFloat: h.ZeroCount.FloatValue}, - NegativeSpans: ToPromBucketSpans(h.NegativeSpans), - NegativeCounts: h.NegativeCounts, - PositiveSpans: ToPromBucketSpans(h.PositiveSpans), - PositiveCounts: h.PositiveCounts, - ResetHint: prompb.Histogram_ResetHint(h.ResetHint), - Timestamp: h.TimestampMillisecond, - } -} -func ToPromBucketSpans(bss []BucketSpan) []prompb.BucketSpan { - spans := make([]prompb.BucketSpan, len(bss)) - for i, bs := range bss { - spans[i] = bs.ToPromBucketSpan() - } - return spans -} - -func (bs *BucketSpan) ToPromBucketSpan() prompb.BucketSpan { - return prompb.BucketSpan{ - Offset: bs.Offset, - Length: bs.Length, - } -} - -func (ts *TimeSeriesBinary) FromHistogram(timestamp int64, h *histogram.Histogram) { - ts.Histograms.Histogram = &Histogram{ - Count: HistogramCount{IsInt: true, IntValue: h.Count}, - Sum: h.Sum, - Schema: h.Schema, - ZeroThreshold: h.ZeroThreshold, - ZeroCount: HistogramZeroCount{IsInt: true, IntValue: h.ZeroCount}, - NegativeSpans: FromPromSpan(h.NegativeSpans), - NegativeBuckets: h.NegativeBuckets, - PositiveSpans: FromPromSpan(h.PositiveSpans), - PositiveBuckets: h.PositiveBuckets, - ResetHint: int32(h.CounterResetHint), - TimestampMillisecond: timestamp, - } -} -func (ts *TimeSeriesBinary) FromFloatHistogram(timestamp int64, h *histogram.FloatHistogram) { - ts.Histograms.FloatHistogram = &FloatHistogram{ - Count: HistogramCount{IsInt: false, FloatValue: h.Count}, - Sum: h.Sum, - Schema: h.Schema, - ZeroThreshold: h.ZeroThreshold, - ZeroCount: HistogramZeroCount{IsInt: false, FloatValue: h.ZeroCount}, - NegativeSpans: FromPromSpan(h.NegativeSpans), - NegativeCounts: h.NegativeBuckets, - PositiveSpans: FromPromSpan(h.PositiveSpans), - PositiveCounts: h.PositiveBuckets, - ResetHint: int32(h.CounterResetHint), - TimestampMillisecond: timestamp, - } -} -func FromPromSpan(spans []histogram.Span) []BucketSpan { - bs := make([]BucketSpan, len(spans)) - for i, s := range spans { - bs[i].Offset = s.Offset - bs[i].Length = s.Length - } - return bs -} - -// FillLabelMapping is what does the conversion from labels.Labels to LabelNames and -// LabelValues while filling in the string map, that is later converted to []string. -func (ts *TimeSeriesBinary) FillLabelMapping(strMapToInt map[string]uint32) { - ts.LabelsNames = setSliceLength(ts.LabelsNames, len(ts.Labels)) - ts.LabelsValues = setSliceLength(ts.LabelsValues, len(ts.Labels)) - - // This is where we deduplicate the ts.Labels into uint32 values - // that map to a string in the strings slice via the index. - for i, v := range ts.Labels { - val, found := strMapToInt[v.Name] - if !found { - val = uint32(len(strMapToInt)) - strMapToInt[v.Name] = val - } - ts.LabelsNames[i] = val - - val, found = strMapToInt[v.Value] - if !found { - val = uint32(len(strMapToInt)) - strMapToInt[v.Value] = val - } - ts.LabelsValues[i] = val - } - -} - -func setSliceLength(lbls []uint32, length int) []uint32 { - if cap(lbls) <= length { - lbls = make([]uint32, length) - } else { - lbls = lbls[:length] - } - return lbls -} - -var tsBinaryPool = sync.Pool{ - New: func() any { - return &TimeSeriesBinary{} - }, -} - -func GetTimeSeriesFromPool() *TimeSeriesBinary { - OutStandingTimeSeriesBinary.Inc() - return tsBinaryPool.Get().(*TimeSeriesBinary) -} - -var OutStandingTimeSeriesBinary = atomic.Int32{} - -func PutTimeSeriesSliceIntoPool(tss []*TimeSeriesBinary) { - for i := 0; i < len(tss); i++ { - PutTimeSeriesIntoPool(tss[i]) - } - -} - -func PutTimeSeriesIntoPool(ts *TimeSeriesBinary) { - OutStandingTimeSeriesBinary.Dec() - ts.LabelsNames = ts.LabelsNames[:0] - ts.LabelsValues = ts.LabelsValues[:0] - ts.Labels = nil - ts.TS = 0 - ts.Value = 0 - ts.Hash = 0 - ts.Histograms.Histogram = nil - ts.Histograms.FloatHistogram = nil - tsBinaryPool.Put(ts) -} - -// DeserializeToSeriesGroup transforms a buffer to a SeriesGroup and converts the stringmap + indexes into actual Labels. -func DeserializeToSeriesGroup(sg *SeriesGroup, buf []byte) (*SeriesGroup, []byte, error) { - buffer, err := sg.UnmarshalMsg(buf) - if err != nil { - return sg, nil, err - } - // Need to fill in the labels. - for _, series := range sg.Series { - if cap(series.Labels) < len(series.LabelsNames) { - series.Labels = make(labels.Labels, len(series.LabelsNames)) - } else { - series.Labels = series.Labels[:len(series.LabelsNames)] - } - // Since the LabelNames/LabelValues are indexes into the Strings slice we can access it like the below. - // 1 Label corresponds to two entries, one in LabelsNames and one in LabelsValues. - for i := range series.LabelsNames { - series.Labels[i] = labels.Label{ - Name: sg.Strings[series.LabelsNames[i]], - Value: sg.Strings[series.LabelsValues[i]], - } - } - series.LabelsNames = series.LabelsNames[:0] - series.LabelsValues = series.LabelsValues[:0] - } - for _, series := range sg.Metadata { - if cap(series.Labels) < len(series.LabelsNames) { - series.Labels = make(labels.Labels, len(series.LabelsNames)) - } else { - series.Labels = series.Labels[:len(series.LabelsNames)] - } - for i := range series.LabelsNames { - series.Labels[i] = labels.Label{ - Name: sg.Strings[series.LabelsNames[i]], - Value: sg.Strings[series.LabelsValues[i]], - } - } - // Finally ensure we reset the labelnames and labelvalues. - series.LabelsNames = series.LabelsNames[:0] - series.LabelsValues = series.LabelsValues[:0] - } - - sg.Strings = sg.Strings[:0] - return sg, buffer, err -} diff --git a/internal/component/prometheus/write/queue/types/serialization_gen.go b/internal/component/prometheus/write/queue/types/serialization_gen.go deleted file mode 100644 index c31dd8d6a4..0000000000 --- a/internal/component/prometheus/write/queue/types/serialization_gen.go +++ /dev/null @@ -1,3294 +0,0 @@ -package types - -// Code generated by github.com/tinylib/msgp DO NOT EDIT. - -import ( - "github.com/tinylib/msgp/msgp" -) - -// DecodeMsg implements msgp.Decodable -func (z *BucketSpan) DecodeMsg(dc *msgp.Reader) (err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "Offset": - z.Offset, err = dc.ReadInt32() - if err != nil { - err = msgp.WrapError(err, "Offset") - return - } - case "Length": - z.Length, err = dc.ReadUint32() - if err != nil { - err = msgp.WrapError(err, "Length") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - return -} - -// EncodeMsg implements msgp.Encodable -func (z BucketSpan) EncodeMsg(en *msgp.Writer) (err error) { - // map header, size 2 - // write "Offset" - err = en.Append(0x82, 0xa6, 0x4f, 0x66, 0x66, 0x73, 0x65, 0x74) - if err != nil { - return - } - err = en.WriteInt32(z.Offset) - if err != nil { - err = msgp.WrapError(err, "Offset") - return - } - // write "Length" - err = en.Append(0xa6, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68) - if err != nil { - return - } - err = en.WriteUint32(z.Length) - if err != nil { - err = msgp.WrapError(err, "Length") - return - } - return -} - -// MarshalMsg implements msgp.Marshaler -func (z BucketSpan) MarshalMsg(b []byte) (o []byte, err error) { - o = msgp.Require(b, z.Msgsize()) - // map header, size 2 - // string "Offset" - o = append(o, 0x82, 0xa6, 0x4f, 0x66, 0x66, 0x73, 0x65, 0x74) - o = msgp.AppendInt32(o, z.Offset) - // string "Length" - o = append(o, 0xa6, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68) - o = msgp.AppendUint32(o, z.Length) - return -} - -// UnmarshalMsg implements msgp.Unmarshaler -func (z *BucketSpan) UnmarshalMsg(bts []byte) (o []byte, err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "Offset": - z.Offset, bts, err = msgp.ReadInt32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "Offset") - return - } - case "Length": - z.Length, bts, err = msgp.ReadUint32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "Length") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - o = bts - return -} - -// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message -func (z BucketSpan) Msgsize() (s int) { - s = 1 + 7 + msgp.Int32Size + 7 + msgp.Uint32Size - return -} - -// DecodeMsg implements msgp.Decodable -func (z *FloatHistogram) DecodeMsg(dc *msgp.Reader) (err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "Count": - var zb0002 uint32 - zb0002, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err, "Count") - return - } - for zb0002 > 0 { - zb0002-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err, "Count") - return - } - switch msgp.UnsafeString(field) { - case "IsInt": - z.Count.IsInt, err = dc.ReadBool() - if err != nil { - err = msgp.WrapError(err, "Count", "IsInt") - return - } - case "IntValue": - z.Count.IntValue, err = dc.ReadUint64() - if err != nil { - err = msgp.WrapError(err, "Count", "IntValue") - return - } - case "FloatValue": - z.Count.FloatValue, err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "Count", "FloatValue") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err, "Count") - return - } - } - } - case "Sum": - z.Sum, err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "Sum") - return - } - case "Schema": - z.Schema, err = dc.ReadInt32() - if err != nil { - err = msgp.WrapError(err, "Schema") - return - } - case "ZeroThreshold": - z.ZeroThreshold, err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "ZeroThreshold") - return - } - case "ZeroCount": - var zb0003 uint32 - zb0003, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err, "ZeroCount") - return - } - for zb0003 > 0 { - zb0003-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err, "ZeroCount") - return - } - switch msgp.UnsafeString(field) { - case "IsInt": - z.ZeroCount.IsInt, err = dc.ReadBool() - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "IsInt") - return - } - case "IntValue": - z.ZeroCount.IntValue, err = dc.ReadUint64() - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "IntValue") - return - } - case "FloatValue": - z.ZeroCount.FloatValue, err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "FloatValue") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err, "ZeroCount") - return - } - } - } - case "NegativeSpans": - var zb0004 uint32 - zb0004, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "NegativeSpans") - return - } - if cap(z.NegativeSpans) >= int(zb0004) { - z.NegativeSpans = (z.NegativeSpans)[:zb0004] - } else { - z.NegativeSpans = make([]BucketSpan, zb0004) - } - for za0001 := range z.NegativeSpans { - var zb0005 uint32 - zb0005, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001) - return - } - for zb0005 > 0 { - zb0005-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001) - return - } - switch msgp.UnsafeString(field) { - case "Offset": - z.NegativeSpans[za0001].Offset, err = dc.ReadInt32() - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001, "Offset") - return - } - case "Length": - z.NegativeSpans[za0001].Length, err = dc.ReadUint32() - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001, "Length") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001) - return - } - } - } - } - case "NegativeDeltas": - var zb0006 uint32 - zb0006, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "NegativeDeltas") - return - } - if cap(z.NegativeDeltas) >= int(zb0006) { - z.NegativeDeltas = (z.NegativeDeltas)[:zb0006] - } else { - z.NegativeDeltas = make([]int64, zb0006) - } - for za0002 := range z.NegativeDeltas { - z.NegativeDeltas[za0002], err = dc.ReadInt64() - if err != nil { - err = msgp.WrapError(err, "NegativeDeltas", za0002) - return - } - } - case "NegativeCounts": - var zb0007 uint32 - zb0007, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "NegativeCounts") - return - } - if cap(z.NegativeCounts) >= int(zb0007) { - z.NegativeCounts = (z.NegativeCounts)[:zb0007] - } else { - z.NegativeCounts = make([]float64, zb0007) - } - for za0003 := range z.NegativeCounts { - z.NegativeCounts[za0003], err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "NegativeCounts", za0003) - return - } - } - case "PositiveSpans": - var zb0008 uint32 - zb0008, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "PositiveSpans") - return - } - if cap(z.PositiveSpans) >= int(zb0008) { - z.PositiveSpans = (z.PositiveSpans)[:zb0008] - } else { - z.PositiveSpans = make([]BucketSpan, zb0008) - } - for za0004 := range z.PositiveSpans { - var zb0009 uint32 - zb0009, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004) - return - } - for zb0009 > 0 { - zb0009-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004) - return - } - switch msgp.UnsafeString(field) { - case "Offset": - z.PositiveSpans[za0004].Offset, err = dc.ReadInt32() - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004, "Offset") - return - } - case "Length": - z.PositiveSpans[za0004].Length, err = dc.ReadUint32() - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004, "Length") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004) - return - } - } - } - } - case "PositiveDeltas": - var zb0010 uint32 - zb0010, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "PositiveDeltas") - return - } - if cap(z.PositiveDeltas) >= int(zb0010) { - z.PositiveDeltas = (z.PositiveDeltas)[:zb0010] - } else { - z.PositiveDeltas = make([]int64, zb0010) - } - for za0005 := range z.PositiveDeltas { - z.PositiveDeltas[za0005], err = dc.ReadInt64() - if err != nil { - err = msgp.WrapError(err, "PositiveDeltas", za0005) - return - } - } - case "PositiveCounts": - var zb0011 uint32 - zb0011, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "PositiveCounts") - return - } - if cap(z.PositiveCounts) >= int(zb0011) { - z.PositiveCounts = (z.PositiveCounts)[:zb0011] - } else { - z.PositiveCounts = make([]float64, zb0011) - } - for za0006 := range z.PositiveCounts { - z.PositiveCounts[za0006], err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "PositiveCounts", za0006) - return - } - } - case "ResetHint": - z.ResetHint, err = dc.ReadInt32() - if err != nil { - err = msgp.WrapError(err, "ResetHint") - return - } - case "TimestampMillisecond": - z.TimestampMillisecond, err = dc.ReadInt64() - if err != nil { - err = msgp.WrapError(err, "TimestampMillisecond") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - return -} - -// EncodeMsg implements msgp.Encodable -func (z *FloatHistogram) EncodeMsg(en *msgp.Writer) (err error) { - // map header, size 13 - // write "Count" - err = en.Append(0x8d, 0xa5, 0x43, 0x6f, 0x75, 0x6e, 0x74) - if err != nil { - return - } - // map header, size 3 - // write "IsInt" - err = en.Append(0x83, 0xa5, 0x49, 0x73, 0x49, 0x6e, 0x74) - if err != nil { - return - } - err = en.WriteBool(z.Count.IsInt) - if err != nil { - err = msgp.WrapError(err, "Count", "IsInt") - return - } - // write "IntValue" - err = en.Append(0xa8, 0x49, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteUint64(z.Count.IntValue) - if err != nil { - err = msgp.WrapError(err, "Count", "IntValue") - return - } - // write "FloatValue" - err = en.Append(0xaa, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteFloat64(z.Count.FloatValue) - if err != nil { - err = msgp.WrapError(err, "Count", "FloatValue") - return - } - // write "Sum" - err = en.Append(0xa3, 0x53, 0x75, 0x6d) - if err != nil { - return - } - err = en.WriteFloat64(z.Sum) - if err != nil { - err = msgp.WrapError(err, "Sum") - return - } - // write "Schema" - err = en.Append(0xa6, 0x53, 0x63, 0x68, 0x65, 0x6d, 0x61) - if err != nil { - return - } - err = en.WriteInt32(z.Schema) - if err != nil { - err = msgp.WrapError(err, "Schema") - return - } - // write "ZeroThreshold" - err = en.Append(0xad, 0x5a, 0x65, 0x72, 0x6f, 0x54, 0x68, 0x72, 0x65, 0x73, 0x68, 0x6f, 0x6c, 0x64) - if err != nil { - return - } - err = en.WriteFloat64(z.ZeroThreshold) - if err != nil { - err = msgp.WrapError(err, "ZeroThreshold") - return - } - // write "ZeroCount" - err = en.Append(0xa9, 0x5a, 0x65, 0x72, 0x6f, 0x43, 0x6f, 0x75, 0x6e, 0x74) - if err != nil { - return - } - // map header, size 3 - // write "IsInt" - err = en.Append(0x83, 0xa5, 0x49, 0x73, 0x49, 0x6e, 0x74) - if err != nil { - return - } - err = en.WriteBool(z.ZeroCount.IsInt) - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "IsInt") - return - } - // write "IntValue" - err = en.Append(0xa8, 0x49, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteUint64(z.ZeroCount.IntValue) - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "IntValue") - return - } - // write "FloatValue" - err = en.Append(0xaa, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteFloat64(z.ZeroCount.FloatValue) - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "FloatValue") - return - } - // write "NegativeSpans" - err = en.Append(0xad, 0x4e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x53, 0x70, 0x61, 0x6e, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.NegativeSpans))) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans") - return - } - for za0001 := range z.NegativeSpans { - // map header, size 2 - // write "Offset" - err = en.Append(0x82, 0xa6, 0x4f, 0x66, 0x66, 0x73, 0x65, 0x74) - if err != nil { - return - } - err = en.WriteInt32(z.NegativeSpans[za0001].Offset) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001, "Offset") - return - } - // write "Length" - err = en.Append(0xa6, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68) - if err != nil { - return - } - err = en.WriteUint32(z.NegativeSpans[za0001].Length) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001, "Length") - return - } - } - // write "NegativeDeltas" - err = en.Append(0xae, 0x4e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x44, 0x65, 0x6c, 0x74, 0x61, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.NegativeDeltas))) - if err != nil { - err = msgp.WrapError(err, "NegativeDeltas") - return - } - for za0002 := range z.NegativeDeltas { - err = en.WriteInt64(z.NegativeDeltas[za0002]) - if err != nil { - err = msgp.WrapError(err, "NegativeDeltas", za0002) - return - } - } - // write "NegativeCounts" - err = en.Append(0xae, 0x4e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.NegativeCounts))) - if err != nil { - err = msgp.WrapError(err, "NegativeCounts") - return - } - for za0003 := range z.NegativeCounts { - err = en.WriteFloat64(z.NegativeCounts[za0003]) - if err != nil { - err = msgp.WrapError(err, "NegativeCounts", za0003) - return - } - } - // write "PositiveSpans" - err = en.Append(0xad, 0x50, 0x6f, 0x73, 0x69, 0x74, 0x69, 0x76, 0x65, 0x53, 0x70, 0x61, 0x6e, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.PositiveSpans))) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans") - return - } - for za0004 := range z.PositiveSpans { - // map header, size 2 - // write "Offset" - err = en.Append(0x82, 0xa6, 0x4f, 0x66, 0x66, 0x73, 0x65, 0x74) - if err != nil { - return - } - err = en.WriteInt32(z.PositiveSpans[za0004].Offset) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004, "Offset") - return - } - // write "Length" - err = en.Append(0xa6, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68) - if err != nil { - return - } - err = en.WriteUint32(z.PositiveSpans[za0004].Length) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004, "Length") - return - } - } - // write "PositiveDeltas" - err = en.Append(0xae, 0x50, 0x6f, 0x73, 0x69, 0x74, 0x69, 0x76, 0x65, 0x44, 0x65, 0x6c, 0x74, 0x61, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.PositiveDeltas))) - if err != nil { - err = msgp.WrapError(err, "PositiveDeltas") - return - } - for za0005 := range z.PositiveDeltas { - err = en.WriteInt64(z.PositiveDeltas[za0005]) - if err != nil { - err = msgp.WrapError(err, "PositiveDeltas", za0005) - return - } - } - // write "PositiveCounts" - err = en.Append(0xae, 0x50, 0x6f, 0x73, 0x69, 0x74, 0x69, 0x76, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.PositiveCounts))) - if err != nil { - err = msgp.WrapError(err, "PositiveCounts") - return - } - for za0006 := range z.PositiveCounts { - err = en.WriteFloat64(z.PositiveCounts[za0006]) - if err != nil { - err = msgp.WrapError(err, "PositiveCounts", za0006) - return - } - } - // write "ResetHint" - err = en.Append(0xa9, 0x52, 0x65, 0x73, 0x65, 0x74, 0x48, 0x69, 0x6e, 0x74) - if err != nil { - return - } - err = en.WriteInt32(z.ResetHint) - if err != nil { - err = msgp.WrapError(err, "ResetHint") - return - } - // write "TimestampMillisecond" - err = en.Append(0xb4, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x4d, 0x69, 0x6c, 0x6c, 0x69, 0x73, 0x65, 0x63, 0x6f, 0x6e, 0x64) - if err != nil { - return - } - err = en.WriteInt64(z.TimestampMillisecond) - if err != nil { - err = msgp.WrapError(err, "TimestampMillisecond") - return - } - return -} - -// MarshalMsg implements msgp.Marshaler -func (z *FloatHistogram) MarshalMsg(b []byte) (o []byte, err error) { - o = msgp.Require(b, z.Msgsize()) - // map header, size 13 - // string "Count" - o = append(o, 0x8d, 0xa5, 0x43, 0x6f, 0x75, 0x6e, 0x74) - // map header, size 3 - // string "IsInt" - o = append(o, 0x83, 0xa5, 0x49, 0x73, 0x49, 0x6e, 0x74) - o = msgp.AppendBool(o, z.Count.IsInt) - // string "IntValue" - o = append(o, 0xa8, 0x49, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendUint64(o, z.Count.IntValue) - // string "FloatValue" - o = append(o, 0xaa, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendFloat64(o, z.Count.FloatValue) - // string "Sum" - o = append(o, 0xa3, 0x53, 0x75, 0x6d) - o = msgp.AppendFloat64(o, z.Sum) - // string "Schema" - o = append(o, 0xa6, 0x53, 0x63, 0x68, 0x65, 0x6d, 0x61) - o = msgp.AppendInt32(o, z.Schema) - // string "ZeroThreshold" - o = append(o, 0xad, 0x5a, 0x65, 0x72, 0x6f, 0x54, 0x68, 0x72, 0x65, 0x73, 0x68, 0x6f, 0x6c, 0x64) - o = msgp.AppendFloat64(o, z.ZeroThreshold) - // string "ZeroCount" - o = append(o, 0xa9, 0x5a, 0x65, 0x72, 0x6f, 0x43, 0x6f, 0x75, 0x6e, 0x74) - // map header, size 3 - // string "IsInt" - o = append(o, 0x83, 0xa5, 0x49, 0x73, 0x49, 0x6e, 0x74) - o = msgp.AppendBool(o, z.ZeroCount.IsInt) - // string "IntValue" - o = append(o, 0xa8, 0x49, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendUint64(o, z.ZeroCount.IntValue) - // string "FloatValue" - o = append(o, 0xaa, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendFloat64(o, z.ZeroCount.FloatValue) - // string "NegativeSpans" - o = append(o, 0xad, 0x4e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x53, 0x70, 0x61, 0x6e, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.NegativeSpans))) - for za0001 := range z.NegativeSpans { - // map header, size 2 - // string "Offset" - o = append(o, 0x82, 0xa6, 0x4f, 0x66, 0x66, 0x73, 0x65, 0x74) - o = msgp.AppendInt32(o, z.NegativeSpans[za0001].Offset) - // string "Length" - o = append(o, 0xa6, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68) - o = msgp.AppendUint32(o, z.NegativeSpans[za0001].Length) - } - // string "NegativeDeltas" - o = append(o, 0xae, 0x4e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x44, 0x65, 0x6c, 0x74, 0x61, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.NegativeDeltas))) - for za0002 := range z.NegativeDeltas { - o = msgp.AppendInt64(o, z.NegativeDeltas[za0002]) - } - // string "NegativeCounts" - o = append(o, 0xae, 0x4e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.NegativeCounts))) - for za0003 := range z.NegativeCounts { - o = msgp.AppendFloat64(o, z.NegativeCounts[za0003]) - } - // string "PositiveSpans" - o = append(o, 0xad, 0x50, 0x6f, 0x73, 0x69, 0x74, 0x69, 0x76, 0x65, 0x53, 0x70, 0x61, 0x6e, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.PositiveSpans))) - for za0004 := range z.PositiveSpans { - // map header, size 2 - // string "Offset" - o = append(o, 0x82, 0xa6, 0x4f, 0x66, 0x66, 0x73, 0x65, 0x74) - o = msgp.AppendInt32(o, z.PositiveSpans[za0004].Offset) - // string "Length" - o = append(o, 0xa6, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68) - o = msgp.AppendUint32(o, z.PositiveSpans[za0004].Length) - } - // string "PositiveDeltas" - o = append(o, 0xae, 0x50, 0x6f, 0x73, 0x69, 0x74, 0x69, 0x76, 0x65, 0x44, 0x65, 0x6c, 0x74, 0x61, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.PositiveDeltas))) - for za0005 := range z.PositiveDeltas { - o = msgp.AppendInt64(o, z.PositiveDeltas[za0005]) - } - // string "PositiveCounts" - o = append(o, 0xae, 0x50, 0x6f, 0x73, 0x69, 0x74, 0x69, 0x76, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.PositiveCounts))) - for za0006 := range z.PositiveCounts { - o = msgp.AppendFloat64(o, z.PositiveCounts[za0006]) - } - // string "ResetHint" - o = append(o, 0xa9, 0x52, 0x65, 0x73, 0x65, 0x74, 0x48, 0x69, 0x6e, 0x74) - o = msgp.AppendInt32(o, z.ResetHint) - // string "TimestampMillisecond" - o = append(o, 0xb4, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x4d, 0x69, 0x6c, 0x6c, 0x69, 0x73, 0x65, 0x63, 0x6f, 0x6e, 0x64) - o = msgp.AppendInt64(o, z.TimestampMillisecond) - return -} - -// UnmarshalMsg implements msgp.Unmarshaler -func (z *FloatHistogram) UnmarshalMsg(bts []byte) (o []byte, err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "Count": - var zb0002 uint32 - zb0002, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "Count") - return - } - for zb0002 > 0 { - zb0002-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err, "Count") - return - } - switch msgp.UnsafeString(field) { - case "IsInt": - z.Count.IsInt, bts, err = msgp.ReadBoolBytes(bts) - if err != nil { - err = msgp.WrapError(err, "Count", "IsInt") - return - } - case "IntValue": - z.Count.IntValue, bts, err = msgp.ReadUint64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "Count", "IntValue") - return - } - case "FloatValue": - z.Count.FloatValue, bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "Count", "FloatValue") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err, "Count") - return - } - } - } - case "Sum": - z.Sum, bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "Sum") - return - } - case "Schema": - z.Schema, bts, err = msgp.ReadInt32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "Schema") - return - } - case "ZeroThreshold": - z.ZeroThreshold, bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroThreshold") - return - } - case "ZeroCount": - var zb0003 uint32 - zb0003, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroCount") - return - } - for zb0003 > 0 { - zb0003-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroCount") - return - } - switch msgp.UnsafeString(field) { - case "IsInt": - z.ZeroCount.IsInt, bts, err = msgp.ReadBoolBytes(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "IsInt") - return - } - case "IntValue": - z.ZeroCount.IntValue, bts, err = msgp.ReadUint64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "IntValue") - return - } - case "FloatValue": - z.ZeroCount.FloatValue, bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "FloatValue") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroCount") - return - } - } - } - case "NegativeSpans": - var zb0004 uint32 - zb0004, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans") - return - } - if cap(z.NegativeSpans) >= int(zb0004) { - z.NegativeSpans = (z.NegativeSpans)[:zb0004] - } else { - z.NegativeSpans = make([]BucketSpan, zb0004) - } - for za0001 := range z.NegativeSpans { - var zb0005 uint32 - zb0005, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001) - return - } - for zb0005 > 0 { - zb0005-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001) - return - } - switch msgp.UnsafeString(field) { - case "Offset": - z.NegativeSpans[za0001].Offset, bts, err = msgp.ReadInt32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001, "Offset") - return - } - case "Length": - z.NegativeSpans[za0001].Length, bts, err = msgp.ReadUint32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001, "Length") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001) - return - } - } - } - } - case "NegativeDeltas": - var zb0006 uint32 - zb0006, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeDeltas") - return - } - if cap(z.NegativeDeltas) >= int(zb0006) { - z.NegativeDeltas = (z.NegativeDeltas)[:zb0006] - } else { - z.NegativeDeltas = make([]int64, zb0006) - } - for za0002 := range z.NegativeDeltas { - z.NegativeDeltas[za0002], bts, err = msgp.ReadInt64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeDeltas", za0002) - return - } - } - case "NegativeCounts": - var zb0007 uint32 - zb0007, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeCounts") - return - } - if cap(z.NegativeCounts) >= int(zb0007) { - z.NegativeCounts = (z.NegativeCounts)[:zb0007] - } else { - z.NegativeCounts = make([]float64, zb0007) - } - for za0003 := range z.NegativeCounts { - z.NegativeCounts[za0003], bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeCounts", za0003) - return - } - } - case "PositiveSpans": - var zb0008 uint32 - zb0008, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans") - return - } - if cap(z.PositiveSpans) >= int(zb0008) { - z.PositiveSpans = (z.PositiveSpans)[:zb0008] - } else { - z.PositiveSpans = make([]BucketSpan, zb0008) - } - for za0004 := range z.PositiveSpans { - var zb0009 uint32 - zb0009, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004) - return - } - for zb0009 > 0 { - zb0009-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004) - return - } - switch msgp.UnsafeString(field) { - case "Offset": - z.PositiveSpans[za0004].Offset, bts, err = msgp.ReadInt32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004, "Offset") - return - } - case "Length": - z.PositiveSpans[za0004].Length, bts, err = msgp.ReadUint32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004, "Length") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004) - return - } - } - } - } - case "PositiveDeltas": - var zb0010 uint32 - zb0010, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveDeltas") - return - } - if cap(z.PositiveDeltas) >= int(zb0010) { - z.PositiveDeltas = (z.PositiveDeltas)[:zb0010] - } else { - z.PositiveDeltas = make([]int64, zb0010) - } - for za0005 := range z.PositiveDeltas { - z.PositiveDeltas[za0005], bts, err = msgp.ReadInt64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveDeltas", za0005) - return - } - } - case "PositiveCounts": - var zb0011 uint32 - zb0011, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveCounts") - return - } - if cap(z.PositiveCounts) >= int(zb0011) { - z.PositiveCounts = (z.PositiveCounts)[:zb0011] - } else { - z.PositiveCounts = make([]float64, zb0011) - } - for za0006 := range z.PositiveCounts { - z.PositiveCounts[za0006], bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveCounts", za0006) - return - } - } - case "ResetHint": - z.ResetHint, bts, err = msgp.ReadInt32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "ResetHint") - return - } - case "TimestampMillisecond": - z.TimestampMillisecond, bts, err = msgp.ReadInt64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "TimestampMillisecond") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - o = bts - return -} - -// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message -func (z *FloatHistogram) Msgsize() (s int) { - s = 1 + 6 + 1 + 6 + msgp.BoolSize + 9 + msgp.Uint64Size + 11 + msgp.Float64Size + 4 + msgp.Float64Size + 7 + msgp.Int32Size + 14 + msgp.Float64Size + 10 + 1 + 6 + msgp.BoolSize + 9 + msgp.Uint64Size + 11 + msgp.Float64Size + 14 + msgp.ArrayHeaderSize + (len(z.NegativeSpans) * (15 + msgp.Int32Size + msgp.Uint32Size)) + 15 + msgp.ArrayHeaderSize + (len(z.NegativeDeltas) * (msgp.Int64Size)) + 15 + msgp.ArrayHeaderSize + (len(z.NegativeCounts) * (msgp.Float64Size)) + 14 + msgp.ArrayHeaderSize + (len(z.PositiveSpans) * (15 + msgp.Int32Size + msgp.Uint32Size)) + 15 + msgp.ArrayHeaderSize + (len(z.PositiveDeltas) * (msgp.Int64Size)) + 15 + msgp.ArrayHeaderSize + (len(z.PositiveCounts) * (msgp.Float64Size)) + 10 + msgp.Int32Size + 21 + msgp.Int64Size - return -} - -// DecodeMsg implements msgp.Decodable -func (z *Histogram) DecodeMsg(dc *msgp.Reader) (err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "Count": - var zb0002 uint32 - zb0002, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err, "Count") - return - } - for zb0002 > 0 { - zb0002-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err, "Count") - return - } - switch msgp.UnsafeString(field) { - case "IsInt": - z.Count.IsInt, err = dc.ReadBool() - if err != nil { - err = msgp.WrapError(err, "Count", "IsInt") - return - } - case "IntValue": - z.Count.IntValue, err = dc.ReadUint64() - if err != nil { - err = msgp.WrapError(err, "Count", "IntValue") - return - } - case "FloatValue": - z.Count.FloatValue, err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "Count", "FloatValue") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err, "Count") - return - } - } - } - case "Sum": - z.Sum, err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "Sum") - return - } - case "Schema": - z.Schema, err = dc.ReadInt32() - if err != nil { - err = msgp.WrapError(err, "Schema") - return - } - case "ZeroThreshold": - z.ZeroThreshold, err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "ZeroThreshold") - return - } - case "ZeroCount": - var zb0003 uint32 - zb0003, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err, "ZeroCount") - return - } - for zb0003 > 0 { - zb0003-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err, "ZeroCount") - return - } - switch msgp.UnsafeString(field) { - case "IsInt": - z.ZeroCount.IsInt, err = dc.ReadBool() - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "IsInt") - return - } - case "IntValue": - z.ZeroCount.IntValue, err = dc.ReadUint64() - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "IntValue") - return - } - case "FloatValue": - z.ZeroCount.FloatValue, err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "FloatValue") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err, "ZeroCount") - return - } - } - } - case "NegativeSpans": - var zb0004 uint32 - zb0004, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "NegativeSpans") - return - } - if cap(z.NegativeSpans) >= int(zb0004) { - z.NegativeSpans = (z.NegativeSpans)[:zb0004] - } else { - z.NegativeSpans = make([]BucketSpan, zb0004) - } - for za0001 := range z.NegativeSpans { - var zb0005 uint32 - zb0005, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001) - return - } - for zb0005 > 0 { - zb0005-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001) - return - } - switch msgp.UnsafeString(field) { - case "Offset": - z.NegativeSpans[za0001].Offset, err = dc.ReadInt32() - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001, "Offset") - return - } - case "Length": - z.NegativeSpans[za0001].Length, err = dc.ReadUint32() - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001, "Length") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001) - return - } - } - } - } - case "NegativeBuckets": - var zb0006 uint32 - zb0006, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "NegativeBuckets") - return - } - if cap(z.NegativeBuckets) >= int(zb0006) { - z.NegativeBuckets = (z.NegativeBuckets)[:zb0006] - } else { - z.NegativeBuckets = make([]int64, zb0006) - } - for za0002 := range z.NegativeBuckets { - z.NegativeBuckets[za0002], err = dc.ReadInt64() - if err != nil { - err = msgp.WrapError(err, "NegativeBuckets", za0002) - return - } - } - case "NegativeCounts": - var zb0007 uint32 - zb0007, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "NegativeCounts") - return - } - if cap(z.NegativeCounts) >= int(zb0007) { - z.NegativeCounts = (z.NegativeCounts)[:zb0007] - } else { - z.NegativeCounts = make([]float64, zb0007) - } - for za0003 := range z.NegativeCounts { - z.NegativeCounts[za0003], err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "NegativeCounts", za0003) - return - } - } - case "PositiveSpans": - var zb0008 uint32 - zb0008, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "PositiveSpans") - return - } - if cap(z.PositiveSpans) >= int(zb0008) { - z.PositiveSpans = (z.PositiveSpans)[:zb0008] - } else { - z.PositiveSpans = make([]BucketSpan, zb0008) - } - for za0004 := range z.PositiveSpans { - var zb0009 uint32 - zb0009, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004) - return - } - for zb0009 > 0 { - zb0009-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004) - return - } - switch msgp.UnsafeString(field) { - case "Offset": - z.PositiveSpans[za0004].Offset, err = dc.ReadInt32() - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004, "Offset") - return - } - case "Length": - z.PositiveSpans[za0004].Length, err = dc.ReadUint32() - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004, "Length") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004) - return - } - } - } - } - case "PositiveBuckets": - var zb0010 uint32 - zb0010, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "PositiveBuckets") - return - } - if cap(z.PositiveBuckets) >= int(zb0010) { - z.PositiveBuckets = (z.PositiveBuckets)[:zb0010] - } else { - z.PositiveBuckets = make([]int64, zb0010) - } - for za0005 := range z.PositiveBuckets { - z.PositiveBuckets[za0005], err = dc.ReadInt64() - if err != nil { - err = msgp.WrapError(err, "PositiveBuckets", za0005) - return - } - } - case "PositiveCounts": - var zb0011 uint32 - zb0011, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "PositiveCounts") - return - } - if cap(z.PositiveCounts) >= int(zb0011) { - z.PositiveCounts = (z.PositiveCounts)[:zb0011] - } else { - z.PositiveCounts = make([]float64, zb0011) - } - for za0006 := range z.PositiveCounts { - z.PositiveCounts[za0006], err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "PositiveCounts", za0006) - return - } - } - case "ResetHint": - z.ResetHint, err = dc.ReadInt32() - if err != nil { - err = msgp.WrapError(err, "ResetHint") - return - } - case "TimestampMillisecond": - z.TimestampMillisecond, err = dc.ReadInt64() - if err != nil { - err = msgp.WrapError(err, "TimestampMillisecond") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - return -} - -// EncodeMsg implements msgp.Encodable -func (z *Histogram) EncodeMsg(en *msgp.Writer) (err error) { - // map header, size 13 - // write "Count" - err = en.Append(0x8d, 0xa5, 0x43, 0x6f, 0x75, 0x6e, 0x74) - if err != nil { - return - } - // map header, size 3 - // write "IsInt" - err = en.Append(0x83, 0xa5, 0x49, 0x73, 0x49, 0x6e, 0x74) - if err != nil { - return - } - err = en.WriteBool(z.Count.IsInt) - if err != nil { - err = msgp.WrapError(err, "Count", "IsInt") - return - } - // write "IntValue" - err = en.Append(0xa8, 0x49, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteUint64(z.Count.IntValue) - if err != nil { - err = msgp.WrapError(err, "Count", "IntValue") - return - } - // write "FloatValue" - err = en.Append(0xaa, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteFloat64(z.Count.FloatValue) - if err != nil { - err = msgp.WrapError(err, "Count", "FloatValue") - return - } - // write "Sum" - err = en.Append(0xa3, 0x53, 0x75, 0x6d) - if err != nil { - return - } - err = en.WriteFloat64(z.Sum) - if err != nil { - err = msgp.WrapError(err, "Sum") - return - } - // write "Schema" - err = en.Append(0xa6, 0x53, 0x63, 0x68, 0x65, 0x6d, 0x61) - if err != nil { - return - } - err = en.WriteInt32(z.Schema) - if err != nil { - err = msgp.WrapError(err, "Schema") - return - } - // write "ZeroThreshold" - err = en.Append(0xad, 0x5a, 0x65, 0x72, 0x6f, 0x54, 0x68, 0x72, 0x65, 0x73, 0x68, 0x6f, 0x6c, 0x64) - if err != nil { - return - } - err = en.WriteFloat64(z.ZeroThreshold) - if err != nil { - err = msgp.WrapError(err, "ZeroThreshold") - return - } - // write "ZeroCount" - err = en.Append(0xa9, 0x5a, 0x65, 0x72, 0x6f, 0x43, 0x6f, 0x75, 0x6e, 0x74) - if err != nil { - return - } - // map header, size 3 - // write "IsInt" - err = en.Append(0x83, 0xa5, 0x49, 0x73, 0x49, 0x6e, 0x74) - if err != nil { - return - } - err = en.WriteBool(z.ZeroCount.IsInt) - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "IsInt") - return - } - // write "IntValue" - err = en.Append(0xa8, 0x49, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteUint64(z.ZeroCount.IntValue) - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "IntValue") - return - } - // write "FloatValue" - err = en.Append(0xaa, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteFloat64(z.ZeroCount.FloatValue) - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "FloatValue") - return - } - // write "NegativeSpans" - err = en.Append(0xad, 0x4e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x53, 0x70, 0x61, 0x6e, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.NegativeSpans))) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans") - return - } - for za0001 := range z.NegativeSpans { - // map header, size 2 - // write "Offset" - err = en.Append(0x82, 0xa6, 0x4f, 0x66, 0x66, 0x73, 0x65, 0x74) - if err != nil { - return - } - err = en.WriteInt32(z.NegativeSpans[za0001].Offset) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001, "Offset") - return - } - // write "Length" - err = en.Append(0xa6, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68) - if err != nil { - return - } - err = en.WriteUint32(z.NegativeSpans[za0001].Length) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001, "Length") - return - } - } - // write "NegativeBuckets" - err = en.Append(0xaf, 0x4e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x42, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.NegativeBuckets))) - if err != nil { - err = msgp.WrapError(err, "NegativeBuckets") - return - } - for za0002 := range z.NegativeBuckets { - err = en.WriteInt64(z.NegativeBuckets[za0002]) - if err != nil { - err = msgp.WrapError(err, "NegativeBuckets", za0002) - return - } - } - // write "NegativeCounts" - err = en.Append(0xae, 0x4e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.NegativeCounts))) - if err != nil { - err = msgp.WrapError(err, "NegativeCounts") - return - } - for za0003 := range z.NegativeCounts { - err = en.WriteFloat64(z.NegativeCounts[za0003]) - if err != nil { - err = msgp.WrapError(err, "NegativeCounts", za0003) - return - } - } - // write "PositiveSpans" - err = en.Append(0xad, 0x50, 0x6f, 0x73, 0x69, 0x74, 0x69, 0x76, 0x65, 0x53, 0x70, 0x61, 0x6e, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.PositiveSpans))) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans") - return - } - for za0004 := range z.PositiveSpans { - // map header, size 2 - // write "Offset" - err = en.Append(0x82, 0xa6, 0x4f, 0x66, 0x66, 0x73, 0x65, 0x74) - if err != nil { - return - } - err = en.WriteInt32(z.PositiveSpans[za0004].Offset) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004, "Offset") - return - } - // write "Length" - err = en.Append(0xa6, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68) - if err != nil { - return - } - err = en.WriteUint32(z.PositiveSpans[za0004].Length) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004, "Length") - return - } - } - // write "PositiveBuckets" - err = en.Append(0xaf, 0x50, 0x6f, 0x73, 0x69, 0x74, 0x69, 0x76, 0x65, 0x42, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.PositiveBuckets))) - if err != nil { - err = msgp.WrapError(err, "PositiveBuckets") - return - } - for za0005 := range z.PositiveBuckets { - err = en.WriteInt64(z.PositiveBuckets[za0005]) - if err != nil { - err = msgp.WrapError(err, "PositiveBuckets", za0005) - return - } - } - // write "PositiveCounts" - err = en.Append(0xae, 0x50, 0x6f, 0x73, 0x69, 0x74, 0x69, 0x76, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.PositiveCounts))) - if err != nil { - err = msgp.WrapError(err, "PositiveCounts") - return - } - for za0006 := range z.PositiveCounts { - err = en.WriteFloat64(z.PositiveCounts[za0006]) - if err != nil { - err = msgp.WrapError(err, "PositiveCounts", za0006) - return - } - } - // write "ResetHint" - err = en.Append(0xa9, 0x52, 0x65, 0x73, 0x65, 0x74, 0x48, 0x69, 0x6e, 0x74) - if err != nil { - return - } - err = en.WriteInt32(z.ResetHint) - if err != nil { - err = msgp.WrapError(err, "ResetHint") - return - } - // write "TimestampMillisecond" - err = en.Append(0xb4, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x4d, 0x69, 0x6c, 0x6c, 0x69, 0x73, 0x65, 0x63, 0x6f, 0x6e, 0x64) - if err != nil { - return - } - err = en.WriteInt64(z.TimestampMillisecond) - if err != nil { - err = msgp.WrapError(err, "TimestampMillisecond") - return - } - return -} - -// MarshalMsg implements msgp.Marshaler -func (z *Histogram) MarshalMsg(b []byte) (o []byte, err error) { - o = msgp.Require(b, z.Msgsize()) - // map header, size 13 - // string "Count" - o = append(o, 0x8d, 0xa5, 0x43, 0x6f, 0x75, 0x6e, 0x74) - // map header, size 3 - // string "IsInt" - o = append(o, 0x83, 0xa5, 0x49, 0x73, 0x49, 0x6e, 0x74) - o = msgp.AppendBool(o, z.Count.IsInt) - // string "IntValue" - o = append(o, 0xa8, 0x49, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendUint64(o, z.Count.IntValue) - // string "FloatValue" - o = append(o, 0xaa, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendFloat64(o, z.Count.FloatValue) - // string "Sum" - o = append(o, 0xa3, 0x53, 0x75, 0x6d) - o = msgp.AppendFloat64(o, z.Sum) - // string "Schema" - o = append(o, 0xa6, 0x53, 0x63, 0x68, 0x65, 0x6d, 0x61) - o = msgp.AppendInt32(o, z.Schema) - // string "ZeroThreshold" - o = append(o, 0xad, 0x5a, 0x65, 0x72, 0x6f, 0x54, 0x68, 0x72, 0x65, 0x73, 0x68, 0x6f, 0x6c, 0x64) - o = msgp.AppendFloat64(o, z.ZeroThreshold) - // string "ZeroCount" - o = append(o, 0xa9, 0x5a, 0x65, 0x72, 0x6f, 0x43, 0x6f, 0x75, 0x6e, 0x74) - // map header, size 3 - // string "IsInt" - o = append(o, 0x83, 0xa5, 0x49, 0x73, 0x49, 0x6e, 0x74) - o = msgp.AppendBool(o, z.ZeroCount.IsInt) - // string "IntValue" - o = append(o, 0xa8, 0x49, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendUint64(o, z.ZeroCount.IntValue) - // string "FloatValue" - o = append(o, 0xaa, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendFloat64(o, z.ZeroCount.FloatValue) - // string "NegativeSpans" - o = append(o, 0xad, 0x4e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x53, 0x70, 0x61, 0x6e, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.NegativeSpans))) - for za0001 := range z.NegativeSpans { - // map header, size 2 - // string "Offset" - o = append(o, 0x82, 0xa6, 0x4f, 0x66, 0x66, 0x73, 0x65, 0x74) - o = msgp.AppendInt32(o, z.NegativeSpans[za0001].Offset) - // string "Length" - o = append(o, 0xa6, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68) - o = msgp.AppendUint32(o, z.NegativeSpans[za0001].Length) - } - // string "NegativeBuckets" - o = append(o, 0xaf, 0x4e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x42, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.NegativeBuckets))) - for za0002 := range z.NegativeBuckets { - o = msgp.AppendInt64(o, z.NegativeBuckets[za0002]) - } - // string "NegativeCounts" - o = append(o, 0xae, 0x4e, 0x65, 0x67, 0x61, 0x74, 0x69, 0x76, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.NegativeCounts))) - for za0003 := range z.NegativeCounts { - o = msgp.AppendFloat64(o, z.NegativeCounts[za0003]) - } - // string "PositiveSpans" - o = append(o, 0xad, 0x50, 0x6f, 0x73, 0x69, 0x74, 0x69, 0x76, 0x65, 0x53, 0x70, 0x61, 0x6e, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.PositiveSpans))) - for za0004 := range z.PositiveSpans { - // map header, size 2 - // string "Offset" - o = append(o, 0x82, 0xa6, 0x4f, 0x66, 0x66, 0x73, 0x65, 0x74) - o = msgp.AppendInt32(o, z.PositiveSpans[za0004].Offset) - // string "Length" - o = append(o, 0xa6, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68) - o = msgp.AppendUint32(o, z.PositiveSpans[za0004].Length) - } - // string "PositiveBuckets" - o = append(o, 0xaf, 0x50, 0x6f, 0x73, 0x69, 0x74, 0x69, 0x76, 0x65, 0x42, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.PositiveBuckets))) - for za0005 := range z.PositiveBuckets { - o = msgp.AppendInt64(o, z.PositiveBuckets[za0005]) - } - // string "PositiveCounts" - o = append(o, 0xae, 0x50, 0x6f, 0x73, 0x69, 0x74, 0x69, 0x76, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.PositiveCounts))) - for za0006 := range z.PositiveCounts { - o = msgp.AppendFloat64(o, z.PositiveCounts[za0006]) - } - // string "ResetHint" - o = append(o, 0xa9, 0x52, 0x65, 0x73, 0x65, 0x74, 0x48, 0x69, 0x6e, 0x74) - o = msgp.AppendInt32(o, z.ResetHint) - // string "TimestampMillisecond" - o = append(o, 0xb4, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x4d, 0x69, 0x6c, 0x6c, 0x69, 0x73, 0x65, 0x63, 0x6f, 0x6e, 0x64) - o = msgp.AppendInt64(o, z.TimestampMillisecond) - return -} - -// UnmarshalMsg implements msgp.Unmarshaler -func (z *Histogram) UnmarshalMsg(bts []byte) (o []byte, err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "Count": - var zb0002 uint32 - zb0002, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "Count") - return - } - for zb0002 > 0 { - zb0002-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err, "Count") - return - } - switch msgp.UnsafeString(field) { - case "IsInt": - z.Count.IsInt, bts, err = msgp.ReadBoolBytes(bts) - if err != nil { - err = msgp.WrapError(err, "Count", "IsInt") - return - } - case "IntValue": - z.Count.IntValue, bts, err = msgp.ReadUint64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "Count", "IntValue") - return - } - case "FloatValue": - z.Count.FloatValue, bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "Count", "FloatValue") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err, "Count") - return - } - } - } - case "Sum": - z.Sum, bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "Sum") - return - } - case "Schema": - z.Schema, bts, err = msgp.ReadInt32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "Schema") - return - } - case "ZeroThreshold": - z.ZeroThreshold, bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroThreshold") - return - } - case "ZeroCount": - var zb0003 uint32 - zb0003, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroCount") - return - } - for zb0003 > 0 { - zb0003-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroCount") - return - } - switch msgp.UnsafeString(field) { - case "IsInt": - z.ZeroCount.IsInt, bts, err = msgp.ReadBoolBytes(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "IsInt") - return - } - case "IntValue": - z.ZeroCount.IntValue, bts, err = msgp.ReadUint64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "IntValue") - return - } - case "FloatValue": - z.ZeroCount.FloatValue, bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroCount", "FloatValue") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err, "ZeroCount") - return - } - } - } - case "NegativeSpans": - var zb0004 uint32 - zb0004, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans") - return - } - if cap(z.NegativeSpans) >= int(zb0004) { - z.NegativeSpans = (z.NegativeSpans)[:zb0004] - } else { - z.NegativeSpans = make([]BucketSpan, zb0004) - } - for za0001 := range z.NegativeSpans { - var zb0005 uint32 - zb0005, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001) - return - } - for zb0005 > 0 { - zb0005-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001) - return - } - switch msgp.UnsafeString(field) { - case "Offset": - z.NegativeSpans[za0001].Offset, bts, err = msgp.ReadInt32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001, "Offset") - return - } - case "Length": - z.NegativeSpans[za0001].Length, bts, err = msgp.ReadUint32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001, "Length") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeSpans", za0001) - return - } - } - } - } - case "NegativeBuckets": - var zb0006 uint32 - zb0006, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeBuckets") - return - } - if cap(z.NegativeBuckets) >= int(zb0006) { - z.NegativeBuckets = (z.NegativeBuckets)[:zb0006] - } else { - z.NegativeBuckets = make([]int64, zb0006) - } - for za0002 := range z.NegativeBuckets { - z.NegativeBuckets[za0002], bts, err = msgp.ReadInt64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeBuckets", za0002) - return - } - } - case "NegativeCounts": - var zb0007 uint32 - zb0007, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeCounts") - return - } - if cap(z.NegativeCounts) >= int(zb0007) { - z.NegativeCounts = (z.NegativeCounts)[:zb0007] - } else { - z.NegativeCounts = make([]float64, zb0007) - } - for za0003 := range z.NegativeCounts { - z.NegativeCounts[za0003], bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "NegativeCounts", za0003) - return - } - } - case "PositiveSpans": - var zb0008 uint32 - zb0008, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans") - return - } - if cap(z.PositiveSpans) >= int(zb0008) { - z.PositiveSpans = (z.PositiveSpans)[:zb0008] - } else { - z.PositiveSpans = make([]BucketSpan, zb0008) - } - for za0004 := range z.PositiveSpans { - var zb0009 uint32 - zb0009, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004) - return - } - for zb0009 > 0 { - zb0009-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004) - return - } - switch msgp.UnsafeString(field) { - case "Offset": - z.PositiveSpans[za0004].Offset, bts, err = msgp.ReadInt32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004, "Offset") - return - } - case "Length": - z.PositiveSpans[za0004].Length, bts, err = msgp.ReadUint32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004, "Length") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveSpans", za0004) - return - } - } - } - } - case "PositiveBuckets": - var zb0010 uint32 - zb0010, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveBuckets") - return - } - if cap(z.PositiveBuckets) >= int(zb0010) { - z.PositiveBuckets = (z.PositiveBuckets)[:zb0010] - } else { - z.PositiveBuckets = make([]int64, zb0010) - } - for za0005 := range z.PositiveBuckets { - z.PositiveBuckets[za0005], bts, err = msgp.ReadInt64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveBuckets", za0005) - return - } - } - case "PositiveCounts": - var zb0011 uint32 - zb0011, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveCounts") - return - } - if cap(z.PositiveCounts) >= int(zb0011) { - z.PositiveCounts = (z.PositiveCounts)[:zb0011] - } else { - z.PositiveCounts = make([]float64, zb0011) - } - for za0006 := range z.PositiveCounts { - z.PositiveCounts[za0006], bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "PositiveCounts", za0006) - return - } - } - case "ResetHint": - z.ResetHint, bts, err = msgp.ReadInt32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "ResetHint") - return - } - case "TimestampMillisecond": - z.TimestampMillisecond, bts, err = msgp.ReadInt64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "TimestampMillisecond") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - o = bts - return -} - -// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message -func (z *Histogram) Msgsize() (s int) { - s = 1 + 6 + 1 + 6 + msgp.BoolSize + 9 + msgp.Uint64Size + 11 + msgp.Float64Size + 4 + msgp.Float64Size + 7 + msgp.Int32Size + 14 + msgp.Float64Size + 10 + 1 + 6 + msgp.BoolSize + 9 + msgp.Uint64Size + 11 + msgp.Float64Size + 14 + msgp.ArrayHeaderSize + (len(z.NegativeSpans) * (15 + msgp.Int32Size + msgp.Uint32Size)) + 16 + msgp.ArrayHeaderSize + (len(z.NegativeBuckets) * (msgp.Int64Size)) + 15 + msgp.ArrayHeaderSize + (len(z.NegativeCounts) * (msgp.Float64Size)) + 14 + msgp.ArrayHeaderSize + (len(z.PositiveSpans) * (15 + msgp.Int32Size + msgp.Uint32Size)) + 16 + msgp.ArrayHeaderSize + (len(z.PositiveBuckets) * (msgp.Int64Size)) + 15 + msgp.ArrayHeaderSize + (len(z.PositiveCounts) * (msgp.Float64Size)) + 10 + msgp.Int32Size + 21 + msgp.Int64Size - return -} - -// DecodeMsg implements msgp.Decodable -func (z *HistogramCount) DecodeMsg(dc *msgp.Reader) (err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "IsInt": - z.IsInt, err = dc.ReadBool() - if err != nil { - err = msgp.WrapError(err, "IsInt") - return - } - case "IntValue": - z.IntValue, err = dc.ReadUint64() - if err != nil { - err = msgp.WrapError(err, "IntValue") - return - } - case "FloatValue": - z.FloatValue, err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "FloatValue") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - return -} - -// EncodeMsg implements msgp.Encodable -func (z HistogramCount) EncodeMsg(en *msgp.Writer) (err error) { - // map header, size 3 - // write "IsInt" - err = en.Append(0x83, 0xa5, 0x49, 0x73, 0x49, 0x6e, 0x74) - if err != nil { - return - } - err = en.WriteBool(z.IsInt) - if err != nil { - err = msgp.WrapError(err, "IsInt") - return - } - // write "IntValue" - err = en.Append(0xa8, 0x49, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteUint64(z.IntValue) - if err != nil { - err = msgp.WrapError(err, "IntValue") - return - } - // write "FloatValue" - err = en.Append(0xaa, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteFloat64(z.FloatValue) - if err != nil { - err = msgp.WrapError(err, "FloatValue") - return - } - return -} - -// MarshalMsg implements msgp.Marshaler -func (z HistogramCount) MarshalMsg(b []byte) (o []byte, err error) { - o = msgp.Require(b, z.Msgsize()) - // map header, size 3 - // string "IsInt" - o = append(o, 0x83, 0xa5, 0x49, 0x73, 0x49, 0x6e, 0x74) - o = msgp.AppendBool(o, z.IsInt) - // string "IntValue" - o = append(o, 0xa8, 0x49, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendUint64(o, z.IntValue) - // string "FloatValue" - o = append(o, 0xaa, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendFloat64(o, z.FloatValue) - return -} - -// UnmarshalMsg implements msgp.Unmarshaler -func (z *HistogramCount) UnmarshalMsg(bts []byte) (o []byte, err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "IsInt": - z.IsInt, bts, err = msgp.ReadBoolBytes(bts) - if err != nil { - err = msgp.WrapError(err, "IsInt") - return - } - case "IntValue": - z.IntValue, bts, err = msgp.ReadUint64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "IntValue") - return - } - case "FloatValue": - z.FloatValue, bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "FloatValue") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - o = bts - return -} - -// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message -func (z HistogramCount) Msgsize() (s int) { - s = 1 + 6 + msgp.BoolSize + 9 + msgp.Uint64Size + 11 + msgp.Float64Size - return -} - -// DecodeMsg implements msgp.Decodable -func (z *HistogramZeroCount) DecodeMsg(dc *msgp.Reader) (err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "IsInt": - z.IsInt, err = dc.ReadBool() - if err != nil { - err = msgp.WrapError(err, "IsInt") - return - } - case "IntValue": - z.IntValue, err = dc.ReadUint64() - if err != nil { - err = msgp.WrapError(err, "IntValue") - return - } - case "FloatValue": - z.FloatValue, err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "FloatValue") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - return -} - -// EncodeMsg implements msgp.Encodable -func (z HistogramZeroCount) EncodeMsg(en *msgp.Writer) (err error) { - // map header, size 3 - // write "IsInt" - err = en.Append(0x83, 0xa5, 0x49, 0x73, 0x49, 0x6e, 0x74) - if err != nil { - return - } - err = en.WriteBool(z.IsInt) - if err != nil { - err = msgp.WrapError(err, "IsInt") - return - } - // write "IntValue" - err = en.Append(0xa8, 0x49, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteUint64(z.IntValue) - if err != nil { - err = msgp.WrapError(err, "IntValue") - return - } - // write "FloatValue" - err = en.Append(0xaa, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteFloat64(z.FloatValue) - if err != nil { - err = msgp.WrapError(err, "FloatValue") - return - } - return -} - -// MarshalMsg implements msgp.Marshaler -func (z HistogramZeroCount) MarshalMsg(b []byte) (o []byte, err error) { - o = msgp.Require(b, z.Msgsize()) - // map header, size 3 - // string "IsInt" - o = append(o, 0x83, 0xa5, 0x49, 0x73, 0x49, 0x6e, 0x74) - o = msgp.AppendBool(o, z.IsInt) - // string "IntValue" - o = append(o, 0xa8, 0x49, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendUint64(o, z.IntValue) - // string "FloatValue" - o = append(o, 0xaa, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendFloat64(o, z.FloatValue) - return -} - -// UnmarshalMsg implements msgp.Unmarshaler -func (z *HistogramZeroCount) UnmarshalMsg(bts []byte) (o []byte, err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "IsInt": - z.IsInt, bts, err = msgp.ReadBoolBytes(bts) - if err != nil { - err = msgp.WrapError(err, "IsInt") - return - } - case "IntValue": - z.IntValue, bts, err = msgp.ReadUint64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "IntValue") - return - } - case "FloatValue": - z.FloatValue, bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "FloatValue") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - o = bts - return -} - -// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message -func (z HistogramZeroCount) Msgsize() (s int) { - s = 1 + 6 + msgp.BoolSize + 9 + msgp.Uint64Size + 11 + msgp.Float64Size - return -} - -// DecodeMsg implements msgp.Decodable -func (z *Histograms) DecodeMsg(dc *msgp.Reader) (err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "Histogram": - if dc.IsNil() { - err = dc.ReadNil() - if err != nil { - err = msgp.WrapError(err, "Histogram") - return - } - z.Histogram = nil - } else { - if z.Histogram == nil { - z.Histogram = new(Histogram) - } - err = z.Histogram.DecodeMsg(dc) - if err != nil { - err = msgp.WrapError(err, "Histogram") - return - } - } - case "FloatHistogram": - if dc.IsNil() { - err = dc.ReadNil() - if err != nil { - err = msgp.WrapError(err, "FloatHistogram") - return - } - z.FloatHistogram = nil - } else { - if z.FloatHistogram == nil { - z.FloatHistogram = new(FloatHistogram) - } - err = z.FloatHistogram.DecodeMsg(dc) - if err != nil { - err = msgp.WrapError(err, "FloatHistogram") - return - } - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - return -} - -// EncodeMsg implements msgp.Encodable -func (z *Histograms) EncodeMsg(en *msgp.Writer) (err error) { - // map header, size 2 - // write "Histogram" - err = en.Append(0x82, 0xa9, 0x48, 0x69, 0x73, 0x74, 0x6f, 0x67, 0x72, 0x61, 0x6d) - if err != nil { - return - } - if z.Histogram == nil { - err = en.WriteNil() - if err != nil { - return - } - } else { - err = z.Histogram.EncodeMsg(en) - if err != nil { - err = msgp.WrapError(err, "Histogram") - return - } - } - // write "FloatHistogram" - err = en.Append(0xae, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x48, 0x69, 0x73, 0x74, 0x6f, 0x67, 0x72, 0x61, 0x6d) - if err != nil { - return - } - if z.FloatHistogram == nil { - err = en.WriteNil() - if err != nil { - return - } - } else { - err = z.FloatHistogram.EncodeMsg(en) - if err != nil { - err = msgp.WrapError(err, "FloatHistogram") - return - } - } - return -} - -// MarshalMsg implements msgp.Marshaler -func (z *Histograms) MarshalMsg(b []byte) (o []byte, err error) { - o = msgp.Require(b, z.Msgsize()) - // map header, size 2 - // string "Histogram" - o = append(o, 0x82, 0xa9, 0x48, 0x69, 0x73, 0x74, 0x6f, 0x67, 0x72, 0x61, 0x6d) - if z.Histogram == nil { - o = msgp.AppendNil(o) - } else { - o, err = z.Histogram.MarshalMsg(o) - if err != nil { - err = msgp.WrapError(err, "Histogram") - return - } - } - // string "FloatHistogram" - o = append(o, 0xae, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x48, 0x69, 0x73, 0x74, 0x6f, 0x67, 0x72, 0x61, 0x6d) - if z.FloatHistogram == nil { - o = msgp.AppendNil(o) - } else { - o, err = z.FloatHistogram.MarshalMsg(o) - if err != nil { - err = msgp.WrapError(err, "FloatHistogram") - return - } - } - return -} - -// UnmarshalMsg implements msgp.Unmarshaler -func (z *Histograms) UnmarshalMsg(bts []byte) (o []byte, err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "Histogram": - if msgp.IsNil(bts) { - bts, err = msgp.ReadNilBytes(bts) - if err != nil { - return - } - z.Histogram = nil - } else { - if z.Histogram == nil { - z.Histogram = new(Histogram) - } - bts, err = z.Histogram.UnmarshalMsg(bts) - if err != nil { - err = msgp.WrapError(err, "Histogram") - return - } - } - case "FloatHistogram": - if msgp.IsNil(bts) { - bts, err = msgp.ReadNilBytes(bts) - if err != nil { - return - } - z.FloatHistogram = nil - } else { - if z.FloatHistogram == nil { - z.FloatHistogram = new(FloatHistogram) - } - bts, err = z.FloatHistogram.UnmarshalMsg(bts) - if err != nil { - err = msgp.WrapError(err, "FloatHistogram") - return - } - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - o = bts - return -} - -// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message -func (z *Histograms) Msgsize() (s int) { - s = 1 + 10 - if z.Histogram == nil { - s += msgp.NilSize - } else { - s += z.Histogram.Msgsize() - } - s += 15 - if z.FloatHistogram == nil { - s += msgp.NilSize - } else { - s += z.FloatHistogram.Msgsize() - } - return -} - -// DecodeMsg implements msgp.Decodable -func (z *SeriesGroup) DecodeMsg(dc *msgp.Reader) (err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "Strings": - var zb0002 uint32 - zb0002, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "Strings") - return - } - if cap(z.Strings) >= int(zb0002) { - z.Strings = (z.Strings)[:zb0002] - } else { - z.Strings = make([]string, zb0002) - } - for za0001 := range z.Strings { - z.Strings[za0001], err = dc.ReadString() - if err != nil { - err = msgp.WrapError(err, "Strings", za0001) - return - } - } - case "Series": - var zb0003 uint32 - zb0003, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "Series") - return - } - if cap(z.Series) >= int(zb0003) { - z.Series = (z.Series)[:zb0003] - } else { - z.Series = make([]*TimeSeriesBinary, zb0003) - } - for za0002 := range z.Series { - if dc.IsNil() { - err = dc.ReadNil() - if err != nil { - err = msgp.WrapError(err, "Series", za0002) - return - } - z.Series[za0002] = nil - } else { - if z.Series[za0002] == nil { - z.Series[za0002] = new(TimeSeriesBinary) - } - err = z.Series[za0002].DecodeMsg(dc) - if err != nil { - err = msgp.WrapError(err, "Series", za0002) - return - } - } - } - case "Metadata": - var zb0004 uint32 - zb0004, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "Metadata") - return - } - if cap(z.Metadata) >= int(zb0004) { - z.Metadata = (z.Metadata)[:zb0004] - } else { - z.Metadata = make([]*TimeSeriesBinary, zb0004) - } - for za0003 := range z.Metadata { - if dc.IsNil() { - err = dc.ReadNil() - if err != nil { - err = msgp.WrapError(err, "Metadata", za0003) - return - } - z.Metadata[za0003] = nil - } else { - if z.Metadata[za0003] == nil { - z.Metadata[za0003] = new(TimeSeriesBinary) - } - err = z.Metadata[za0003].DecodeMsg(dc) - if err != nil { - err = msgp.WrapError(err, "Metadata", za0003) - return - } - } - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - return -} - -// EncodeMsg implements msgp.Encodable -func (z *SeriesGroup) EncodeMsg(en *msgp.Writer) (err error) { - // map header, size 3 - // write "Strings" - err = en.Append(0x83, 0xa7, 0x53, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.Strings))) - if err != nil { - err = msgp.WrapError(err, "Strings") - return - } - for za0001 := range z.Strings { - err = en.WriteString(z.Strings[za0001]) - if err != nil { - err = msgp.WrapError(err, "Strings", za0001) - return - } - } - // write "Series" - err = en.Append(0xa6, 0x53, 0x65, 0x72, 0x69, 0x65, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.Series))) - if err != nil { - err = msgp.WrapError(err, "Series") - return - } - for za0002 := range z.Series { - if z.Series[za0002] == nil { - err = en.WriteNil() - if err != nil { - return - } - } else { - err = z.Series[za0002].EncodeMsg(en) - if err != nil { - err = msgp.WrapError(err, "Series", za0002) - return - } - } - } - // write "Metadata" - err = en.Append(0xa8, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.Metadata))) - if err != nil { - err = msgp.WrapError(err, "Metadata") - return - } - for za0003 := range z.Metadata { - if z.Metadata[za0003] == nil { - err = en.WriteNil() - if err != nil { - return - } - } else { - err = z.Metadata[za0003].EncodeMsg(en) - if err != nil { - err = msgp.WrapError(err, "Metadata", za0003) - return - } - } - } - return -} - -// MarshalMsg implements msgp.Marshaler -func (z *SeriesGroup) MarshalMsg(b []byte) (o []byte, err error) { - o = msgp.Require(b, z.Msgsize()) - // map header, size 3 - // string "Strings" - o = append(o, 0x83, 0xa7, 0x53, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.Strings))) - for za0001 := range z.Strings { - o = msgp.AppendString(o, z.Strings[za0001]) - } - // string "Series" - o = append(o, 0xa6, 0x53, 0x65, 0x72, 0x69, 0x65, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.Series))) - for za0002 := range z.Series { - if z.Series[za0002] == nil { - o = msgp.AppendNil(o) - } else { - o, err = z.Series[za0002].MarshalMsg(o) - if err != nil { - err = msgp.WrapError(err, "Series", za0002) - return - } - } - } - // string "Metadata" - o = append(o, 0xa8, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61) - o = msgp.AppendArrayHeader(o, uint32(len(z.Metadata))) - for za0003 := range z.Metadata { - if z.Metadata[za0003] == nil { - o = msgp.AppendNil(o) - } else { - o, err = z.Metadata[za0003].MarshalMsg(o) - if err != nil { - err = msgp.WrapError(err, "Metadata", za0003) - return - } - } - } - return -} - -// UnmarshalMsg implements msgp.Unmarshaler -func (z *SeriesGroup) UnmarshalMsg(bts []byte) (o []byte, err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "Strings": - var zb0002 uint32 - zb0002, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "Strings") - return - } - if cap(z.Strings) >= int(zb0002) { - z.Strings = (z.Strings)[:zb0002] - } else { - z.Strings = make([]string, zb0002) - } - for za0001 := range z.Strings { - z.Strings[za0001], bts, err = msgp.ReadStringBytes(bts) - if err != nil { - err = msgp.WrapError(err, "Strings", za0001) - return - } - } - case "Series": - var zb0003 uint32 - zb0003, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "Series") - return - } - if cap(z.Series) >= int(zb0003) { - z.Series = (z.Series)[:zb0003] - } else { - z.Series = make([]*TimeSeriesBinary, zb0003) - } - for za0002 := range z.Series { - if msgp.IsNil(bts) { - bts, err = msgp.ReadNilBytes(bts) - if err != nil { - return - } - z.Series[za0002] = nil - } else { - if z.Series[za0002] == nil { - z.Series[za0002] = new(TimeSeriesBinary) - } - bts, err = z.Series[za0002].UnmarshalMsg(bts) - if err != nil { - err = msgp.WrapError(err, "Series", za0002) - return - } - } - } - case "Metadata": - var zb0004 uint32 - zb0004, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "Metadata") - return - } - if cap(z.Metadata) >= int(zb0004) { - z.Metadata = (z.Metadata)[:zb0004] - } else { - z.Metadata = make([]*TimeSeriesBinary, zb0004) - } - for za0003 := range z.Metadata { - if msgp.IsNil(bts) { - bts, err = msgp.ReadNilBytes(bts) - if err != nil { - return - } - z.Metadata[za0003] = nil - } else { - if z.Metadata[za0003] == nil { - z.Metadata[za0003] = new(TimeSeriesBinary) - } - bts, err = z.Metadata[za0003].UnmarshalMsg(bts) - if err != nil { - err = msgp.WrapError(err, "Metadata", za0003) - return - } - } - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - o = bts - return -} - -// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message -func (z *SeriesGroup) Msgsize() (s int) { - s = 1 + 8 + msgp.ArrayHeaderSize - for za0001 := range z.Strings { - s += msgp.StringPrefixSize + len(z.Strings[za0001]) - } - s += 7 + msgp.ArrayHeaderSize - for za0002 := range z.Series { - if z.Series[za0002] == nil { - s += msgp.NilSize - } else { - s += z.Series[za0002].Msgsize() - } - } - s += 9 + msgp.ArrayHeaderSize - for za0003 := range z.Metadata { - if z.Metadata[za0003] == nil { - s += msgp.NilSize - } else { - s += z.Metadata[za0003].Msgsize() - } - } - return -} - -// DecodeMsg implements msgp.Decodable -func (z *TimeSeriesBinary) DecodeMsg(dc *msgp.Reader) (err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, err = dc.ReadMapHeader() - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, err = dc.ReadMapKeyPtr() - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "LabelsNames": - var zb0002 uint32 - zb0002, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "LabelsNames") - return - } - if cap(z.LabelsNames) >= int(zb0002) { - z.LabelsNames = (z.LabelsNames)[:zb0002] - } else { - z.LabelsNames = make([]uint32, zb0002) - } - for za0001 := range z.LabelsNames { - z.LabelsNames[za0001], err = dc.ReadUint32() - if err != nil { - err = msgp.WrapError(err, "LabelsNames", za0001) - return - } - } - case "LabelsValues": - var zb0003 uint32 - zb0003, err = dc.ReadArrayHeader() - if err != nil { - err = msgp.WrapError(err, "LabelsValues") - return - } - if cap(z.LabelsValues) >= int(zb0003) { - z.LabelsValues = (z.LabelsValues)[:zb0003] - } else { - z.LabelsValues = make([]uint32, zb0003) - } - for za0002 := range z.LabelsValues { - z.LabelsValues[za0002], err = dc.ReadUint32() - if err != nil { - err = msgp.WrapError(err, "LabelsValues", za0002) - return - } - } - case "TS": - z.TS, err = dc.ReadInt64() - if err != nil { - err = msgp.WrapError(err, "TS") - return - } - case "Value": - z.Value, err = dc.ReadFloat64() - if err != nil { - err = msgp.WrapError(err, "Value") - return - } - case "Hash": - z.Hash, err = dc.ReadUint64() - if err != nil { - err = msgp.WrapError(err, "Hash") - return - } - case "Histograms": - err = z.Histograms.DecodeMsg(dc) - if err != nil { - err = msgp.WrapError(err, "Histograms") - return - } - default: - err = dc.Skip() - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - return -} - -// EncodeMsg implements msgp.Encodable -func (z *TimeSeriesBinary) EncodeMsg(en *msgp.Writer) (err error) { - // map header, size 6 - // write "LabelsNames" - err = en.Append(0x86, 0xab, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x4e, 0x61, 0x6d, 0x65, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.LabelsNames))) - if err != nil { - err = msgp.WrapError(err, "LabelsNames") - return - } - for za0001 := range z.LabelsNames { - err = en.WriteUint32(z.LabelsNames[za0001]) - if err != nil { - err = msgp.WrapError(err, "LabelsNames", za0001) - return - } - } - // write "LabelsValues" - err = en.Append(0xac, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73) - if err != nil { - return - } - err = en.WriteArrayHeader(uint32(len(z.LabelsValues))) - if err != nil { - err = msgp.WrapError(err, "LabelsValues") - return - } - for za0002 := range z.LabelsValues { - err = en.WriteUint32(z.LabelsValues[za0002]) - if err != nil { - err = msgp.WrapError(err, "LabelsValues", za0002) - return - } - } - // write "TS" - err = en.Append(0xa2, 0x54, 0x53) - if err != nil { - return - } - err = en.WriteInt64(z.TS) - if err != nil { - err = msgp.WrapError(err, "TS") - return - } - // write "Value" - err = en.Append(0xa5, 0x56, 0x61, 0x6c, 0x75, 0x65) - if err != nil { - return - } - err = en.WriteFloat64(z.Value) - if err != nil { - err = msgp.WrapError(err, "Value") - return - } - // write "Hash" - err = en.Append(0xa4, 0x48, 0x61, 0x73, 0x68) - if err != nil { - return - } - err = en.WriteUint64(z.Hash) - if err != nil { - err = msgp.WrapError(err, "Hash") - return - } - // write "Histograms" - err = en.Append(0xaa, 0x48, 0x69, 0x73, 0x74, 0x6f, 0x67, 0x72, 0x61, 0x6d, 0x73) - if err != nil { - return - } - err = z.Histograms.EncodeMsg(en) - if err != nil { - err = msgp.WrapError(err, "Histograms") - return - } - return -} - -// MarshalMsg implements msgp.Marshaler -func (z *TimeSeriesBinary) MarshalMsg(b []byte) (o []byte, err error) { - o = msgp.Require(b, z.Msgsize()) - // map header, size 6 - // string "LabelsNames" - o = append(o, 0x86, 0xab, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x4e, 0x61, 0x6d, 0x65, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.LabelsNames))) - for za0001 := range z.LabelsNames { - o = msgp.AppendUint32(o, z.LabelsNames[za0001]) - } - // string "LabelsValues" - o = append(o, 0xac, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73) - o = msgp.AppendArrayHeader(o, uint32(len(z.LabelsValues))) - for za0002 := range z.LabelsValues { - o = msgp.AppendUint32(o, z.LabelsValues[za0002]) - } - // string "TS" - o = append(o, 0xa2, 0x54, 0x53) - o = msgp.AppendInt64(o, z.TS) - // string "Value" - o = append(o, 0xa5, 0x56, 0x61, 0x6c, 0x75, 0x65) - o = msgp.AppendFloat64(o, z.Value) - // string "Hash" - o = append(o, 0xa4, 0x48, 0x61, 0x73, 0x68) - o = msgp.AppendUint64(o, z.Hash) - // string "Histograms" - o = append(o, 0xaa, 0x48, 0x69, 0x73, 0x74, 0x6f, 0x67, 0x72, 0x61, 0x6d, 0x73) - o, err = z.Histograms.MarshalMsg(o) - if err != nil { - err = msgp.WrapError(err, "Histograms") - return - } - return -} - -// UnmarshalMsg implements msgp.Unmarshaler -func (z *TimeSeriesBinary) UnmarshalMsg(bts []byte) (o []byte, err error) { - var field []byte - _ = field - var zb0001 uint32 - zb0001, bts, err = msgp.ReadMapHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - for zb0001 > 0 { - zb0001-- - field, bts, err = msgp.ReadMapKeyZC(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - switch msgp.UnsafeString(field) { - case "LabelsNames": - var zb0002 uint32 - zb0002, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "LabelsNames") - return - } - if cap(z.LabelsNames) >= int(zb0002) { - z.LabelsNames = (z.LabelsNames)[:zb0002] - } else { - z.LabelsNames = make([]uint32, zb0002) - } - for za0001 := range z.LabelsNames { - z.LabelsNames[za0001], bts, err = msgp.ReadUint32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "LabelsNames", za0001) - return - } - } - case "LabelsValues": - var zb0003 uint32 - zb0003, bts, err = msgp.ReadArrayHeaderBytes(bts) - if err != nil { - err = msgp.WrapError(err, "LabelsValues") - return - } - if cap(z.LabelsValues) >= int(zb0003) { - z.LabelsValues = (z.LabelsValues)[:zb0003] - } else { - z.LabelsValues = make([]uint32, zb0003) - } - for za0002 := range z.LabelsValues { - z.LabelsValues[za0002], bts, err = msgp.ReadUint32Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "LabelsValues", za0002) - return - } - } - case "TS": - z.TS, bts, err = msgp.ReadInt64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "TS") - return - } - case "Value": - z.Value, bts, err = msgp.ReadFloat64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "Value") - return - } - case "Hash": - z.Hash, bts, err = msgp.ReadUint64Bytes(bts) - if err != nil { - err = msgp.WrapError(err, "Hash") - return - } - case "Histograms": - bts, err = z.Histograms.UnmarshalMsg(bts) - if err != nil { - err = msgp.WrapError(err, "Histograms") - return - } - default: - bts, err = msgp.Skip(bts) - if err != nil { - err = msgp.WrapError(err) - return - } - } - } - o = bts - return -} - -// Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message -func (z *TimeSeriesBinary) Msgsize() (s int) { - s = 1 + 12 + msgp.ArrayHeaderSize + (len(z.LabelsNames) * (msgp.Uint32Size)) + 13 + msgp.ArrayHeaderSize + (len(z.LabelsValues) * (msgp.Uint32Size)) + 3 + msgp.Int64Size + 6 + msgp.Float64Size + 5 + msgp.Uint64Size + 11 + z.Histograms.Msgsize() - return -} diff --git a/internal/component/prometheus/write/queue/types/serialization_gen_test.go b/internal/component/prometheus/write/queue/types/serialization_gen_test.go deleted file mode 100644 index e6e18c7901..0000000000 --- a/internal/component/prometheus/write/queue/types/serialization_gen_test.go +++ /dev/null @@ -1,914 +0,0 @@ -package types - -// Code generated by github.com/tinylib/msgp DO NOT EDIT. - -import ( - "bytes" - "testing" - - "github.com/tinylib/msgp/msgp" -) - -func TestMarshalUnmarshalBucketSpan(t *testing.T) { - v := BucketSpan{} - bts, err := v.MarshalMsg(nil) - if err != nil { - t.Fatal(err) - } - left, err := v.UnmarshalMsg(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left) - } - - left, err = msgp.Skip(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after Skip(): %q", len(left), left) - } -} - -func BenchmarkMarshalMsgBucketSpan(b *testing.B) { - v := BucketSpan{} - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.MarshalMsg(nil) - } -} - -func BenchmarkAppendMsgBucketSpan(b *testing.B) { - v := BucketSpan{} - bts := make([]byte, 0, v.Msgsize()) - bts, _ = v.MarshalMsg(bts[0:0]) - b.SetBytes(int64(len(bts))) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - bts, _ = v.MarshalMsg(bts[0:0]) - } -} - -func BenchmarkUnmarshalBucketSpan(b *testing.B) { - v := BucketSpan{} - bts, _ := v.MarshalMsg(nil) - b.ReportAllocs() - b.SetBytes(int64(len(bts))) - b.ResetTimer() - for i := 0; i < b.N; i++ { - _, err := v.UnmarshalMsg(bts) - if err != nil { - b.Fatal(err) - } - } -} - -func TestEncodeDecodeBucketSpan(t *testing.T) { - v := BucketSpan{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - - m := v.Msgsize() - if buf.Len() > m { - t.Log("WARNING: TestEncodeDecodeBucketSpan Msgsize() is inaccurate") - } - - vn := BucketSpan{} - err := msgp.Decode(&buf, &vn) - if err != nil { - t.Error(err) - } - - buf.Reset() - msgp.Encode(&buf, &v) - err = msgp.NewReader(&buf).Skip() - if err != nil { - t.Error(err) - } -} - -func BenchmarkEncodeBucketSpan(b *testing.B) { - v := BucketSpan{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - en := msgp.NewWriter(msgp.Nowhere) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.EncodeMsg(en) - } - en.Flush() -} - -func BenchmarkDecodeBucketSpan(b *testing.B) { - v := BucketSpan{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - rd := msgp.NewEndlessReader(buf.Bytes(), b) - dc := msgp.NewReader(rd) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - err := v.DecodeMsg(dc) - if err != nil { - b.Fatal(err) - } - } -} - -func TestMarshalUnmarshalFloatHistogram(t *testing.T) { - v := FloatHistogram{} - bts, err := v.MarshalMsg(nil) - if err != nil { - t.Fatal(err) - } - left, err := v.UnmarshalMsg(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left) - } - - left, err = msgp.Skip(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after Skip(): %q", len(left), left) - } -} - -func BenchmarkMarshalMsgFloatHistogram(b *testing.B) { - v := FloatHistogram{} - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.MarshalMsg(nil) - } -} - -func BenchmarkAppendMsgFloatHistogram(b *testing.B) { - v := FloatHistogram{} - bts := make([]byte, 0, v.Msgsize()) - bts, _ = v.MarshalMsg(bts[0:0]) - b.SetBytes(int64(len(bts))) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - bts, _ = v.MarshalMsg(bts[0:0]) - } -} - -func BenchmarkUnmarshalFloatHistogram(b *testing.B) { - v := FloatHistogram{} - bts, _ := v.MarshalMsg(nil) - b.ReportAllocs() - b.SetBytes(int64(len(bts))) - b.ResetTimer() - for i := 0; i < b.N; i++ { - _, err := v.UnmarshalMsg(bts) - if err != nil { - b.Fatal(err) - } - } -} - -func TestEncodeDecodeFloatHistogram(t *testing.T) { - v := FloatHistogram{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - - m := v.Msgsize() - if buf.Len() > m { - t.Log("WARNING: TestEncodeDecodeFloatHistogram Msgsize() is inaccurate") - } - - vn := FloatHistogram{} - err := msgp.Decode(&buf, &vn) - if err != nil { - t.Error(err) - } - - buf.Reset() - msgp.Encode(&buf, &v) - err = msgp.NewReader(&buf).Skip() - if err != nil { - t.Error(err) - } -} - -func BenchmarkEncodeFloatHistogram(b *testing.B) { - v := FloatHistogram{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - en := msgp.NewWriter(msgp.Nowhere) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.EncodeMsg(en) - } - en.Flush() -} - -func BenchmarkDecodeFloatHistogram(b *testing.B) { - v := FloatHistogram{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - rd := msgp.NewEndlessReader(buf.Bytes(), b) - dc := msgp.NewReader(rd) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - err := v.DecodeMsg(dc) - if err != nil { - b.Fatal(err) - } - } -} - -func TestMarshalUnmarshalHistogram(t *testing.T) { - v := Histogram{} - bts, err := v.MarshalMsg(nil) - if err != nil { - t.Fatal(err) - } - left, err := v.UnmarshalMsg(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left) - } - - left, err = msgp.Skip(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after Skip(): %q", len(left), left) - } -} - -func BenchmarkMarshalMsgHistogram(b *testing.B) { - v := Histogram{} - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.MarshalMsg(nil) - } -} - -func BenchmarkAppendMsgHistogram(b *testing.B) { - v := Histogram{} - bts := make([]byte, 0, v.Msgsize()) - bts, _ = v.MarshalMsg(bts[0:0]) - b.SetBytes(int64(len(bts))) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - bts, _ = v.MarshalMsg(bts[0:0]) - } -} - -func BenchmarkUnmarshalHistogram(b *testing.B) { - v := Histogram{} - bts, _ := v.MarshalMsg(nil) - b.ReportAllocs() - b.SetBytes(int64(len(bts))) - b.ResetTimer() - for i := 0; i < b.N; i++ { - _, err := v.UnmarshalMsg(bts) - if err != nil { - b.Fatal(err) - } - } -} - -func TestEncodeDecodeHistogram(t *testing.T) { - v := Histogram{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - - m := v.Msgsize() - if buf.Len() > m { - t.Log("WARNING: TestEncodeDecodeHistogram Msgsize() is inaccurate") - } - - vn := Histogram{} - err := msgp.Decode(&buf, &vn) - if err != nil { - t.Error(err) - } - - buf.Reset() - msgp.Encode(&buf, &v) - err = msgp.NewReader(&buf).Skip() - if err != nil { - t.Error(err) - } -} - -func BenchmarkEncodeHistogram(b *testing.B) { - v := Histogram{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - en := msgp.NewWriter(msgp.Nowhere) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.EncodeMsg(en) - } - en.Flush() -} - -func BenchmarkDecodeHistogram(b *testing.B) { - v := Histogram{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - rd := msgp.NewEndlessReader(buf.Bytes(), b) - dc := msgp.NewReader(rd) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - err := v.DecodeMsg(dc) - if err != nil { - b.Fatal(err) - } - } -} - -func TestMarshalUnmarshalHistogramCount(t *testing.T) { - v := HistogramCount{} - bts, err := v.MarshalMsg(nil) - if err != nil { - t.Fatal(err) - } - left, err := v.UnmarshalMsg(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left) - } - - left, err = msgp.Skip(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after Skip(): %q", len(left), left) - } -} - -func BenchmarkMarshalMsgHistogramCount(b *testing.B) { - v := HistogramCount{} - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.MarshalMsg(nil) - } -} - -func BenchmarkAppendMsgHistogramCount(b *testing.B) { - v := HistogramCount{} - bts := make([]byte, 0, v.Msgsize()) - bts, _ = v.MarshalMsg(bts[0:0]) - b.SetBytes(int64(len(bts))) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - bts, _ = v.MarshalMsg(bts[0:0]) - } -} - -func BenchmarkUnmarshalHistogramCount(b *testing.B) { - v := HistogramCount{} - bts, _ := v.MarshalMsg(nil) - b.ReportAllocs() - b.SetBytes(int64(len(bts))) - b.ResetTimer() - for i := 0; i < b.N; i++ { - _, err := v.UnmarshalMsg(bts) - if err != nil { - b.Fatal(err) - } - } -} - -func TestEncodeDecodeHistogramCount(t *testing.T) { - v := HistogramCount{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - - m := v.Msgsize() - if buf.Len() > m { - t.Log("WARNING: TestEncodeDecodeHistogramCount Msgsize() is inaccurate") - } - - vn := HistogramCount{} - err := msgp.Decode(&buf, &vn) - if err != nil { - t.Error(err) - } - - buf.Reset() - msgp.Encode(&buf, &v) - err = msgp.NewReader(&buf).Skip() - if err != nil { - t.Error(err) - } -} - -func BenchmarkEncodeHistogramCount(b *testing.B) { - v := HistogramCount{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - en := msgp.NewWriter(msgp.Nowhere) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.EncodeMsg(en) - } - en.Flush() -} - -func BenchmarkDecodeHistogramCount(b *testing.B) { - v := HistogramCount{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - rd := msgp.NewEndlessReader(buf.Bytes(), b) - dc := msgp.NewReader(rd) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - err := v.DecodeMsg(dc) - if err != nil { - b.Fatal(err) - } - } -} - -func TestMarshalUnmarshalHistogramZeroCount(t *testing.T) { - v := HistogramZeroCount{} - bts, err := v.MarshalMsg(nil) - if err != nil { - t.Fatal(err) - } - left, err := v.UnmarshalMsg(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left) - } - - left, err = msgp.Skip(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after Skip(): %q", len(left), left) - } -} - -func BenchmarkMarshalMsgHistogramZeroCount(b *testing.B) { - v := HistogramZeroCount{} - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.MarshalMsg(nil) - } -} - -func BenchmarkAppendMsgHistogramZeroCount(b *testing.B) { - v := HistogramZeroCount{} - bts := make([]byte, 0, v.Msgsize()) - bts, _ = v.MarshalMsg(bts[0:0]) - b.SetBytes(int64(len(bts))) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - bts, _ = v.MarshalMsg(bts[0:0]) - } -} - -func BenchmarkUnmarshalHistogramZeroCount(b *testing.B) { - v := HistogramZeroCount{} - bts, _ := v.MarshalMsg(nil) - b.ReportAllocs() - b.SetBytes(int64(len(bts))) - b.ResetTimer() - for i := 0; i < b.N; i++ { - _, err := v.UnmarshalMsg(bts) - if err != nil { - b.Fatal(err) - } - } -} - -func TestEncodeDecodeHistogramZeroCount(t *testing.T) { - v := HistogramZeroCount{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - - m := v.Msgsize() - if buf.Len() > m { - t.Log("WARNING: TestEncodeDecodeHistogramZeroCount Msgsize() is inaccurate") - } - - vn := HistogramZeroCount{} - err := msgp.Decode(&buf, &vn) - if err != nil { - t.Error(err) - } - - buf.Reset() - msgp.Encode(&buf, &v) - err = msgp.NewReader(&buf).Skip() - if err != nil { - t.Error(err) - } -} - -func BenchmarkEncodeHistogramZeroCount(b *testing.B) { - v := HistogramZeroCount{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - en := msgp.NewWriter(msgp.Nowhere) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.EncodeMsg(en) - } - en.Flush() -} - -func BenchmarkDecodeHistogramZeroCount(b *testing.B) { - v := HistogramZeroCount{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - rd := msgp.NewEndlessReader(buf.Bytes(), b) - dc := msgp.NewReader(rd) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - err := v.DecodeMsg(dc) - if err != nil { - b.Fatal(err) - } - } -} - -func TestMarshalUnmarshalHistograms(t *testing.T) { - v := Histograms{} - bts, err := v.MarshalMsg(nil) - if err != nil { - t.Fatal(err) - } - left, err := v.UnmarshalMsg(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left) - } - - left, err = msgp.Skip(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after Skip(): %q", len(left), left) - } -} - -func BenchmarkMarshalMsgHistograms(b *testing.B) { - v := Histograms{} - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.MarshalMsg(nil) - } -} - -func BenchmarkAppendMsgHistograms(b *testing.B) { - v := Histograms{} - bts := make([]byte, 0, v.Msgsize()) - bts, _ = v.MarshalMsg(bts[0:0]) - b.SetBytes(int64(len(bts))) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - bts, _ = v.MarshalMsg(bts[0:0]) - } -} - -func BenchmarkUnmarshalHistograms(b *testing.B) { - v := Histograms{} - bts, _ := v.MarshalMsg(nil) - b.ReportAllocs() - b.SetBytes(int64(len(bts))) - b.ResetTimer() - for i := 0; i < b.N; i++ { - _, err := v.UnmarshalMsg(bts) - if err != nil { - b.Fatal(err) - } - } -} - -func TestEncodeDecodeHistograms(t *testing.T) { - v := Histograms{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - - m := v.Msgsize() - if buf.Len() > m { - t.Log("WARNING: TestEncodeDecodeHistograms Msgsize() is inaccurate") - } - - vn := Histograms{} - err := msgp.Decode(&buf, &vn) - if err != nil { - t.Error(err) - } - - buf.Reset() - msgp.Encode(&buf, &v) - err = msgp.NewReader(&buf).Skip() - if err != nil { - t.Error(err) - } -} - -func BenchmarkEncodeHistograms(b *testing.B) { - v := Histograms{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - en := msgp.NewWriter(msgp.Nowhere) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.EncodeMsg(en) - } - en.Flush() -} - -func BenchmarkDecodeHistograms(b *testing.B) { - v := Histograms{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - rd := msgp.NewEndlessReader(buf.Bytes(), b) - dc := msgp.NewReader(rd) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - err := v.DecodeMsg(dc) - if err != nil { - b.Fatal(err) - } - } -} - -func TestMarshalUnmarshalSeriesGroup(t *testing.T) { - v := SeriesGroup{} - bts, err := v.MarshalMsg(nil) - if err != nil { - t.Fatal(err) - } - left, err := v.UnmarshalMsg(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left) - } - - left, err = msgp.Skip(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after Skip(): %q", len(left), left) - } -} - -func BenchmarkMarshalMsgSeriesGroup(b *testing.B) { - v := SeriesGroup{} - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.MarshalMsg(nil) - } -} - -func BenchmarkAppendMsgSeriesGroup(b *testing.B) { - v := SeriesGroup{} - bts := make([]byte, 0, v.Msgsize()) - bts, _ = v.MarshalMsg(bts[0:0]) - b.SetBytes(int64(len(bts))) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - bts, _ = v.MarshalMsg(bts[0:0]) - } -} - -func BenchmarkUnmarshalSeriesGroup(b *testing.B) { - v := SeriesGroup{} - bts, _ := v.MarshalMsg(nil) - b.ReportAllocs() - b.SetBytes(int64(len(bts))) - b.ResetTimer() - for i := 0; i < b.N; i++ { - _, err := v.UnmarshalMsg(bts) - if err != nil { - b.Fatal(err) - } - } -} - -func TestEncodeDecodeSeriesGroup(t *testing.T) { - v := SeriesGroup{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - - m := v.Msgsize() - if buf.Len() > m { - t.Log("WARNING: TestEncodeDecodeSeriesGroup Msgsize() is inaccurate") - } - - vn := SeriesGroup{} - err := msgp.Decode(&buf, &vn) - if err != nil { - t.Error(err) - } - - buf.Reset() - msgp.Encode(&buf, &v) - err = msgp.NewReader(&buf).Skip() - if err != nil { - t.Error(err) - } -} - -func BenchmarkEncodeSeriesGroup(b *testing.B) { - v := SeriesGroup{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - en := msgp.NewWriter(msgp.Nowhere) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.EncodeMsg(en) - } - en.Flush() -} - -func BenchmarkDecodeSeriesGroup(b *testing.B) { - v := SeriesGroup{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - rd := msgp.NewEndlessReader(buf.Bytes(), b) - dc := msgp.NewReader(rd) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - err := v.DecodeMsg(dc) - if err != nil { - b.Fatal(err) - } - } -} - -func TestMarshalUnmarshalTimeSeriesBinary(t *testing.T) { - v := TimeSeriesBinary{} - bts, err := v.MarshalMsg(nil) - if err != nil { - t.Fatal(err) - } - left, err := v.UnmarshalMsg(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after UnmarshalMsg(): %q", len(left), left) - } - - left, err = msgp.Skip(bts) - if err != nil { - t.Fatal(err) - } - if len(left) > 0 { - t.Errorf("%d bytes left over after Skip(): %q", len(left), left) - } -} - -func BenchmarkMarshalMsgTimeSeriesBinary(b *testing.B) { - v := TimeSeriesBinary{} - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.MarshalMsg(nil) - } -} - -func BenchmarkAppendMsgTimeSeriesBinary(b *testing.B) { - v := TimeSeriesBinary{} - bts := make([]byte, 0, v.Msgsize()) - bts, _ = v.MarshalMsg(bts[0:0]) - b.SetBytes(int64(len(bts))) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - bts, _ = v.MarshalMsg(bts[0:0]) - } -} - -func BenchmarkUnmarshalTimeSeriesBinary(b *testing.B) { - v := TimeSeriesBinary{} - bts, _ := v.MarshalMsg(nil) - b.ReportAllocs() - b.SetBytes(int64(len(bts))) - b.ResetTimer() - for i := 0; i < b.N; i++ { - _, err := v.UnmarshalMsg(bts) - if err != nil { - b.Fatal(err) - } - } -} - -func TestEncodeDecodeTimeSeriesBinary(t *testing.T) { - v := TimeSeriesBinary{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - - m := v.Msgsize() - if buf.Len() > m { - t.Log("WARNING: TestEncodeDecodeTimeSeriesBinary Msgsize() is inaccurate") - } - - vn := TimeSeriesBinary{} - err := msgp.Decode(&buf, &vn) - if err != nil { - t.Error(err) - } - - buf.Reset() - msgp.Encode(&buf, &v) - err = msgp.NewReader(&buf).Skip() - if err != nil { - t.Error(err) - } -} - -func BenchmarkEncodeTimeSeriesBinary(b *testing.B) { - v := TimeSeriesBinary{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - en := msgp.NewWriter(msgp.Nowhere) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - v.EncodeMsg(en) - } - en.Flush() -} - -func BenchmarkDecodeTimeSeriesBinary(b *testing.B) { - v := TimeSeriesBinary{} - var buf bytes.Buffer - msgp.Encode(&buf, &v) - b.SetBytes(int64(buf.Len())) - rd := msgp.NewEndlessReader(buf.Bytes(), b) - dc := msgp.NewReader(rd) - b.ReportAllocs() - b.ResetTimer() - for i := 0; i < b.N; i++ { - err := v.DecodeMsg(dc) - if err != nil { - b.Fatal(err) - } - } -} diff --git a/internal/component/prometheus/write/queue/types/serialization_test.go b/internal/component/prometheus/write/queue/types/serialization_test.go deleted file mode 100644 index 59f6d077ae..0000000000 --- a/internal/component/prometheus/write/queue/types/serialization_test.go +++ /dev/null @@ -1,59 +0,0 @@ -package types - -import ( - "fmt" - "math/rand" - "testing" - - "github.com/prometheus/prometheus/model/labels" - "github.com/stretchr/testify/require" -) - -func TestLabels(t *testing.T) { - lblsMap := make(map[string]string) - unique := make(map[string]struct{}) - for i := 0; i < 1_000; i++ { - k := fmt.Sprintf("key_%d", i) - v := randString() - lblsMap[k] = v - unique[k] = struct{}{} - unique[v] = struct{}{} - } - sg := &SeriesGroup{ - Series: make([]*TimeSeriesBinary, 1), - } - sg.Series[0] = GetTimeSeriesFromPool() - defer PutTimeSeriesIntoPool(sg.Series[0]) - sg.Series[0].Labels = labels.FromMap(lblsMap) - strMap := make(map[string]uint32) - - sg.Series[0].FillLabelMapping(strMap) - stringsSlice := make([]string, len(strMap)) - for k, v := range strMap { - stringsSlice[v] = k - } - sg.Strings = stringsSlice - buf, err := sg.MarshalMsg(nil) - require.NoError(t, err) - newSg := &SeriesGroup{} - newSg, _, err = DeserializeToSeriesGroup(newSg, buf) - require.NoError(t, err) - series1 := newSg.Series[0] - series2 := sg.Series[0] - require.Len(t, series2.Labels, len(series1.Labels)) - // Ensure we were able to convert back and forth properly. - for i, lbl := range series2.Labels { - require.Equal(t, lbl.Name, series1.Labels[i].Name) - require.Equal(t, lbl.Value, series1.Labels[i].Value) - } -} - -var letterRunes = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ") - -func randString() string { - b := make([]rune, rand.Intn(20)) - for i := range b { - b[i] = letterRunes[rand.Intn(len(letterRunes))] - } - return string(b) -} diff --git a/internal/component/prometheus/write/queue/types/serializer.go b/internal/component/prometheus/write/queue/types/serializer.go deleted file mode 100644 index d0041242cc..0000000000 --- a/internal/component/prometheus/write/queue/types/serializer.go +++ /dev/null @@ -1,24 +0,0 @@ -package types - -import ( - "context" - "time" -) - -const AlloyFileVersion = "alloy.metrics.queue.v1" - -type SerializerConfig struct { - // MaxSignalsInBatch controls what the max batch size is. - MaxSignalsInBatch uint32 - // FlushFrequency controls how often to write to disk regardless of MaxSignalsInBatch. - FlushFrequency time.Duration -} - -// Serializer handles converting a set of signals into a binary representation to be written to storage. -type Serializer interface { - Start() - Stop() - SendSeries(ctx context.Context, data *TimeSeriesBinary) error - SendMetadata(ctx context.Context, data *TimeSeriesBinary) error - UpdateConfig(ctx context.Context, cfg SerializerConfig) error -} diff --git a/internal/component/prometheus/write/queue/types/stats.go b/internal/component/prometheus/write/queue/types/stats.go deleted file mode 100644 index 732b6255aa..0000000000 --- a/internal/component/prometheus/write/queue/types/stats.go +++ /dev/null @@ -1,289 +0,0 @@ -package types - -import ( - "time" - - "github.com/prometheus/client_golang/prometheus" -) - -// TODO @mattdurham separate this into more manageable chunks, and likely 3 stats series: series, metadata and new ones. - -type SerializerStats struct { - SeriesStored int - MetadataStored int - Errors int - NewestTimestamp int64 -} - -type PrometheusStats struct { - // Network Stats - NetworkSeriesSent prometheus.Counter - NetworkFailures prometheus.Counter - NetworkRetries prometheus.Counter - NetworkRetries429 prometheus.Counter - NetworkRetries5XX prometheus.Counter - NetworkSentDuration prometheus.Histogram - NetworkErrors prometheus.Counter - NetworkNewestOutTimeStampSeconds prometheus.Gauge - - // Serializer Stats - SerializerInSeries prometheus.Counter - SerializerNewestInTimeStampSeconds prometheus.Gauge - SerializerErrors prometheus.Counter - - // Backwards compatibility metrics - SamplesTotal prometheus.Counter - HistogramsTotal prometheus.Counter - MetadataTotal prometheus.Counter - - FailedSamplesTotal prometheus.Counter - FailedHistogramsTotal prometheus.Counter - FailedMetadataTotal prometheus.Counter - - RetriedSamplesTotal prometheus.Counter - RetriedHistogramsTotal prometheus.Counter - RetriedMetadataTotal prometheus.Counter - - EnqueueRetriesTotal prometheus.Counter - SentBatchDuration prometheus.Histogram - HighestSentTimestamp prometheus.Gauge - - SentBytesTotal prometheus.Counter - MetadataBytesTotal prometheus.Counter - RemoteStorageInTimestamp prometheus.Gauge - RemoteStorageOutTimestamp prometheus.Gauge - RemoteStorageDuration prometheus.Histogram -} - -func NewStats(namespace, subsystem string, registry prometheus.Registerer) *PrometheusStats { - s := &PrometheusStats{ - SerializerInSeries: prometheus.NewCounter(prometheus.CounterOpts{ - Namespace: namespace, - Subsystem: subsystem, - Name: "serializer_incoming_signals", - }), - SerializerNewestInTimeStampSeconds: prometheus.NewGauge(prometheus.GaugeOpts{ - Namespace: namespace, - Subsystem: subsystem, - Name: "serializer_incoming_timestamp_seconds", - }), - SerializerErrors: prometheus.NewGauge(prometheus.GaugeOpts{ - Namespace: namespace, - Subsystem: subsystem, - Name: "serializer_errors", - }), - NetworkNewestOutTimeStampSeconds: prometheus.NewGauge(prometheus.GaugeOpts{ - Namespace: namespace, - Subsystem: subsystem, - Name: "network_timestamp_seconds", - }), - RemoteStorageDuration: prometheus.NewHistogram(prometheus.HistogramOpts{ - Name: "prometheus_remote_storage_queue_duration_seconds", - }), - NetworkSeriesSent: prometheus.NewCounter(prometheus.CounterOpts{ - Namespace: namespace, - Subsystem: subsystem, - Name: "network_sent", - }), - NetworkFailures: prometheus.NewCounter(prometheus.CounterOpts{ - Namespace: namespace, - Subsystem: subsystem, - Name: "network_failed", - }), - NetworkRetries: prometheus.NewCounter(prometheus.CounterOpts{ - Namespace: namespace, - Subsystem: subsystem, - Name: "network_retried", - }), - NetworkRetries429: prometheus.NewCounter(prometheus.CounterOpts{ - Namespace: namespace, - Subsystem: subsystem, - Name: "network_retried_429", - }), - NetworkRetries5XX: prometheus.NewCounter(prometheus.CounterOpts{ - Namespace: namespace, - Subsystem: subsystem, - Name: "network_retried_5xx", - }), - NetworkSentDuration: prometheus.NewHistogram(prometheus.HistogramOpts{ - Namespace: namespace, - Subsystem: subsystem, - Name: "network_duration_seconds", - NativeHistogramBucketFactor: 1.1, - }), - NetworkErrors: prometheus.NewCounter(prometheus.CounterOpts{ - Namespace: namespace, - Subsystem: subsystem, - Name: "network_errors", - }), - RemoteStorageOutTimestamp: prometheus.NewGauge(prometheus.GaugeOpts{ - Name: "prometheus_remote_storage_queue_highest_sent_timestamp_seconds", - }), - RemoteStorageInTimestamp: prometheus.NewGauge(prometheus.GaugeOpts{ - Name: "prometheus_remote_storage_highest_timestamp_in_seconds", - }), - SamplesTotal: prometheus.NewCounter(prometheus.CounterOpts{ - Name: "prometheus_remote_storage_samples_total", - Help: "Total number of samples sent to remote storage.", - }), - HistogramsTotal: prometheus.NewCounter(prometheus.CounterOpts{ - Name: "prometheus_remote_storage_histograms_total", - Help: "Total number of histograms sent to remote storage.", - }), - MetadataTotal: prometheus.NewCounter(prometheus.CounterOpts{ - Name: "prometheus_remote_storage_metadata_total", - Help: "Total number of metadata sent to remote storage.", - }), - FailedSamplesTotal: prometheus.NewCounter(prometheus.CounterOpts{ - Name: "prometheus_remote_storage_samples_failed_total", - Help: "Total number of samples which failed on send to remote storage, non-recoverable errors.", - }), - FailedHistogramsTotal: prometheus.NewCounter(prometheus.CounterOpts{ - Name: "prometheus_remote_storage_histograms_failed_total", - Help: "Total number of histograms which failed on send to remote storage, non-recoverable errors.", - }), - FailedMetadataTotal: prometheus.NewCounter(prometheus.CounterOpts{ - Name: "prometheus_remote_storage_metadata_failed_total", - Help: "Total number of metadata entries which failed on send to remote storage, non-recoverable errors.", - }), - - RetriedSamplesTotal: prometheus.NewCounter(prometheus.CounterOpts{ - Name: "prometheus_remote_storage_samples_retried_total", - Help: "Total number of samples which failed on send to remote storage but were retried because the send error was recoverable.", - }), - RetriedHistogramsTotal: prometheus.NewCounter(prometheus.CounterOpts{ - Name: "prometheus_remote_storage_histograms_retried_total", - Help: "Total number of histograms which failed on send to remote storage but were retried because the send error was recoverable.", - }), - RetriedMetadataTotal: prometheus.NewCounter(prometheus.CounterOpts{ - Name: "prometheus_remote_storage_metadata_retried_total", - Help: "Total number of metadata entries which failed on send to remote storage but were retried because the send error was recoverable.", - }), - SentBytesTotal: prometheus.NewCounter(prometheus.CounterOpts{ - Name: "prometheus_remote_storage_sent_bytes_total", - Help: "The total number of bytes of data (not metadata) sent by the queue after compression. Note that when exemplars over remote write is enabled the exemplars included in a remote write request count towards this metric.", - }), - MetadataBytesTotal: prometheus.NewCounter(prometheus.CounterOpts{ - Name: "prometheus_remote_storage_metadata_bytes_total", - Help: "The total number of bytes of metadata sent by the queue after compression.", - }), - } - registry.MustRegister( - s.NetworkSentDuration, - s.NetworkRetries5XX, - s.NetworkRetries429, - s.NetworkRetries, - s.NetworkFailures, - s.NetworkSeriesSent, - s.NetworkErrors, - s.NetworkNewestOutTimeStampSeconds, - s.SerializerInSeries, - s.SerializerErrors, - s.SerializerNewestInTimeStampSeconds, - ) - return s -} - -func (s *PrometheusStats) SeriesBackwardsCompatibility(registry prometheus.Registerer) { - registry.MustRegister( - s.RemoteStorageDuration, - s.RemoteStorageInTimestamp, - s.RemoteStorageOutTimestamp, - s.SamplesTotal, - s.HistogramsTotal, - s.FailedSamplesTotal, - s.FailedHistogramsTotal, - s.RetriedSamplesTotal, - s.RetriedHistogramsTotal, - s.SentBytesTotal, - ) -} - -func (s *PrometheusStats) MetaBackwardsCompatibility(registry prometheus.Registerer) { - registry.MustRegister( - s.MetadataTotal, - s.FailedMetadataTotal, - s.RetriedMetadataTotal, - s.MetadataBytesTotal, - ) -} - -func (s *PrometheusStats) UpdateNetwork(stats NetworkStats) { - s.NetworkSeriesSent.Add(float64(stats.TotalSent())) - s.NetworkRetries.Add(float64(stats.TotalRetried())) - s.NetworkFailures.Add(float64(stats.TotalFailed())) - s.NetworkRetries429.Add(float64(stats.Total429())) - s.NetworkRetries5XX.Add(float64(stats.Total5XX())) - s.NetworkSentDuration.Observe(stats.SendDuration.Seconds()) - s.RemoteStorageDuration.Observe(stats.SendDuration.Seconds()) - // The newest timestamp is no always sent. - if stats.NewestTimestamp != 0 { - s.RemoteStorageOutTimestamp.Set(float64(stats.NewestTimestamp)) - s.NetworkNewestOutTimeStampSeconds.Set(float64(stats.NewestTimestamp)) - } - - s.SamplesTotal.Add(float64(stats.Series.SeriesSent)) - s.MetadataTotal.Add(float64(stats.Metadata.SeriesSent)) - s.HistogramsTotal.Add(float64(stats.Histogram.SeriesSent)) - - s.FailedSamplesTotal.Add(float64(stats.Series.FailedSamples)) - s.FailedMetadataTotal.Add(float64(stats.Metadata.FailedSamples)) - s.FailedHistogramsTotal.Add(float64(stats.Histogram.FailedSamples)) - - s.RetriedSamplesTotal.Add(float64(stats.Series.RetriedSamples)) - s.RetriedHistogramsTotal.Add(float64(stats.Histogram.RetriedSamples)) - s.RetriedMetadataTotal.Add(float64(stats.Metadata.RetriedSamples)) - - s.MetadataBytesTotal.Add(float64(stats.MetadataBytes)) - s.SentBytesTotal.Add(float64(stats.SeriesBytes)) -} - -func (s *PrometheusStats) UpdateSerializer(stats SerializerStats) { - s.SerializerInSeries.Add(float64(stats.SeriesStored)) - s.SerializerInSeries.Add(float64(stats.MetadataStored)) - s.SerializerErrors.Add(float64(stats.Errors)) - if stats.NewestTimestamp != 0 { - s.SerializerNewestInTimeStampSeconds.Set(float64(stats.NewestTimestamp)) - s.RemoteStorageInTimestamp.Set(float64(stats.NewestTimestamp)) - } - -} - -type NetworkStats struct { - Series CategoryStats - Histogram CategoryStats - Metadata CategoryStats - SendDuration time.Duration - NewestTimestamp int64 - SeriesBytes int - MetadataBytes int -} - -func (ns NetworkStats) TotalSent() int { - return ns.Series.SeriesSent + ns.Histogram.SeriesSent + ns.Metadata.SeriesSent -} - -func (ns NetworkStats) TotalRetried() int { - return ns.Series.RetriedSamples + ns.Histogram.RetriedSamples + ns.Metadata.RetriedSamples -} - -func (ns NetworkStats) TotalFailed() int { - return ns.Series.FailedSamples + ns.Histogram.FailedSamples + ns.Metadata.FailedSamples -} - -func (ns NetworkStats) Total429() int { - return ns.Series.RetriedSamples429 + ns.Histogram.RetriedSamples429 + ns.Metadata.RetriedSamples429 -} - -func (ns NetworkStats) Total5XX() int { - return ns.Series.RetriedSamples5XX + ns.Histogram.RetriedSamples5XX + ns.Metadata.RetriedSamples5XX -} - -type CategoryStats struct { - RetriedSamples int - RetriedSamples429 int - RetriedSamples5XX int - SeriesSent int - FailedSamples int - NetworkSamplesFailed int -} diff --git a/internal/component/prometheus/write/queue/types/storage.go b/internal/component/prometheus/write/queue/types/storage.go deleted file mode 100644 index 6fe262ab46..0000000000 --- a/internal/component/prometheus/write/queue/types/storage.go +++ /dev/null @@ -1,11 +0,0 @@ -package types - -import ( - "context" -) - -type FileStorage interface { - Start() - Stop() - Store(ctx context.Context, meta map[string]string, value []byte) error -} diff --git a/internal/component/prometheus/write/queue/types/storage_test.go b/internal/component/prometheus/write/queue/types/storage_test.go deleted file mode 100644 index 4b58550601..0000000000 --- a/internal/component/prometheus/write/queue/types/storage_test.go +++ /dev/null @@ -1,24 +0,0 @@ -package types - -import ( - "testing" - - "github.com/prometheus/prometheus/model/labels" - "github.com/stretchr/testify/require" -) - -func TestStorage(t *testing.T) { - ts := GetTimeSeriesFromPool() - ts.Labels = labels.FromStrings("one", "two") - ts.LabelsValues = make([]uint32, 1) - ts.LabelsNames = make([]uint32, 1) - ts.LabelsValues[0] = 1 - ts.LabelsNames[0] = 2 - - PutTimeSeriesIntoPool(ts) - ts = GetTimeSeriesFromPool() - defer PutTimeSeriesIntoPool(ts) - require.Len(t, ts.Labels, 0) - require.Len(t, ts.LabelsValues, 0) - require.Len(t, ts.LabelsNames, 0) -} From c01bfa80ef9515d339dc21e428bde796214e457c Mon Sep 17 00:00:00 2001 From: Sam DeHaan Date: Fri, 15 Nov 2024 09:29:48 -0500 Subject: [PATCH 20/32] Capture second metrics sample in support bundle to provide metrics delta for investigating issues (#2085) * Capture second metrics sample to provide metrics delta for investigating issues * Update names of metrics samples --- CHANGELOG.md | 4 + docs/sources/troubleshoot/support_bundle.md | 3 +- internal/service/http/supportbundle.go | 109 +++++++++++--------- 3 files changed, 66 insertions(+), 50 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 99a49d0eb2..69682dd19c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,10 @@ Main (unreleased) - Add `otelcol.receiver.solace` component to receive traces from a Solace broker. (@wildum) +### Enhancements + +- Add second metrics sample to the support bundle to provide delta information (@dehaansa) + ### Bugfixes - Fixed an issue in the `prometheus.exporter.postgres` component that would leak goroutines when the target was not reachable (@dehaansa) diff --git a/docs/sources/troubleshoot/support_bundle.md b/docs/sources/troubleshoot/support_bundle.md index 2bb870bc5b..d38c9dd41b 100644 --- a/docs/sources/troubleshoot/support_bundle.md +++ b/docs/sources/troubleshoot/support_bundle.md @@ -38,7 +38,8 @@ A support bundle contains the following data: `/api/v0/web/components` endpoint. * `alloy-logs.txt` contains the logs during the bundle generation. * `alloy-metadata.yaml` contains the {{< param "PRODUCT_NAME" >}} build version and the installation's operating system, architecture, and uptime. -* `alloy-metrics.txt` contains a snapshot of the internal metrics for {{< param "PRODUCT_NAME" >}}. +* `alloy-metrics-sample-start.txt` contains a snapshot of the internal metrics for {{< param "PRODUCT_NAME" >}} at the start of the bundle collection. +* `alloy-metrics-sample-end.txt` contains a snapshot of the internal metrics for {{< param "PRODUCT_NAME" >}} at the end of the bundle collection. * `alloy-peers.json` contains information about the identified cluster peers of this {{< param "PRODUCT_NAME" >}} instance, generated by the `/api/v0/web/peers` endpoint. * `alloy-runtime-flags.txt` contains the values of the runtime flags available in {{< param "PRODUCT_NAME" >}}. diff --git a/internal/service/http/supportbundle.go b/internal/service/http/supportbundle.go index 3c75c35150..ac0898ce5a 100644 --- a/internal/service/http/supportbundle.go +++ b/internal/service/http/supportbundle.go @@ -28,16 +28,17 @@ type SupportBundleContext struct { // Bundle collects all the data that is exposed as a support bundle. type Bundle struct { - meta []byte - alloyMetrics []byte - components []byte - peers []byte - runtimeFlags []byte - heapBuf *bytes.Buffer - goroutineBuf *bytes.Buffer - blockBuf *bytes.Buffer - mutexBuf *bytes.Buffer - cpuBuf *bytes.Buffer + meta []byte + alloyMetricsStart []byte + alloyMetricsEnd []byte + components []byte + peers []byte + runtimeFlags []byte + heapBuf *bytes.Buffer + goroutineBuf *bytes.Buffer + blockBuf *bytes.Buffer + mutexBuf *bytes.Buffer + cpuBuf *bytes.Buffer } // Metadata contains general runtime information about the current Alloy environment. @@ -50,6 +51,26 @@ type Metadata struct { // ExportSupportBundle gathers the information required for the support bundle. func ExportSupportBundle(ctx context.Context, runtimeFlags []string, srvAddress string, dialContext server.DialContextFunc) (*Bundle, error) { + var httpClient http.Client + httpClient.Transport = &http.Transport{DialContext: dialContext} + + // Gather Alloy's own metrics. + alloyMetricsStart, err := retrieveAPIEndpoint(httpClient, srvAddress, "metrics") + if err != nil { + return nil, fmt.Errorf("failed to get internal Alloy metrics: %s", err) + } + + // Gather running component configuration + components, err := retrieveAPIEndpoint(httpClient, srvAddress, "api/v0/web/components") + if err != nil { + return nil, fmt.Errorf("failed to get component details: %s", err) + } + // Gather cluster peers information + peers, err := retrieveAPIEndpoint(httpClient, srvAddress, "api/v0/web/peers") + if err != nil { + return nil, fmt.Errorf("failed to get peer details: %s", err) + } + // The block profiler is disabled by default. Temporarily enable recording // of all blocking events. Also, temporarily record all mutex contentions, // and defer restoring of earlier mutex profiling fraction. @@ -76,24 +97,6 @@ func ExportSupportBundle(ctx context.Context, runtimeFlags []string, srvAddress return nil, fmt.Errorf("failed to marshal support bundle metadata: %s", err) } - var httpClient http.Client - httpClient.Transport = &http.Transport{DialContext: dialContext} - // Gather Alloy's own metrics. - alloyMetrics, err := retrieveAPIEndpoint(httpClient, srvAddress, "metrics") - if err != nil { - return nil, fmt.Errorf("failed to get internal Alloy metrics: %s", err) - } - // Gather running component configuration - components, err := retrieveAPIEndpoint(httpClient, srvAddress, "api/v0/web/components") - if err != nil { - return nil, fmt.Errorf("failed to get component details: %s", err) - } - // Gather cluster peers information - peers, err := retrieveAPIEndpoint(httpClient, srvAddress, "api/v0/web/peers") - if err != nil { - return nil, fmt.Errorf("failed to get peer details: %s", err) - } - // Export pprof data. var ( cpuBuf bytes.Buffer @@ -129,19 +132,26 @@ func ExportSupportBundle(ctx context.Context, runtimeFlags []string, srvAddress return nil, err } + // Gather Alloy's own metrics after the profile completes + alloyMetricsEnd, err := retrieveAPIEndpoint(httpClient, srvAddress, "metrics") + if err != nil { + return nil, fmt.Errorf("failed to get internal Alloy metrics: %s", err) + } + // Finally, bundle everything up to be served, either as a zip from // memory, or exported to a directory. bundle := &Bundle{ - meta: meta, - alloyMetrics: alloyMetrics, - components: components, - peers: peers, - runtimeFlags: []byte(strings.Join(runtimeFlags, "\n")), - heapBuf: &heapBuf, - goroutineBuf: &goroutineBuf, - blockBuf: &blockBuf, - mutexBuf: &mutexBuf, - cpuBuf: &cpuBuf, + meta: meta, + alloyMetricsStart: alloyMetricsStart, + alloyMetricsEnd: alloyMetricsEnd, + components: components, + peers: peers, + runtimeFlags: []byte(strings.Join(runtimeFlags, "\n")), + heapBuf: &heapBuf, + goroutineBuf: &goroutineBuf, + blockBuf: &blockBuf, + mutexBuf: &mutexBuf, + cpuBuf: &cpuBuf, } return bundle, nil @@ -169,17 +179,18 @@ func ServeSupportBundle(rw http.ResponseWriter, b *Bundle, logsBuf *bytes.Buffer rw.Header().Set("Content-Disposition", "attachment; filename=\"alloy-support-bundle.zip\"") zipStructure := map[string][]byte{ - "alloy-metadata.yaml": b.meta, - "alloy-components.json": b.components, - "alloy-peers.json": b.peers, - "alloy-metrics.txt": b.alloyMetrics, - "alloy-runtime-flags.txt": b.runtimeFlags, - "alloy-logs.txt": logsBuf.Bytes(), - "pprof/cpu.pprof": b.cpuBuf.Bytes(), - "pprof/heap.pprof": b.heapBuf.Bytes(), - "pprof/goroutine.pprof": b.goroutineBuf.Bytes(), - "pprof/mutex.pprof": b.mutexBuf.Bytes(), - "pprof/block.pprof": b.blockBuf.Bytes(), + "alloy-metadata.yaml": b.meta, + "alloy-components.json": b.components, + "alloy-peers.json": b.peers, + "alloy-metrics-sample-start.txt": b.alloyMetricsStart, + "alloy-metrics-sample-end.txt": b.alloyMetricsEnd, + "alloy-runtime-flags.txt": b.runtimeFlags, + "alloy-logs.txt": logsBuf.Bytes(), + "pprof/cpu.pprof": b.cpuBuf.Bytes(), + "pprof/heap.pprof": b.heapBuf.Bytes(), + "pprof/goroutine.pprof": b.goroutineBuf.Bytes(), + "pprof/mutex.pprof": b.mutexBuf.Bytes(), + "pprof/block.pprof": b.blockBuf.Bytes(), } for fn, b := range zipStructure { From 57623e7af77537958a799d47f5364b33f99fa2b1 Mon Sep 17 00:00:00 2001 From: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> Date: Fri, 15 Nov 2024 10:26:34 -0800 Subject: [PATCH 21/32] Add a topic explaining Amazon ECS and AWS Fargate telemetry collection (#1466) * First draft of an ECS collect topic * First draft of an ECS collect topic * Initiall restructing of the topic * more general cleanup and alignment * Update heading levels * Cleaning up the AWS steps * Update text and wording * Apply suggestions from code review * More content fixes and cleanup * fix up some links * Fix typos --- docs/sources/collect/ecs-openteletry-data.md | 102 +++++++++++++++++++ 1 file changed, 102 insertions(+) create mode 100644 docs/sources/collect/ecs-openteletry-data.md diff --git a/docs/sources/collect/ecs-openteletry-data.md b/docs/sources/collect/ecs-openteletry-data.md new file mode 100644 index 0000000000..1a68d1578d --- /dev/null +++ b/docs/sources/collect/ecs-openteletry-data.md @@ -0,0 +1,102 @@ +--- +canonical: https://grafana.com/docs/alloy/latest/collect/ecs-opentelemetry-data/ +description: Learn how to collect Amazon ECS or AWS Fargate OpenTelemetry data and forward it to any OpenTelemetry-compatible endpoint +menuTitle: Collect ECS or Fargate OpenTelemetry data +title: Collect Amazon Elastic Container Service or AWS Fargate OpenTelemetry data +weight: 500 +--- + +# Collect Amazon Elastic Container Service or AWS Fargate OpenTelemetry data + +You can configure {{< param "FULL_PRODUCT_NAME" >}} to collect OpenTelemetry-compatible data from Amazon Elastic Container Service (ECS) or AWS Fargate and forward it to any OpenTelemetry-compatible endpoint. + +There are three different ways you can use {{< param "PRODUCT_NAME" >}} to collect Amazon ECS or AWS Fargate telemetry data. + +1. [Use a custom OpenTelemetry configuration file from the SSM Parameter store](#use-a-custom-opentelemetry-configuration-file-from-the-ssm-parameter-store). +1. [Create an ECS task definition](#create-an-ecs-task-definition). +1. [Run {{< param "PRODUCT_NAME" >}} directly in your instance, or as a Kubernetes sidecar](#run-alloy-directly-in-your-instance-or-as-a-kubernetes-sidecar). + +## Before you begin + +* Ensure that you have basic familiarity with instrumenting applications with OpenTelemetry. +* Have an available Amazon ECS or AWS Fargate deployment. +* Identify where {{< param "PRODUCT_NAME" >}} writes received telemetry data. +* Be familiar with the concept of [Components][] in {{< param "PRODUCT_NAME" >}}. + +## Use a custom OpenTelemetry configuration file from the SSM Parameter store + +You can upload a custom OpenTelemetry configuration file to the SSM Parameter store and use {{< param "PRODUCT_NAME" >}} as a telemetry data collector. + +You can configure the AWS Distro for OpenTelemetry Collector with the `AOT_CONFIG_CONTENT` environment variable. +This environment variable contains a full collector configuration file and it overrides the configuration file used in the collector entry point command. +In ECS, you can set the values of environment variables from AWS Systems Manager Parameters. + +### Update the task definition + + 1.Select the task definition. + + 1. Open the AWS Systems Manager console. + 1. Select Elastic Container Service. + 1. In the navigation pane, choose *Task definition*. + 1. Choose *Create new revision*. + +1. Add an environment variable. + + 1. Select the AWS Distro for OpenTelemetry Collector container and navigate to the Environment variables section. + 1. Add an environment variable named `AOT_CONFIG_CONTENT`. + 1. Select `ValueFrom` to tell ECS to get the value from the SSM Parameter, and set the value to `otel-collector-config`. + +1. Finish updating the task definition and create your revision. + +### Create the SSM parameter + +1. Open the AWS Systems Manager console. +1. In the navigation pane, choose *Parameter Store*. +1. Choose *Create parameter*. +1. Create a parameter with the following values: + + * `Name`: otel-collector-config + * `Tier`: Standard + * `Type`: String + * `Data type`: Text + * `Value`: Copy and paste your custom OpenTelemetry configuration file or [{{< param "PRODUCT_NAME" >}} configuration file][configure]. + +### Run your task + +When you run a task with this Task Definition, it uses your custom OpenTelemetry configuration file from the SSM Parameter store. + +Refer to [Running an application as an Amazon ECS task][run] for more information about running your application as a task. + +## Create an ECS Task definition + +To create an ECS Task Definition for AWS Fargate with an ADOT collector, complete the following steps. + +1. Download the [ECS Fargate task definition template][template] from GitHub. +1. Edit the task definition template and add the following parameters. + * `{{region}}`: The region the data is sent to. + * `{{ecsTaskRoleArn}}`: The AWSOTTaskRole ARN. + * `{{ecsExecutionRoleArn}}`: The AWSOTTaskExcutionRole ARN. + * `command` - Assign a value to the command variable to select the path to the configuration file. + The AWS Collector comes with two configurations. Select one of them based on your environment: + * Use `--config=/etc/ecs/ecs-default-config.yaml` to consume StatsD metrics, OTLP metrics and traces, and X-Ray SDK traces. + * Use `--config=/etc/ecs/container-insights/otel-task-metrics-config.yaml` to use StatsD, OTLP, Xray, and Container Resource utilization metrics. +1. Follow the ECS Fargate setup instructions to [create a task definition][task] using the template. + +## Run {{% param "PRODUCT_NAME" %}} directly in your instance, or as a Kubernetes sidecar + +SSH or connect to the Amazon ECS or AWS Fargate-managed container. Refer to [9 steps to SSH into an AWS Fargate managed container][steps] for more information about using SSH with Amazon ECS or AWS Fargate. + +You can also use your own method to connect to the Amazon ECS or AWS Fargate-managed container as long as you can pass the parameters needed to install and configure {{< param "PRODUCT_NAME" >}}. + +### Install Grafana Alloy + +After connecting to your instance, follow the {{< param "PRODUCT_NAME" >}} [installation][install], [configuration][configure] and [deployment][deploy] instructions. + +[Components]: https://grafana.com/docs/alloy//get-started/components +[template]: https://github.com/aws-observability/aws-otel-collector/blob/master/examples/ecs/aws-cloudwatch/ecs-fargate-sidecar.json +[configure]: https://grafana.com/docs/alloy//configure/ +[steps]: https://medium.com/ci-t/9-steps-to-ssh-into-an-aws-fargate-managed-container-46c1d5f834e2 +[install]: https://grafana.com/docs/alloy//set-up/install/linux/ +[deploy]: https://grafana.com/docs/alloy//set-up/deploy/ +[task]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html +[run]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/standalone-task-create.html From 2eecbfe56cead4dc1978bc3adef507945fbff06d Mon Sep 17 00:00:00 2001 From: Bob Cotton Date: Fri, 15 Nov 2024 12:19:24 -0700 Subject: [PATCH 22/32] Update local.file.md (#2072) * Update local.file.md when trying to use `local.file` it seemed to be missing an example of how to use it. * Update docs/sources/reference/components/local/local.file.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/local/local.file.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Fixed the workding of the example --------- Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> --- docs/sources/reference/components/local/local.file.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/sources/reference/components/local/local.file.md b/docs/sources/reference/components/local/local.file.md index 392b23fceb..08edc29f9b 100644 --- a/docs/sources/reference/components/local/local.file.md +++ b/docs/sources/reference/components/local/local.file.md @@ -46,7 +46,9 @@ Name | Type | Description ----------|----------------------|--------------------------------------------------- `content` | `string` or `secret` | The contents of the file from the most recent read -The `content` field will have the `secret` type only if the `is_secret` argument was true. +The `content` field will have the `secret` type only if the `is_secret` argument was true. + +You can use `local.file.LABEL.content` to access the contents of the file. ## Component health @@ -66,9 +68,15 @@ The read error will be exposed as a log message and in the debug information for ## Example +The following example shows a simple `local.file` configuration that watches a passwords text file and uses the exported content field. + ```alloy local.file "secret_key" { filename = "/var/secrets/password.txt" is_secret = true } +grafana_cloud.stack "receivers" { + stack_name = "mystack" + token = local.file.secret_key.content +} ``` From e3b134bbff908e66f6371ad81bb5b79f2f5227f2 Mon Sep 17 00:00:00 2001 From: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> Date: Fri, 15 Nov 2024 11:27:18 -0800 Subject: [PATCH 23/32] Fix indentation on steps (#2108) --- docs/sources/collect/ecs-openteletry-data.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/sources/collect/ecs-openteletry-data.md b/docs/sources/collect/ecs-openteletry-data.md index 1a68d1578d..3a7a53a483 100644 --- a/docs/sources/collect/ecs-openteletry-data.md +++ b/docs/sources/collect/ecs-openteletry-data.md @@ -33,12 +33,12 @@ In ECS, you can set the values of environment variables from AWS Systems Manager ### Update the task definition - 1.Select the task definition. +1. Select the task definition. - 1. Open the AWS Systems Manager console. - 1. Select Elastic Container Service. - 1. In the navigation pane, choose *Task definition*. - 1. Choose *Create new revision*. + 1. Open the AWS Systems Manager console. + 1. Select Elastic Container Service. + 1. In the navigation pane, choose *Task definition*. + 1. Choose *Create new revision*. 1. Add an environment variable. From df34fdb6e727a01a05337900e51c1b7e1ebc5670 Mon Sep 17 00:00:00 2001 From: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> Date: Fri, 15 Nov 2024 16:08:43 -0800 Subject: [PATCH 24/32] Fix typo in vars block (#2116) --- docs/sources/set-up/install/ansible.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sources/set-up/install/ansible.md b/docs/sources/set-up/install/ansible.md index 8ef90561b9..aad2d5c7f2 100644 --- a/docs/sources/set-up/install/ansible.md +++ b/docs/sources/set-up/install/ansible.md @@ -33,7 +33,7 @@ To add {{% param "PRODUCT_NAME" %}} to a host: ansible.builtin.include_role: name: grafana.grafana.alloy vars: - config: | + alloy_config: | prometheus.scrape "default" { targets = [{"__address__" = "localhost:12345"}] forward_to = [prometheus.remote_write.prom.receiver] From 9c04d735219fe19dab06e38c19111e43b7a41c43 Mon Sep 17 00:00:00 2001 From: mattdurham Date: Mon, 18 Nov 2024 10:04:28 -0500 Subject: [PATCH 25/32] Update wal doc (#1917) * Docs changes. * Clean up tables * True up the documentation with the code. --- .../prometheus/prometheus.write.queue.md | 51 ++++++++++--------- 1 file changed, 27 insertions(+), 24 deletions(-) diff --git a/docs/sources/reference/components/prometheus/prometheus.write.queue.md b/docs/sources/reference/components/prometheus/prometheus.write.queue.md index f958956a42..5c1c04653d 100644 --- a/docs/sources/reference/components/prometheus/prometheus.write.queue.md +++ b/docs/sources/reference/components/prometheus/prometheus.write.queue.md @@ -39,20 +39,20 @@ prometheus.write.queue "LABEL" { The following arguments are supported: -Name | Type | Description | Default | Required ----- | ---- | ----------- | ------- | -------- -`ttl` | `time` | `duration` | How long the samples can be queued for before they are discarded. | `2h` | no + Name | Type | Description | Default | Required +-------|--------|-------------|-------------------------------------------------------------------|---------- + `ttl` | `time` | `duration` | How long the samples can be queued for before they are discarded. | `2h` | no ## Blocks The following blocks are supported inside the definition of `prometheus.write.queue`: -Hierarchy | Block | Description | Required ---------- | ----- | ----------- | -------- -persistence | [persistence][] | Configuration for persistence | no -endpoint | [endpoint][] | Location to send metrics to. | no -endpoint > basic_auth | [basic_auth][] | Configure basic_auth for authenticating to the endpoint. | no + Hierarchy | Block | Description | Required +-----------------------|-----------------|----------------------------------------------------------|---------- + persistence | [persistence][] | Configuration for persistence | no + endpoint | [endpoint][] | Location to send metrics to. | no + endpoint > basic_auth | [basic_auth][] | Configure basic_auth for authenticating to the endpoint. | no The `>` symbol indicates deeper levels of nesting. For example, `endpoint > basic_auth` refers to a `basic_auth` block defined inside an @@ -69,10 +69,10 @@ are shared for each `endpoint`. The following arguments are supported: -Name | Type | Description | Default | Required ----- | ---- |-------------------------------------------------------------------------------|---------| -------- -`max_signals_to_batch` | `uint` | The maximum number of signals before they are batched to disk. | `10000` | no -`batch_interval` | `duration` | How often to batch signals to disk if `max_signals_to_batch` is not reached. | `5s` | no + Name | Type | Description | Default | Required +------------------------|------------|------------------------------------------------------------------------------|---------|---------- + `max_signals_to_batch` | `uint` | The maximum number of signals before they are batched to disk. | `10000` | no + `batch_interval` | `duration` | How often to batch signals to disk if `max_signals_to_batch` is not reached. | `5s` | no ### endpoint block @@ -83,21 +83,24 @@ The `endpoint` block describes a single location to send metrics to. Multiple The following arguments are supported: -Name | Type | Description | Default | Required ----- | ---- |--------------------------------------------------------------------| ------ | -------- -`url` | `string` | Full URL to send metrics to. | | yes -`bearer_token` | `secret` | Bearer token to authenticate with. | | no -`write_timeout` | `duration` | Timeout for requests made to the URL. | `"30s"` | no -`retry_backoff` | `duration` | How often to wait between retries. | `1s` | no -`max_retry_attempts` | Maximum number of retries before dropping the batch. | `0` | no -`batch_count` | `uint` | How many series to queue in each queue. | `1000` | no -`flush_interval` | `duration` | How often to wait until sending if `batch_count` is not triggered. | `1s` | no -`parallelism` | `uint` | How many parallel batches to write. | 10 | no -`external_labels` | `map(string)` | Labels to add to metrics sent over the network. | | no + Name | Type | Description | Default | Required +----------------------|---------------|-----------------------------------------------------------------|---------|---------- + `url` | `string` | Full URL to send metrics to. | | yes +`bearer_token` | `secret` | Bearer token to authenticate with. | | no + `write_timeout` | `duration` | Timeout for requests made to the URL. | `"30s"` | no + `retry_backoff` | `duration` | How long to wait between retries. | `1s` | no + `max_retry_attempts` | `uint` | Maximum number of retries before dropping the batch. | `0` | no + `batch_count` | `uint` | How many series to queue in each queue. | `1000` | no + `flush_interval` | `duration` | How long to wait until sending if `batch_count` is not trigger. | `1s` | no + `parallelism` | `uint` | How many parallel batches to write. | 10 | no + `external_labels` | `map(string)` | Labels to add to metrics sent over the network. | | no ### basic_auth block -{{< docs/shared lookup="reference/components/basic-auth-block.md" source="alloy" version="" >}} +Name | Type | Description | Default | Required +----------------|----------|------------------------------------------|---------|--------- +`password` | `secret` | Basic auth password. | | no +`username` | `string` | Basic auth username. | | no ## Exported fields From 1ba29eaf1c118d5ae1e1270e4c46dc784fbef6d8 Mon Sep 17 00:00:00 2001 From: William Dumont Date: Mon, 18 Nov 2024 16:55:42 +0100 Subject: [PATCH 26/32] fix key/pattern logic for the attribute processor (#2124) --- CHANGELOG.md | 2 + .../component/otelcol/config_attraction.go | 46 +++++++++++++++- .../otelcol/config_attraction_test.go | 54 +++++++++++++++++++ .../processor/attributes/attributes.go | 4 ++ .../processor/attributes/attributes_test.go | 17 ++++++ 5 files changed, 121 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 69682dd19c..ea9f8dd8ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,8 @@ Main (unreleased) - Fixed issue with reloading configuration and prometheus metrics duplication in `prometheus.write.queue`. (@mattdurham) +- Fixed an issue in the `otelcol.processor.attribute` component where the actions `delete` and `hash` could not be used with the `pattern` argument. (@wildum) + ### Other changes - Change the stability of the `livedebugging` feature from "experimental" to "generally available". (@wildum) diff --git a/internal/component/otelcol/config_attraction.go b/internal/component/otelcol/config_attraction.go index b009397112..b6c8b4082c 100644 --- a/internal/component/otelcol/config_attraction.go +++ b/internal/component/otelcol/config_attraction.go @@ -1,5 +1,16 @@ package otelcol +import ( + "errors" + "fmt" + "strings" +) + +const ( + delete = "delete" + hash = "hash" +) + type AttrActionKeyValueSlice []AttrActionKeyValue func (actions AttrActionKeyValueSlice) Convert() []interface{} { @@ -15,10 +26,27 @@ func (actions AttrActionKeyValueSlice) Convert() []interface{} { return res } +func (actions AttrActionKeyValueSlice) Validate() error { + var validationErrors []error + + for i, action := range actions { + if err := action.validate(); err != nil { + wrappedErr := fmt.Errorf("validation failed for action block number %d: %w", i+1, err) + validationErrors = append(validationErrors, wrappedErr) + } + } + + if len(validationErrors) > 0 { + return errors.Join(validationErrors...) + } + return nil +} + type AttrActionKeyValue struct { // Key specifies the attribute to act upon. - // This is a required field. - Key string `alloy:"key,attr"` + // The actions `delete` and `hash` can use the `pattern`` argument instead of/with the `key` argument. + // The field is required for all other actions. + Key string `alloy:"key,attr,optional"` // Value specifies the value to populate for the key. // The type of the value is inferred from the configuration. @@ -91,3 +119,17 @@ func (args *AttrActionKeyValue) convert() map[string]interface{} { "converted_type": args.ConvertedType, } } + +func (args *AttrActionKeyValue) validate() error { + switch strings.ToLower(args.Action) { + case delete, hash: + if args.Key == "" && args.RegexPattern == "" { + return fmt.Errorf("the action %s requires at least the key argument or the pattern argument to be set", args.Action) + } + default: + if args.Key == "" { + return fmt.Errorf("the action %s requires the key argument to be set", args.Action) + } + } + return nil +} diff --git a/internal/component/otelcol/config_attraction_test.go b/internal/component/otelcol/config_attraction_test.go index 5879bde815..3e6655b03f 100644 --- a/internal/component/otelcol/config_attraction_test.go +++ b/internal/component/otelcol/config_attraction_test.go @@ -1,6 +1,7 @@ package otelcol_test import ( + "strings" "testing" "github.com/grafana/alloy/internal/component/otelcol" @@ -58,3 +59,56 @@ func TestConvertAttrAction(t *testing.T) { result := inputActions.Convert() require.Equal(t, expectedActions, result) } + +func TestValidateAttrAction(t *testing.T) { + inputActions := otelcol.AttrActionKeyValueSlice{ + { + // ok - only key + Action: "insert", + Value: 123, + Key: "attribute1", + }, + { + // not ok - missing key + Action: "insert", + Value: 123, + RegexPattern: "pattern", // pattern is useless here + }, + { + // ok - only key + Action: "delete", + Key: "key", + }, + { + // ok - only pattern + Action: "delete", + RegexPattern: "pattern", + }, + { + // ok - both + Action: "delete", + Key: "key", + RegexPattern: "pattern", + }, + { + // not ok - missing key and pattern + Action: "delete", + }, + { + // ok - only pattern + Action: "hash", + RegexPattern: "pattern", + }, + { + // ok - with uppercase + Action: "HaSH", + RegexPattern: "pattern", + }, + } + + expectedErrors := []string{ + "validation failed for action block number 2: the action insert requires the key argument to be set", + "validation failed for action block number 6: the action delete requires at least the key argument or the pattern argument to be set", + } + require.EqualError(t, inputActions.Validate(), strings.Join(expectedErrors, "\n")) +} diff --git a/internal/component/otelcol/processor/attributes/attributes.go b/internal/component/otelcol/processor/attributes/attributes.go index 48495b3152..4301f77927 100644 --- a/internal/component/otelcol/processor/attributes/attributes.go +++ b/internal/component/otelcol/processor/attributes/attributes.go @@ -55,6 +55,10 @@ func (args *Arguments) SetToDefault() { args.DebugMetrics.SetToDefault() } +func (args *Arguments) Validate() error { + return args.Actions.Validate() +} + // Convert implements processor.Arguments. func (args Arguments) Convert() (otelcomponent.Config, error) { input := make(map[string]interface{}) diff --git a/internal/component/otelcol/processor/attributes/attributes_test.go b/internal/component/otelcol/processor/attributes/attributes_test.go index e2019982cd..6412730700 100644 --- a/internal/component/otelcol/processor/attributes/attributes_test.go +++ b/internal/component/otelcol/processor/attributes/attributes_test.go @@ -134,6 +134,23 @@ func testRunProcessorWithContext(ctx context.Context, t *testing.T, processorCon processortest.TestRunProcessor(prc) } +// Test that the validate function is called. The validation logic for the actions is tested in the otelcol pkg. +func Test_Validate(t *testing.T) { + cfg := ` + action { + value = 111111 + action = "insert" + } + + output { + // no-op: will be overridden by test code. + } + ` + expectedErr := "validation failed for action block number 1: the action insert requires the key argument to be set" + var args attributes.Arguments + require.ErrorContains(t, syntax.Unmarshal([]byte(cfg), &args), expectedErr) +} + func Test_Insert(t *testing.T) { cfg := ` action { From 8349b3cdeebd7eef7a2a833c830c930863d14b8d Mon Sep 17 00:00:00 2001 From: mattdurham Date: Mon, 18 Nov 2024 13:05:37 -0500 Subject: [PATCH 27/32] Update build image to 1.23.3. (#2126) * Update build image to 1.23.3. * Update build image to 1.23.3. --- .drone/drone.yml | 6 +++--- .drone/pipelines/build_images.jsonnet | 4 ++-- .github/workflows/check-linux-build-image.yml | 4 ++-- tools/build-image/windows/Dockerfile | 2 +- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.drone/drone.yml b/.drone/drone.yml index 079294a17c..2f14b17c71 100644 --- a/.drone/drone.yml +++ b/.drone/drone.yml @@ -10,7 +10,7 @@ steps: - docker login -u $DOCKER_LOGIN -p $DOCKER_PASSWORD - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes - docker buildx create --name multiarch --driver docker-container --use - - docker buildx build --build-arg="GO_RUNTIME=golang:1.22.7-bullseye" --push --platform + - docker buildx build --build-arg="GO_RUNTIME=golang:1.23.3-bullseye" --push --platform linux/amd64,linux/arm64 -t grafana/alloy-build-image:$IMAGE_TAG ./tools/build-image environment: DOCKER_LOGIN: @@ -44,7 +44,7 @@ steps: - docker login -u $DOCKER_LOGIN -p $DOCKER_PASSWORD - docker run --rm --privileged multiarch/qemu-user-static --reset -p yes - docker buildx create --name multiarch --driver docker-container --use - - docker buildx build --build-arg="GO_RUNTIME=mcr.microsoft.com/oss/go/microsoft/golang:1.22.7-bullseye" + - docker buildx build --build-arg="GO_RUNTIME=mcr.microsoft.com/oss/go/microsoft/golang:1.23.3-bullseye" --push --platform linux/amd64,linux/arm64 -t grafana/alloy-build-image:$IMAGE_TAG ./tools/build-image environment: @@ -836,6 +836,6 @@ kind: secret name: updater_private_key --- kind: signature -hmac: fd0699c2276e04bf3f9957b8eca427f3812c0aaf17bb969760f66574dcaf250c +hmac: 4dfd6aff0a0f50f6d95e84563afe02bc941cdc2ac4447aebe2ba97492f6e7db2 ... diff --git a/.drone/pipelines/build_images.jsonnet b/.drone/pipelines/build_images.jsonnet index d8a90562ce..efd9810e2e 100644 --- a/.drone/pipelines/build_images.jsonnet +++ b/.drone/pipelines/build_images.jsonnet @@ -32,7 +32,7 @@ local locals = { 'docker login -u $DOCKER_LOGIN -p $DOCKER_PASSWORD', 'docker run --rm --privileged multiarch/qemu-user-static --reset -p yes', 'docker buildx create --name multiarch --driver docker-container --use', - 'docker buildx build --build-arg="GO_RUNTIME=golang:1.22.7-bullseye" --push --platform linux/amd64,linux/arm64 -t grafana/alloy-build-image:$IMAGE_TAG ./tools/build-image', + 'docker buildx build --build-arg="GO_RUNTIME=golang:1.23.3-bullseye" --push --platform linux/amd64,linux/arm64 -t grafana/alloy-build-image:$IMAGE_TAG ./tools/build-image', ], }], volumes: [{ @@ -55,7 +55,7 @@ local locals = { 'docker login -u $DOCKER_LOGIN -p $DOCKER_PASSWORD', 'docker run --rm --privileged multiarch/qemu-user-static --reset -p yes', 'docker buildx create --name multiarch --driver docker-container --use', - 'docker buildx build --build-arg="GO_RUNTIME=mcr.microsoft.com/oss/go/microsoft/golang:1.22.7-bullseye" --push --platform linux/amd64,linux/arm64 -t grafana/alloy-build-image:$IMAGE_TAG ./tools/build-image', + 'docker buildx build --build-arg="GO_RUNTIME=mcr.microsoft.com/oss/go/microsoft/golang:1.23.3-bullseye" --push --platform linux/amd64,linux/arm64 -t grafana/alloy-build-image:$IMAGE_TAG ./tools/build-image', ], }], volumes: [{ diff --git a/.github/workflows/check-linux-build-image.yml b/.github/workflows/check-linux-build-image.yml index 88130b6cad..9c37ab5303 100644 --- a/.github/workflows/check-linux-build-image.yml +++ b/.github/workflows/check-linux-build-image.yml @@ -25,7 +25,7 @@ jobs: push: false tags: grafana/alloy-build-image:latest build-args: | - GO_RUNTIME=golang:1.22.7-bullseye + GO_RUNTIME=golang:1.22.3-bullseye - name: Create test Linux build image for boring crypto uses: docker/build-push-action@v6 @@ -34,4 +34,4 @@ jobs: push: false tags: grafana/alloy-build-image:latest build-args: | - GO_RUNTIME=mcr.microsoft.com/oss/go/microsoft/golang:1.22.7-bullseye + GO_RUNTIME=mcr.microsoft.com/oss/go/microsoft/golang:1.23.3-bullseye diff --git a/tools/build-image/windows/Dockerfile b/tools/build-image/windows/Dockerfile index 99625c27f6..e543ceb626 100644 --- a/tools/build-image/windows/Dockerfile +++ b/tools/build-image/windows/Dockerfile @@ -1,4 +1,4 @@ -FROM library/golang:1.22.7-windowsservercore-1809 +FROM library/golang:1.23.3-windowsservercore-1809 SHELL ["powershell", "-command"] From ac24a10ae00199c55b2ce9107de63ff2f5882c24 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 19 Nov 2024 10:44:09 +0000 Subject: [PATCH 28/32] build(deps): bump cross-spawn from 7.0.3 to 7.0.6 in /internal/web/ui (#2127) Bumps [cross-spawn](https://github.com/moxystudio/node-cross-spawn) from 7.0.3 to 7.0.6. - [Changelog](https://github.com/moxystudio/node-cross-spawn/blob/master/CHANGELOG.md) - [Commits](https://github.com/moxystudio/node-cross-spawn/compare/v7.0.3...v7.0.6) --- updated-dependencies: - dependency-name: cross-spawn dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- internal/web/ui/yarn.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/internal/web/ui/yarn.lock b/internal/web/ui/yarn.lock index 026aa65ebc..404f58cca3 100644 --- a/internal/web/ui/yarn.lock +++ b/internal/web/ui/yarn.lock @@ -4455,9 +4455,9 @@ cosmiconfig@^8.1.3: path-type "^4.0.0" cross-spawn@^7.0.2, cross-spawn@^7.0.3: - version "7.0.3" - resolved "https://registry.yarnpkg.com/cross-spawn/-/cross-spawn-7.0.3.tgz#f73a85b9d5d41d045551c177e2882d4ac85728a6" - integrity sha512-iRDPJKUPVEND7dHPO8rkbOnPpyDygcDFtWjpeWNCgy8WP2rXcxXL8TskReQl6OrB2G7+UJrags1q15Fudc7G6w== + version "7.0.6" + resolved "https://registry.yarnpkg.com/cross-spawn/-/cross-spawn-7.0.6.tgz#8a58fe78f00dcd70c370451759dfbfaf03e8ee9f" + integrity sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA== dependencies: path-key "^3.1.0" shebang-command "^2.0.0" From dc9ec03eda40807b01b710930aadc581a3cfe073 Mon Sep 17 00:00:00 2001 From: YusifAghalar <41161340+YusifAghalar@users.noreply.github.com> Date: Tue, 19 Nov 2024 18:16:49 +0400 Subject: [PATCH 29/32] Fixed an issue in the `otlp.exporter.prometheus` component (#2102) * Fixed an issue in the `otlp.exporter.prometheus` component * Fixed an issue in the `otlp.exporter.prometheus` component --- CHANGELOG.md | 1 + .../otelcol/exporter/prometheus/internal/convert/convert.go | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ea9f8dd8ef..fb66cae2e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,7 @@ Main (unreleased) ### Bugfixes - Fixed an issue in the `prometheus.exporter.postgres` component that would leak goroutines when the target was not reachable (@dehaansa) +- Fixed an issue in the `otelcol.exporter.prometheus` component that would set series value incorrectly for stale metrics (@YusifAghalar) - Fixed issue with reloading configuration and prometheus metrics duplication in `prometheus.write.queue`. (@mattdurham) diff --git a/internal/component/otelcol/exporter/prometheus/internal/convert/convert.go b/internal/component/otelcol/exporter/prometheus/internal/convert/convert.go index 364f94e6ce..ad3928bd64 100644 --- a/internal/component/otelcol/exporter/prometheus/internal/convert/convert.go +++ b/internal/component/otelcol/exporter/prometheus/internal/convert/convert.go @@ -339,7 +339,7 @@ func writeSeries(app storage.Appender, series *memorySeries, dp otelcolDataPoint series.SetTimestamp(ts) if dp.Flags().NoRecordedValue() { - val = float64(value.StaleNaN) + val = math.Float64frombits(value.StaleNaN) } series.SetValue(val) From c99137c10c98001f9368944f5bb79d3bd08ce6d3 Mon Sep 17 00:00:00 2001 From: Piotr <17101802+thampiotr@users.noreply.github.com> Date: Tue, 19 Nov 2024 14:42:32 +0000 Subject: [PATCH 30/32] Fix potential deadlock in import statements (#2129) * Fix potential deadlock in import statements * change * typo --- CHANGELOG.md | 4 +++- .../runtime/internal/controller/node_config_import.go | 10 ++++++++-- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fb66cae2e3..5800cf7598 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,7 +31,9 @@ Main (unreleased) - Fixed issue with reloading configuration and prometheus metrics duplication in `prometheus.write.queue`. (@mattdurham) -- Fixed an issue in the `otelcol.processor.attribute` component where the actions `delete` and `hash` could not be used with the `pattern` argument. (@wildum) +- Fixed an issue in the `otelcol.processor.attribute` component where the actions `delete` and `hash` could not be used with the `pattern` argument. (@wildum) + +- Fixed a race condition that could lead to a deadlock when using `import` statements, which could lead to a memory leak on `/metrics` endpoint of an Alloy instance. (@thampiotr) ### Other changes diff --git a/internal/runtime/internal/controller/node_config_import.go b/internal/runtime/internal/controller/node_config_import.go index 5d6e5a200a..30a480fa5b 100644 --- a/internal/runtime/internal/controller/node_config_import.go +++ b/internal/runtime/internal/controller/node_config_import.go @@ -47,12 +47,14 @@ type ImportConfigNode struct { importChildrenUpdateChan chan struct{} // used to trigger an update of the running children + // NOTE: To avoid deadlocks, whenever we need both locks we must always first lock the mut, then healthMut. mut sync.RWMutex importedContent map[string]string importConfigNodesChildren map[string]*ImportConfigNode importChildrenRunning bool importedDeclares map[string]ast.Body + // NOTE: To avoid deadlocks, whenever we need both locks we must always first lock the mut, then healthMut. healthMut sync.RWMutex evalHealth component.Health // Health of the last source evaluation runHealth component.Health // Health of running @@ -156,10 +158,14 @@ func (cn *ImportConfigNode) setContentHealth(t component.HealthType, msg string) // 4. Health reported from the source. // 5. Health reported from the nested imports. func (cn *ImportConfigNode) CurrentHealth() component.Health { - cn.healthMut.RLock() - defer cn.healthMut.RUnlock() + // NOTE: Since other code paths such as onContentUpdate -> setContentHealth will + // also end up acquiring both of these mutexes, it's _essential_ to keep the + // order in which they're locked consistent to avoid deadlocks. We must always first + // lock the mut, then healthMut. cn.mut.RLock() defer cn.mut.RUnlock() + cn.healthMut.RLock() + defer cn.healthMut.RUnlock() health := component.LeastHealthy( cn.runHealth, From 55f76e008e6a1e2811eb7a6f13c052177d232dab Mon Sep 17 00:00:00 2001 From: William Dumont Date: Tue, 19 Nov 2024 16:38:47 +0100 Subject: [PATCH 31/32] Update updating OpenTelemetry Collector dependencies doc (#2049) * Update updating OpenTelemetry Collector dependencies doc * Update docs/developer/updating-otel/README.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/developer/updating-otel/README.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/developer/updating-otel/README.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/developer/updating-otel/README.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> --------- Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> --- docs/developer/updating-otel/README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/developer/updating-otel/README.md b/docs/developer/updating-otel/README.md index 143c78870d..ff2dfa7e60 100644 --- a/docs/developer/updating-otel/README.md +++ b/docs/developer/updating-otel/README.md @@ -95,6 +95,18 @@ Unfortunately, updating Otel dependencies is not straightforward: You can refer to [PR grafana/agent#5290](https://github.com/grafana/agent/pull/5290) for an example on how to update Alloy. +### Notify community component contributors + +You can find the community contributor GitHub handles at the top of the community component files (the components that have "Community" set to true). +Once the PR is created, you can ping them in the PR and message them on Slack. Each contributor should create a PR in your branch with the updated component. +Notify the contributors a few days before the release so they have enough time to do the update. +It should not be a blocker for the release. If the contributors can't do the update, you can fix the code to resolve the breaking changes or disable the component if that's too complicated. + +### Check for metric updates + +Some Otel metrics are used in the k8s monitoring helm chart and in integrations. +Make sure to update this [list](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring-v1/default_allow_lists/alloy_integration.yaml) if any of these metrics has been removed or renamed. + ## Testing ### Testing a tracing pipeline locally From f0211d054ab3b927d5f2d3f76b871ee28dfd60af Mon Sep 17 00:00:00 2001 From: Sam DeHaan Date: Tue, 19 Nov 2024 18:07:10 -0500 Subject: [PATCH 32/32] nit: Update otel links that didn't use OTEL_VERSION (#2131) * Update otel links that didn't use OTEL_VERSION * Update docs/sources/reference/components/otelcol/otelcol.processor.tail_sampling.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.processor.filter.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.exporter.awss3.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.exporter.awss3.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update docs/sources/reference/components/otelcol/otelcol.processor.filter.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Standardize on shortened version of using OTEL_VERSION variable, at least in filter doc --------- Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> --- .../otelcol/otelcol.exporter.awss3.md | 4 ++-- .../otelcol/otelcol.processor.filter.md | 22 +++++++++---------- .../otelcol.processor.tail_sampling.md | 2 +- 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/sources/reference/components/otelcol/otelcol.exporter.awss3.md b/docs/sources/reference/components/otelcol/otelcol.exporter.awss3.md index f405f65cfb..64422f417e 100644 --- a/docs/sources/reference/components/otelcol/otelcol.exporter.awss3.md +++ b/docs/sources/reference/components/otelcol/otelcol.exporter.awss3.md @@ -110,9 +110,9 @@ Name | Type | Description Encoding overrides the marshaler if it's present and sets it to use the encoding extension defined in the collector configuration. -Refer to the Open Telemetry [encoding extensions][] documentation for more information. +Refer to the Open Telemetry [encoding extensions][encoding] documentation for more information. -[encoding]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding +[encoding]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree//extension/encoding ### Compression diff --git a/docs/sources/reference/components/otelcol/otelcol.processor.filter.md b/docs/sources/reference/components/otelcol/otelcol.processor.filter.md index beb1031944..df75e477e4 100644 --- a/docs/sources/reference/components/otelcol/otelcol.processor.filter.md +++ b/docs/sources/reference/components/otelcol/otelcol.processor.filter.md @@ -286,17 +286,17 @@ Some values in the {{< param "PRODUCT_NAME" >}} syntax strings are [escaped][]: [escaped]: ../../../../get-started/configuration-syntax/expressions/types_and_values/#strings -[OTTL]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.85.0/pkg/ottl/README.md -[OTTL span context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/{{< param "OTEL_VERSION" >}}/pkg/ottl/contexts/ottlspan/README.md -[OTTL spanevent context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/{{< param "OTEL_VERSION" >}}/pkg/ottl/contexts/ottlspanevent/README.md -[OTTL metric context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/{{< param "OTEL_VERSION" >}}/pkg/ottl/contexts/ottlmetric/README.md -[OTTL datapoint context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/{{< param "OTEL_VERSION" >}}/pkg/ottl/contexts/ottldatapoint/README.md -[OTTL log context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/{{< param "OTEL_VERSION" >}}/pkg/ottl/contexts/ottllog/README.md -[OTTL Converter functions]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs#converters -[HasAttrKeyOnDataPoint]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md#hasattrkeyondatapoint -[HasAttrOnDataPoint]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md#hasattrondatapoint -[OTTL booleans]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.85.0/pkg/ottl#booleans -[OTTL math expressions]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.85.0/pkg/ottl#math-expressions +[OTTL]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob//pkg/ottl/README.md +[OTTL span context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob//pkg/ottl/contexts/ottlspan/README.md +[OTTL spanevent context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob//pkg/ottl/contexts/ottlspanevent/README.md +[OTTL metric context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob//pkg/ottl/contexts/ottlmetric/README.md +[OTTL datapoint context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob//pkg/ottl/contexts/ottldatapoint/README.md +[OTTL log context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob//pkg/ottl/contexts/ottllog/README.md +[OTTL Converter functions]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree//pkg/ottl/ottlfuncs#converters +[HasAttrKeyOnDataPoint]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob//processor/filterprocessor/README.md#hasattrkeyondatapoint +[HasAttrOnDataPoint]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob//processor/filterprocessor/README.md#hasattrondatapoint +[OTTL booleans]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree//pkg/ottl#booleans +[OTTL math expressions]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree//pkg/ottl#math-expressions ## Compatible components diff --git a/docs/sources/reference/components/otelcol/otelcol.processor.tail_sampling.md b/docs/sources/reference/components/otelcol/otelcol.processor.tail_sampling.md index 7e7ef2f7eb..de0e94fc04 100644 --- a/docs/sources/reference/components/otelcol/otelcol.processor.tail_sampling.md +++ b/docs/sources/reference/components/otelcol/otelcol.processor.tail_sampling.md @@ -253,7 +253,7 @@ The following arguments are supported: ### ottl_condition block The `ottl_condition` block configures a policy of type `ottl_condition`. The policy samples based on a given boolean -[OTTL](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl) condition (span and span event). +[OTTL](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree//pkg/ottl) condition (span and span event). The following arguments are supported: