Backport of Default "stats_flush_interval" to 1 minute for Consul Telemetry Collector into release/1.16.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #19663 to be assessed for backporting due to the inclusion of the label backport/1.16.

The below text is copied from the body of the original PR.

---

Description

Context
We want to reduce the number of metrics processed by the Consul Telemetry Collector.
By default, Envoy has a stats_flush_interval of 5 seconds (docs)

Changes

• For users that setup the consul telemetry collector, we default the stats_flush_interval to 1 minute (60 seconds). However, we avoid doing this override in 2 cases:
  • 1. They have a custom envoy_stats_flush_interval, we don't want to override their use case.
  • 2. They have a custom stats sink: we don't want to impact their custom metrics processing.
    • A custom stats sink can be set in multiple ways: via the envoy_statsd_url, envoy_dogstatsd_url or envoy_extra_stats_sinks_json. In order to address all these cases, we can simply check for an empty args.StatsSinksJSON after the stats sinks are built up and before the collector sink is setup.

Testing & Reproduction steps

Setup

1. Ran Makefile command to copy bootstrap config in consul-dataplane by modifying the url to be my branch.
2. Build custom consul-dataplane Docker images that I deployed to my own docker.io.
3. Applied consul-k8s installation with custom docker images to link a HCP self managed cluster to see how the Telemetry Gateway reacts. Essentially, an end to end test of cloud observability.

View helm value overrides

==> Consul Installation Summary
    Name: consul
    Namespace: consul
    
    Helm value overrides
    --------------------
    connectInject:
      enabled: true
    controller:
      enabled: true
    global:
      acls:
        bootstrapToken:
          secretKey: token
          secretName: consul-bootstrap-token
        manageSystemACLs: true
      cloud:
        apiHost:
          secretKey: api-hostname
          secretName: consul-hcp-api-host
        authUrl:
          secretKey: auth-url
          secretName: consul-hcp-auth-url
        clientId:
          secretKey: client-id
          secretName: consul-hcp-client-id
        clientSecret:
          secretKey: client-secret
          secretName: consul-hcp-client-secret
        enabled: true
        resourceId:
          secretKey: resource-id
          secretName: consul-hcp-resource-id
        scadaAddress:
          secretKey: scada-address
          secretName: consul-hcp-scada-address
      datacenter: test-flush-60-seconds
      gossipEncryption:
        secretKey: key
        secretName: consul-gossip-key
      image: docker.io/achooo/consul01:latest
      imageConsulDataplane: docker.io/achooo/consul-dataplane:1.4.0-dev
      metrics:
        enableTelemetryCollector: true
      tls:
        caCert:
          secretKey: tls.crt
          secretName: consul-server-ca
        enableAutoEncrypt: true
        enabled: true
    server:
      affinity: null
      replicas: 3
      serverCert:
        secretName: consul-server-cert
    telemetryCollector:
      cloud:
        clientId:
          secretKey: client-id
          secretName: consul-hcp-observability-client-id
        clientSecret:
          secretKey: client-secret
          secretName: consul-hcp-observability-client-secret
      enabled: true

Scenario 1: No custom stats sink / no custom flush interval

4.🥳 After the above installation with the new consul-dataplane image and the collector deployed, we see the default stats_flush_interval is 60s.

Scenario 2: Custom flush interval

5. Applied a ProxyDefaults configuration to set a custom flush interval

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
spec:
  config:
    envoy_stats_flush_interval: "10s"


❯ kubectl apply -f proxydefaults.yaml -n consul
proxydefaults.consul.hashicorp.com/global created

6. I then restarted all the pods.
7. 🥳 I verify the “”stats_flush_interval” value again, which is 10s as expected

Scenario 3: Custom sinks

8. I re-created the entire cluster again
9. I setup ProxyDefaults with a statsd url

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
spec:
  config:
    envoy_statsd_url: "udp://127.0.0.1:8125"

❯ kubectl apply -f proxydefaults.yaml -n consul
proxydefaults.consul.hashicorp.com/global created

10. 🥳 Validate no bootstrap “stats_flush_interval” configuration ( as we expect Envoy to use a default of 5s internally), which is empty as we can see:

Links

PR Checklist

• updated test coverage
• external facing docs updated
• appropriate backport labels added
• not a security concern

---

Overview of commits

• 129d6bf - 5d597a3 - a6e8c5a - 00c7207 - 74e632e - 4a9a7a1

[github-team-consul-core-pr-approver]

Auto approved Consul Bot automated PR

[johnbuonassisi]

Closing to manually create a backport in another PR