1.32 DNS Regression #38016

trivialkettle · 2025-01-15T11:11:39Z

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
[email protected] where the issue will be triaged appropriately.

Title: DNS does not find docker containers in same network

Description:

I updated from envoyproxy/envoy-distroless:v1.31.5 to envoyproxy/envoy-distroless:v1.32.3 and now envoy does not resolve other docker containers in the same network anymore.

I enabled some logs and see the following:

[2025-01-15 10:48:49.504][28][debug][router] [source/common/router/router.cc:527] [Tags: "ConnectionId":"0","StreamId":"5305907000402733376"] cluster 'ext-authz' match for URL '/envoy.service.auth.v3.Authorization/Check'
[2025-01-15 10:48:49.505][28][debug][connection] [source/common/network/connection_impl.cc:1017] [Tags: "ConnectionId":"2"] connecting to <ip>:80

If I start envoy 1.31 the IP is the IP of my extauthz container in the docker network. Envoy 1.32 probably queries not the docker network resolver but my host resolver and gets a wrong IP. I would expect envoy to resolve the IP of the container.

Other clusters have the same problem, if I change to type: static everything works.
Names like auth.example.com get resolved correctly too.

Repro steps:

A simple gRPC request that matches /my.prefix/ and is forwarded to ext authz triggers this

Note: The Envoy_collect tool
gathers a tarball with debug logs, config and the following admin
endpoints: /stats, /clusters and /server_info. Please note if there are
privacy concerns, sanitize the data prior to sharing the tarball/pasting.

Admin and Stats Output:

Include the admin output for the following endpoints: /stats,
/clusters, /routes, /server_info. For more information, refer to the
admin endpoint documentation.

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Config:

layered_runtime:
  layers:
  - name: static_layer_0
    static_layer:
      re2:
        max_program_size:
          error_level: 10000
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 80 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              typed_per_filter_config:
                envoy.filters.http.cors:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.cors.v3.CorsPolicy
                  allow_origin_string_match:
                    - prefix: "*"
                  allow_methods: GET, PUT, DELETE, POST, OPTIONS
                  allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout,authorization
                  allow_credentials: true
                  max_age: "1728000"
                  expose_headers: grpc-status,grpc-message,set-cookie
              routes:
              - match:
                  prefix: "/my.prefix/"
                route:
                  cluster: app
                  auto_host_rewrite: true
                  max_stream_duration:
                    grpc_timeout_header_max: 0s
          http_filters:
          - name: envoy.filters.http.grpc_web
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
          - name: envoy.filters.http.cors
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors
          - name: envoy.filters.http.jwt_authn
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
              providers:
                auth:
                  issuer: http://auth.example.com
                  remote_jwks:
                    http_uri:
                      uri: http://auth.example.com/jwt/jwks.json
                      cluster: auth
                      timeout: 1s
                    async_fetch:
                      fast_listener: false
                      failed_refetch_duration: 5s
                  forward: true
                  forward_payload_header: "jwt-payload"
                  pad_forward_payload_header: true
              rules:
                - match:
                    prefix: /some.prefix
                - match:
                    prefix: /
                  requires:
                    provider_name: auth
          - name: envoy.filters.http.ext_authz
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
              transport_api_version: V3
              grpc_service:
                envoy_grpc:
                  cluster_name: ext-authz
                  authority: authz
                timeout: 20s
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: app
    type: static
    connect_timeout: 20s
    typed_extension_protocol_options: &http2_protocol_options
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options:
            { }
    lb_policy: round_robin
    dns_refresh_rate: 5s
    load_assignment:
      cluster_name: app
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: 172.19.0.4
                    port_value: 50051
    dns_lookup_family: V4_ONLY
  - name: ext-authz
    lb_policy: round_robin
    type: logical_dns
    connect_timeout: 20s
    dns_refresh_rate: 5s
    dns_lookup_family: V4_ONLY
    typed_extension_protocol_options: *http2_protocol_options
    load_assignment:
      cluster_name: ext-authz
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: authz
                port_value: 80
  - name: auth
    connect_timeout: 30s
    type: LOGICAL_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: auth
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: auth.example.com
                port_value: 80

Logs:

[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:426] initializing epoch 0 (base id=0, hot restart version=11.104)
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:428] statically linked extensions:
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.proto_message_extraction, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.thrift_to_metadata, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.connection_handler: envoy.connection_handler.default
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.generic_proxy.access_loggers: envoy.generic_proxy.access_loggers.file
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.built_in_formatters.generic_proxy: envoy.built_in_formatters.generic_poxy.default
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.http.header_validators: envoy.http.header_validators.envoy_default
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.matching.input_matchers: envoy.matching.input_matchers.generic_request_matcher, envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip, envoy.matching.matchers.metadata_matcher, envoy.matching.matchers.runtime_fraction
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.network.application_protocols, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.ratelimit.hits_addend, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.built_in_formatters.http: envoy.built_in_formatters.http.default
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.load_balancing_policies: envoy.load_balancing_policies.client_side_weighted_round_robin, envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   quic.http_server_connection: quic.http_server_connection.default
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.route_config_update_requester: envoy.route_config_update_requester.default
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.dynatrace
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.matching.generic_proxy_request_input.input: envoy.matching.generic_proxy.input.host, envoy.matching.generic_proxy.input.method, envoy.matching.generic_proxy.input.path, envoy.matching.generic_proxy.input.property, envoy.matching.generic_proxy.input.request, envoy.matching.generic_proxy.input.service
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec, envoy.filters.http.wasm
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.dynamic_metadata, envoy.matching.inputs.filter_state, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.formatter: envoy.formatter.cel, envoy.formatter.metadata, envoy.formatter.req_without_query
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment, envoy.tracers.opentelemetry.resource_detectors.static_config
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.generic_proxy, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.geoip_providers: envoy.geoip_providers.maxmind
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.resource_monitors: envoy.resource_monitors.cpu_utilization, envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.common.key_value: envoy.key_value.file_based
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.generic_proxy.filters: envoy.filters.generic.router
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.srds_factory: envoy.srds_factory.default
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.path.match: envoy.path.match.uri_template.uri_template_matcher
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.dubbo_proxy.protocols: dubbo
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.request_id: envoy.request_id.uuid
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.string_matcher: envoy.string_matcher.lua
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.health_checkers: envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.http_capsule
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.built_in_formatters.stream_info: envoy.built_in_formatters.stream_info.default
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.rds_factory: envoy.rds_factory.default
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.generic_proxy.codecs: envoy.generic_proxy.codecs.dubbo, envoy.generic_proxy.codecs.http1
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.resolvers: envoy.ip
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.regex_engines: envoy.regex_engines.google_re2
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   network.connection.client: default, envoy_internal
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.http.injected_credentials: envoy.http.injected_credentials.generic, envoy.http.injected_credentials.oauth2
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.quic.server_preferred_address: quic.server_preferred_address.datasource, quic.server_preferred_address.fixed
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.tracers: envoy.tracers.datadog, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.ssl.server_context_factory: envoy.ssl.server_context_factory.default
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.quic.connection_debug_visitor: envoy.quic.connection_debug_visitor.basic
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.route.early_data_policy: envoy.route.early_data_policy.default
[2025-01-15 11:06:41.080][1][info][main] [source/server/server.cc:430]   envoy.health_check.event_sinks: envoy.health_check.event_sink.file
[2025-01-15 11:06:41.090][1][info][main] [source/server/server.cc:490] HTTP header map info:
[2025-01-15 11:06:41.090][1][info][main] [source/server/server.cc:493]   request header map: 656 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-request-id
[2025-01-15 11:06:41.090][1][info][main] [source/server/server.cc:493]   request trailer map: 120 bytes: 
[2025-01-15 11:06:41.090][1][info][main] [source/server/server.cc:493]   response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2025-01-15 11:06:41.090][1][info][main] [source/server/server.cc:493]   response trailer map: 144 bytes: grpc-message,grpc-status
[2025-01-15 11:06:41.134][1][info][main] [source/server/server.cc:879] runtime: layers:
  - name: static_layer_0
    static_layer:
      re2:
        max_program_size:
          error_level: 10000
[2025-01-15 11:06:41.134][1][info][main] [source/server/server.cc:731] No admin address given, so no admin HTTP server started.
[2025-01-15 11:06:41.134][1][info][config] [source/server/configuration_impl.cc:168] loading tracing configuration
[2025-01-15 11:06:41.134][1][info][config] [source/server/configuration_impl.cc:124] loading 0 static secret(s)
[2025-01-15 11:06:41.134][1][info][config] [source/server/configuration_impl.cc:130] loading 3 cluster(s)
[2025-01-15 11:06:41.135][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:601] c-ares library initialized.
[2025-01-15 11:06:41.136][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:391] dns resolution for authz started
[2025-01-15 11:06:41.136][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:391] dns resolution for auth.example.com started
[2025-01-15 11:06:41.137][1][info][config] [source/server/configuration_impl.cc:138] loading 1 listener(s)
[2025-01-15 11:06:41.138][1][warning][misc] [source/extensions/filters/network/http_connection_manager/config.cc:88] internal_address_config is not configured. The existing default behaviour will trust RFC1918 IP addresses, but this will be changed in next release. Please explictily config internal address config as the migration step or config the envoy.reloadable_features.explicit_internal_address_config to true to untrust all ips by default
[2025-01-15 11:06:41.139][1][info][config] [source/server/configuration_impl.cc:154] loading stats configuration
[2025-01-15 11:06:41.139][1][warning][main] [source/server/server.cc:948] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor.
[2025-01-15 11:06:41.139][1][info][main] [source/server/server.cc:990] starting main dispatch loop
[2025-01-15 11:06:41.139][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:308] dns resolution for authz completed with status 0
[2025-01-15 11:06:41.140][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:308] dns resolution for auth.example.com completed with status 0
[2025-01-15 11:06:41.140][1][info][runtime] [source/common/runtime/runtime_impl.cc:631] RTDS has finished initialization
[2025-01-15 11:06:41.140][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:249] cm init: all clusters initialized
[2025-01-15 11:06:41.140][1][info][main] [source/server/server.cc:970] all clusters initialized. initializing init manager
[2025-01-15 11:06:41.140][1][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"0"] current connecting state: true
[2025-01-15 11:06:41.140][1][debug][connection] [source/common/network/connection_impl.cc:1017] [Tags: "ConnectionId":"0"] connecting to <correct ip>:80
[2025-01-15 11:06:41.140][1][debug][connection] [source/common/network/connection_impl.cc:1036] [Tags: "ConnectionId":"0"] connection in progress
[2025-01-15 11:06:41.141][1][debug][connection] [source/common/network/connection_impl.cc:746] [Tags: "ConnectionId":"0"] connected
[2025-01-15 11:06:41.145][1][info][config] [source/common/listener_manager/listener_manager_impl.cc:944] all dependencies initialized. starting workers
[2025-01-15 11:06:43.648][24][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"1"] current connecting state: false
[2025-01-15 11:06:43.649][24][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"2"] current connecting state: true
[2025-01-15 11:06:43.649][24][debug][connection] [source/common/network/connection_impl.cc:1017] [Tags: "ConnectionId":"2"] connecting to <wrong ip>:80
[2025-01-15 11:06:43.649][24][debug][connection] [source/common/network/connection_impl.cc:1036] [Tags: "ConnectionId":"2"] connection in progress
[2025-01-15 11:07:03.653][24][debug][connection] [source/common/network/connection_impl.cc:150] [Tags: "ConnectionId":"2"] closing data_to_write=0 type=1
[2025-01-15 11:07:03.653][24][debug][connection] [source/common/network/connection_impl.cc:276] [Tags: "ConnectionId":"2"] closing socket: 1
[2025-01-15 11:07:03.655][24][debug][connection] [source/common/network/connection_impl.cc:714] [Tags: "ConnectionId":"1"] remote close
[2025-01-15 11:07:03.655][24][debug][connection] [source/common/network/connection_impl.cc:276] [Tags: "ConnectionId":"1"] closing socket: 0

The text was updated successfully, but these errors were encountered:

agrawroh · 2025-01-16T09:59:37Z

@trivialkettle Could you try explicitly passing in the resolver address using typed_dns_resolver_config?

trivialkettle · 2025-01-17T14:14:02Z

@agrawroh thanks for the answer. I played around with typed_dns_resolver_config but could not solve it.

At the end I solved it passing

dns_search: ""

to the envoy service in my compose.yaml. Though, strange that it happens only on v1.32. Maybe the order is now different?

rogercoll · 2025-01-22T11:07:57Z

I had the same issue when deploying the OpenTelemetry Demo. Setting dns_search: "" fixed the issue.

aapeliv · 2025-02-01T19:54:34Z

Had the same issue: search domain being example.com and *.example.com resolves to an IP. Causes envoy to stop connecting to clusters with 503 UF. Setting dns_search: "" fixes it, but seems hacky.

trivialkettle added bug triage Issue requires triage labels Jan 15, 2025

jmarantz added area/dns and removed triage Issue requires triage labels Jan 15, 2025

rogercoll mentioned this issue Jan 22, 2025

fix: do not use host DNS search domains for frontend-proxy Docker container open-telemetry/opentelemetry-demo#1936

Merged

1 task

aapeliv mentioned this issue Feb 1, 2025

Fix envoy with search domains Couchers-org/couchers#5581

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1.32 DNS Regression #38016

1.32 DNS Regression #38016

trivialkettle commented Jan 15, 2025

agrawroh commented Jan 16, 2025

trivialkettle commented Jan 17, 2025

rogercoll commented Jan 22, 2025

aapeliv commented Feb 1, 2025

1.32 DNS Regression #38016

1.32 DNS Regression #38016

Comments

trivialkettle commented Jan 15, 2025

agrawroh commented Jan 16, 2025

trivialkettle commented Jan 17, 2025

rogercoll commented Jan 22, 2025

aapeliv commented Feb 1, 2025