Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

injection stack traces when deploying multi-port application #1594

Closed
darkn3rd opened this issue Oct 6, 2022 · 8 comments
Closed

injection stack traces when deploying multi-port application #1594

darkn3rd opened this issue Oct 6, 2022 · 8 comments
Labels
type/bug Something isn't working

Comments

@darkn3rd
Copy link

darkn3rd commented Oct 6, 2022 •
edited
Loading

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Cannot deploy multi-port application with metrics enabled. The StatefulSet returns this event:

Warning  FailedCreate      5m39s (x19 over 27m)  statefulset-controller  create Pod dgraph-dgraph-alpha-0 in StatefulSet dgraph-dgraph-alpha failed error: admission webhook "consul-connect-injector.consul.hashicorp.com" denied the request: multi port services are not compatible with metrics

Reproduction Steps

Overview of Steps:

  • Deploy Consul Service Mesh 
    global:
  name: consul
  datacenter: dc1
  metrics:
    enabled: true
    enableAgentMetrics: true
    agentMetricsRetentionTime: "1m"
connectInject:
  enabled: true
controller:
  enabled: true
ui:
  enabled: true
  metrics:
    enabled: true
    provider: prometheus
    baseURL: http://prometheus-server.prometheus.svc.cluster.local
  • Deploy o11y per Layer 7 Observability with Prometheus, Grafana, and Kubernetes guide.
  • Deploy Dgraph that uses multiport

NOTE: I can provide step-by-step to reproduce this, but since this is easy to reproduce, gave the minimal steps.

Logs

Consul Connect Logs 
2022-10-06T14:01:28.873Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": "0.0.0.0:9444"}
2022-10-06T14:01:28.874Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/mutate"}
2022-10-06T14:01:28.874Z	INFO	attempting to acquire leader lease consul/consul-controller-lock...

2022-10-06T14:01:28.875Z	INFO	starting metrics server	{"path": "/metrics"}
2022-10-06T14:01:28.875Z	INFO	controller-runtime.webhook.webhooks	starting webhook server
2022-10-06T14:01:28.875Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2022-10-06T14:01:28.876Z	INFO	controller-runtime.webhook	serving webhook server	{"host": "", "port": 8080}
2022-10-06T14:01:28.890Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
2022-10-06T14:11:20.505Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-zero-0", "ns": "dgraph"}
2022-10-06T14:11:20.511Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:20.518Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:20.519Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:20.552Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:20.557Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:20.557Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:20.595Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:20.620Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:20.620Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:20.674Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:20.680Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:20.680Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:20.738Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:20.743Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:20.743Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:20.836Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:20.840Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:20.840Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:21.016Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:21.022Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:21.022Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:21.356Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:21.363Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:21.363Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:22.016Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:22.022Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:22.022Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:23.314Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:23.320Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:23.320Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:25.896Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:25.901Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:25.901Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:31.040Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:31.114Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:31.115Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:41.372Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:11:41.378Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:11:41.378Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:11:58.861Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-zero-1", "ns": "dgraph"}
2022-10-06T14:12:01.872Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:12:01.880Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:12:01.880Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:12:36.396Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-zero-2", "ns": "dgraph"}
2022-10-06T14:12:42.858Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:12:42.863Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:12:42.863Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:14:04.801Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:14:04.836Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:14:04.836Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:16:48.709Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:16:48.715Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:16:48.715Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966
2022-10-06T14:22:16.432Z	INFO	handler.connect	received pod	{"name": "dgraph-dgraph-alpha-0", "ns": "dgraph"}
2022-10-06T14:22:16.438Z	INFO	handler.connect	processing multiport pod
2022-10-06T14:22:16.438Z	ERROR	handler.connect	checking unsupported cases for multi port pods	{"error": "multi port services are not compatible with metrics"}
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:99
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:101
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
	/home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:76
net/http.HandlerFunc.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2084
net/http.(*ServeMux).ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2462
net/http.serverHandler.ServeHTTP
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2916
net/http.(*conn).serve
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:1966

Expected behavior

I expected that I would be able to deploy a multi-port application and be able to use metrics, and I expected that there would be errors with the consul-connect-injector.

Environment details

Additionally, please provide details regarding the Kubernetes Infrastructure, as shown below:

  • Kubernetes version: v1.22
  • Cloud Provider: GKE
  • Networking CNI plugin in use: default whatever that is
@darkn3rd darkn3rd added the type/bug Something isn't working label Oct 6, 2022
@t-eckert
Copy link
Contributor

t-eckert commented Oct 6, 2022

Hi @darkn3rd,

Thank you for opening this issue. This is a known limitation of multiport deployments as listed in the "Caveats for Multiport Pods" section of the Connect docs: https://www.consul.io/docs/k8s/connect#caveats-for-multi-port-pods

Best,
Thomas

@t-eckert t-eckert closed this as completed Oct 6, 2022
@darkn3rd
Copy link
Author

darkn3rd commented Oct 6, 2022

@t-eckert The fact that I got feedback in the logs and such was great, that helped me understand the issue at hand. Thanks for the doc link.

You know when metrics will be supported with multi-port? or will multi-port get greater support later?

@ishustava
Copy link
Contributor

You know when metrics will be supported with multi-port? or will multi-port get greater support later?

We don't have anything definitive yet, but it's definitely something we're going to look into soon!

@darkn3rd
Copy link
Author

darkn3rd commented Nov 7, 2022

Pinging to keep this alive. Let me know if anything will be added to the roadmap

@darkn3rd
Copy link
Author

Pinging to keep this alive again, so automation doesn't erase this issue.

@t-eckert
Copy link
Contributor

@darkn3rd, thank you for your commitment to this issue. I can assure you that we don't have automation that will erase this issue and we are tracking it internally as we plan our roadmap for Consul on Kubernetes. When we plan to address this, you will certainly be kept appraised.

@darkn3rd
Copy link
Author

@t-eckert Thank you. That's promising. I remember some important issue in another project (Terraform) were closed and never addressed, which was demoralizing. I am glad that is not the case here, very positive.

Also, I talked with the PM on this at the recent HashiConf23 in SF, so I am glad about the new developments with the new schema for the v2 service catalog.

@t-eckert
Copy link
Contributor

Good to hear! V2 will bring huge improvements to Multiport that we are excited about.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@darkn3rd @ishustava @t-eckert