connector-inject pod crashing with GatewayClass error

[Rudraksh95]

Hi Guys!,
Im experiencing an issue with the connnector-inject pod where my pod crashes with the following error. Any help would be greatly appreciated.
Pod logs
`2023-06-07T15:59:19.740Z [INFO] consul-server-connection-manager: trying to connect to a Consul server
2023-06-07T15:59:19.841Z [INFO] consul-server-connection-manager: discovered Consul servers: addresses=[192.168.111.128:8502, 192.168.71.138:8502, 192.168.84.180:8502]
2023-06-07T15:59:19.842Z [INFO] consul-server-connection-manager: current prioritized list of known Consul servers: addresses=[192.168.111.128:8502, 192.168.71.138:8502, 192.168.84.180:8502]
2023-06-07T15:59:19.940Z [INFO] consul-server-connection-manager: connected to Consul server: address=192.168.111.128:8502
2023-06-07T15:59:19.941Z [INFO] consul-server-connection-manager: updated known Consul servers from watch stream: addresses=[192.168.71.138:8502, 192.168.84.180:8502, 192.168.111.128:8502]
2023-06-07T15:59:20.991Z INFO Waited for 1.041443747s due to client-side throttling, not priority and fairness, request: GET:https://192.168.0.1:443/apis/config.gatekeeper.sh/v1alpha1?timeout=32s

2023-06-07T15:59:22.640Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "0.0.0.0:9444"}
2023-06-07T15:59:25.640Z ERROR setup unable to register field indexes {"error": "no matches for kind "GatewayClass" in version "gateway.networking.k8s.io/v1beta1""}
github.com/hashicorp/consul-k8s/control-plane/subcommand/inject-connect.(*Command).Run
/home/runner/work/consul-k8s/consul-k8s/control-plane/subcommand/inject-connect/command.go:466
github.com/mitchellh/cli.(*CLI).Run
/home/runner/go/pkg/mod/github.com/mitchellh/[email protected]/cli.go:260
main.main
/home/runner/work/consul-k8s/consul-k8s/control-plane/main.go:21
runtime.main
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:250
2023-06-07T15:59:25.641Z [INFO] consul-server-connection-manager: stopping
Stream closed EOF for gftx-fs/consul-connect-injector-79cdf676c4-mpd85 (sidecar-injector)`

connect-inject overrides
`connectInject:
enabled: true
replicas: 1
image: null
default: false

apiGateway:
manageExternalCRDs: true
managedGatewayClass:
nodeSelector: null
tolerations: null
serviceType: ClusterIP
copyAnnotations:
service: null
deployment:
defaultInstances: 1
maxInstances: 1
minInstances: 1
serviceAccount:
annotations: null
resources:
requests:
memory: "100Mi"
cpu: "100m"
limits:
memory: "100Mi"
cpu: "100m"`

[david-yu]

Could you try with Consul K8s 1.2 and Consul 1.16 to see if the latest GA release solves your issue?

[david-yu]

Will go ahead and close, as there has been no response.

[boostrack]

Seeing this crash in 1.2.2

│
│ stream logs failed container "sidecar-injector" in pod "consul-connect-injector-8565d45c66-5vrjg" is waiting to start: ContainerCreating for consul/consul-connect-injector-8565d45c66-5vrjg (sidecar-injector)                           │
│ 2023-10-14T17:36:16.101Z [INFO]  consul-server-connection-manager: trying to connect to a Consul server                                                                                                                                   │
│ 2023-10-14T17:36:16.102Z [INFO]  consul-server-connection-manager: discovered Consul servers: addresses=[172.20.20.10:8502]                                                                                                               │
│ 2023-10-14T17:36:16.102Z [INFO]  consul-server-connection-manager: current prioritized list of known Consul servers: addresses=[172.20.20.10:8502]                                                                                        │
│ 2023-10-14T17:36:16.199Z [INFO]  consul-server-connection-manager: connected to Consul server: address=172.20.20.10:8502                                                                                                                  │
│ 2023-10-14T17:36:16.202Z [INFO]  consul-server-connection-manager: updated known Consul servers from watch stream: addresses=[172.20.20.10:8502]                                                                                          │
│ 2023-10-14T17:36:16.403Z    INFO    controller-runtime.metrics    Metrics server is starting to listen    {"addr": "0.0.0.0:9444"}                                                                                                        │
│ 2023-10-14T17:36:16.601Z    ERROR    setup    unable to register field indexes    {"error": "no matches for kind \"GatewayClass\" in version \"gateway.networking.k8s.io/v1beta1\""}                                                      │
│ github.com/hashicorp/consul-k8s/control-plane/subcommand/inject-connect.(*Command).Run                                                                                                                                                    │
│     /home/runner/work/consul-k8s/consul-k8s/control-plane/subcommand/inject-connect/command.go:490                                                                                                                                        │
│ github.com/mitchellh/cli.(*CLI).Run                                                                                                                                                                                                       │
│     /home/runner/go/pkg/mod/github.com/mitchellh/[email protected]/cli.go:260                                                                                                                                                                    │
│ main.main                                                                                                                                                                                                                                 │
│     /home/runner/work/consul-k8s/consul-k8s/control-plane/main.go:21                                                                                                                                                                      │
│ runtime.main                                                                                                                                                                                                                              │
│     /opt/hostedtoolcache/go/1.20.8/x64/src/runtime/proc.go:250                                                                                                                                                                            │
│ 2023-10-14T17:36:16.602Z [INFO]  consul-server-connection-manager: stopping                                                                                                                                                               │
│ Stream closed EOF for consul/consul-connect-injector-8565d45c66-5vrjg (sidecar-injector)                                                                                                                                                  

consul server

from=10.42.0.108:46042 error="No cluster leader"
2023-10-14T17:35:47.500Z [WARN]  agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2023-10-14T17:35:47.501Z [INFO]  agent.server.raft: entering candidate state: node="Node at 172.20.20.10:8300 [Candidate]" term=3
2023-10-14T17:35:47.503Z [INFO]  agent.server.raft: election won: term=3 tally=1
2023-10-14T17:35:47.503Z [INFO]  agent.server.raft: entering leader state: leader="Node at 172.20.20.10:8300 [Leader]"
2023-10-14T17:35:47.504Z [INFO]  agent.server: cluster leadership acquired
2023-10-14T17:35:47.505Z [INFO]  agent.server: New leader elected: payload=agent-one
2023-10-14T17:35:47.516Z [INFO]  agent.server.autopilot: reconciliation now enabled
2023-10-14T17:35:47.520Z [INFO]  agent.leader: started routine: routine="federation state anti-entropy"
2023-10-14T17:35:47.520Z [INFO]  agent.leader: started routine: routine="federation state pruning"
2023-10-14T17:35:47.520Z [INFO]  agent.leader: started routine: routine="streaming peering resources"
2023-10-14T17:35:47.520Z [INFO]  agent.leader: started routine: routine="metrics for streaming peering resources"
2023-10-14T17:35:47.520Z [INFO]  agent.leader: started routine: routine="peering deferred deletion"
2023-10-14T17:35:47.521Z [INFO]  connect.ca: initialized primary datacenter CA from existing CARoot with provider: provider=consul
2023-10-14T17:35:47.522Z [INFO]  agent.leader: started routine: routine="intermediate cert renew watch"
2023-10-14T17:35:47.522Z [INFO]  agent.leader: started routine: routine="CA root pruning"
2023-10-14T17:35:47.522Z [INFO]  agent.leader: started routine: routine="CA root expiration metric"
2023-10-14T17:35:47.522Z [INFO]  agent.leader: started routine: routine="CA signing expiration metric"
2023-10-14T17:35:47.522Z [INFO]  agent.leader: started routine: routine="virtual IP version check"
2023-10-14T17:35:47.522Z [INFO]  agent.leader: started routine: routine="config entry controllers"
2023-10-14T17:35:47.522Z [INFO]  agent.leader: stopping routine: routine="virtual IP version check"
2023-10-14T17:35:47.522Z [INFO]  agent.leader: stopped routine: routine="virtual IP version check"
2023-10-14T17:35:49.101Z [ERROR] agent.server.autopilot: Failed to reconcile current state with the desired state

this seems related somehow with server unhealthy quorum server ( rebooted all consul VMs) and after getting cluster stabilised found not way to make pod recover from that crash. deleting k8s deployments for gateway and connect-injector and redeploying fixes the issue although in my case i need to reconfigure the service mesh peering once again. This seems to only have caused issues on 1 side of the mesh