[NET-6938] Create workloads in Consul for mesh gateway pods

[nathancoleman]

Changes proposed in this PR

The Pod controller currently only creates Workloads in Consul for mesh-injected Pods.
Pods for xGateways are not mesh-injected because it doesn't make sense to add a proxy sidecar to a Pod that's already running a proxy as its primary workload; however, they still require Workloads to be registered in Consul so that we can generate ProxyStateTemplates for them (in a future PR).

This PR modifies the Pod controller to create Workloads in Consul for xGateway pods as well, relying on the fact that a known annotation specifying the gateway-kind is added to all v2 xGateway Pods.

Notably, this also results in a ProxyConfiguration being created for the mesh gateway workload, and I disabled the health endpoint override functionality by adding a predefined annotation to the gateway pods as it doesn't make sense for workloads that are not mesh-injected/don't have a sidecar.

How I've tested this PR

helm install from main with this build for global.imageK8S, meshGateway.enabled=true, and experiments=[resource-apis].

Verify that the Workload and ProxyConfiguration are created in Consul by shelling into consul-server and using the following commands:

Example commands + output

Workload resource:

$ consul resource list catalog.v2beta1.Workload
{
    "resources": [
        {
            "data": {
                "addresses": [
                    {
                        "host": "10.60.2.11",
                        "ports": [
                            "proxy-health",
                            "wan",
                            "mesh"
                        ]
                    }
                ],
                "identity": "mesh-gateway",
                "locality": {
                    "region": "us-east1",
                    "zone": "us-east1-b"
                },
                "ports": {
                    "mesh": {
                        "port": 20000,
                        "protocol": "PROTOCOL_MESH"
                    },
                    "proxy-health": {
                        "port": 21000
                    },
                    "wan": {
                        "port": 8443
                    }
                }
            },
            "generation": "01HHQJDE6HA98NP5F6Y088XKJ7",
            "id": {
                "name": "mesh-gateway-6b4c5cd676-n5z2j",
                "tenancy": {
                    "namespace": "default",
                    "partition": "default",
                    "peerName": "local"
                },
                "type": {
                    "group": "catalog",
                    "groupVersion": "v2beta1",
                    "kind": "Workload"
                },
                "uid": "01HHQJDDCGD3XFGXP5ESRFHWYT"
            },
            "metadata": {
                "gateway-kind": "mesh-gateway",
                "k8s-namespace": "consul",
                "managed-by": "consul-k8s-pod-controller"
            },
            "status": {
                "consul.io/workload-health": {
                    "conditions": [
                        {
                            "message": "One or more workload health checks are not passing",
                            "reason": "HEALTH_CRITICAL",
                            "state": "STATE_FALSE",
                            "type": "healthy"
                        }
                    ],
                    "observedGeneration": "01HHQJDE6HA98NP5F6Y088XKJ7",
                    "updatedAt": "2023-12-15T20:24:17.881165934Z"
                }
            },
            "version": "715"
        }
    ]
}

ProxyConfiguration resource:

$ consul resource list mesh.v2beta1.ProxyConfiguration
{
    "resources": [
        {
            "data": {
                "dynamicConfig": {
                    "mode": "PROXY_MODE_TRANSPARENT",
                    "transparentProxy": {
                        "outboundListenerPort": 15001
                    }
                },
                "workloads": {
                    "names": [
                        "mesh-gateway-6b4c5cd676-n5z2j"
                    ]
                }
            },
            "generation": "01HHQJDE6CNEE6QNNR4XG5PN45",
            "id": {
                "name": "mesh-gateway-6b4c5cd676-n5z2j",
                "tenancy": {
                    "namespace": "default",
                    "partition": "default",
                    "peerName": "local"
                },
                "type": {
                    "group": "mesh",
                    "groupVersion": "v2beta1",
                    "kind": "ProxyConfiguration"
                },
                "uid": "01HHQJDDCBWT3ZFD2ETDWYRJM3"
            },
            "metadata": {
                "gateway-kind": "mesh-gateway",
                "k8s-namespace": "consul",
                "managed-by": "consul-k8s-pod-controller"
            },
            "version": "712"
        }
    ]
}

How I expect reviewers to test this PR

See above

Checklist

• Tests added
• CHANGELOG entry added

[nathancoleman]

Personal review

[nathancoleman]

This adds helpful logging analogous to what we do elsewhere

consul-k8s/control-plane/connect-inject/controllers/serviceaccount/serviceaccount_controller.go

Line 112 in 2b0ef6a

r.Log.Info("registering workload identity with Consul", getLogFieldsForResource(workloadIdentityResource.Id)...)

consul-k8s/control-plane/connect-inject/controllers/endpointsv2/endpoints_controller.go

Line 336 in 2b0ef6a

r.Log.Info("writing service to Consul", getLogFieldsForResource(consulSvcResource.Id)...)

[nathancoleman]

This utilizes a known annotation added to all v2 mesh gateway (and api/terminating in the future) pods

consul-k8s/control-plane/gateways/deployment.go

Line 65 in c8e5d9e

constants.AnnotationGatewayKind: meshGatewayAnnotationKind,

[nathancoleman]

Might be helpful to move this out into a commonly-used package at some point but didn't do this now

[sarahalsmiller]

Looks good to me, thanks for walking me through it