NET-6394 Create/update/delete ServiceAccount on MeshGateway reconcile

[nathancoleman]

Changes proposed in this PR:
When reconciling a MeshGateway resource in the v2 API, create/update/delete a corresponding ServiceAccount.
Notably, we need to ensure we aren't updating or deleting resources that our controller didn't create simply because they have the expected name.

How I've tested this PR:
Added unit test coverage

How I expect reviewers to test this PR:
See above

Checklist:

• Tests added
• CHANGELOG entry added

[andrewstucki]

So, the only thing I really see missing here is triggering reconciliation on owned object changes. For that I think you'll probably need to extend

consul-k8s/control-plane/config-entries/controllersv2/meshconfig_controller.go

Line 209 in 6638261

func setupWithManager(mgr ctrl.Manager, resource client.Object, reconciler reconcile.Reconciler) error {

and add some ability to specify additional Owns references (maybe just make a closure for doing additional arbitrary resource watch configuration before calling Complete?), similar to how we do in our current impl:

consul-k8s/control-plane/api-gateway/controllers/gateway_controller.go

Lines 420 to 423 in 6638261

	For(&gwv1beta1.Gateway{}).
	Owns(&appsv1.Deployment{}).
	Owns(&corev1.Service{}).
	Owns(&corev1.Pod{}).

otherwise any changes to the managed ServiceAccount won't re-trigger reconciliation and you'll never hit the ifExists branch and will get configuration drift.

[andrewstucki]

I'm thinking we can just let the garbage collector clean these up? By setting the owner reference field, if the owner is deleted, these will actually get cleaned up automatically. If we're cool with just letting the GC do its work, then the only thing we have to worry about on deletion is cleaning up any side-effects that we may have done in our controller upsert path.

[nathancoleman]

My assumption was that there's a reason we do explicit cleanup in our v1 implementation here, but I don't know what that might have been. Any ideas?

Does the GC handle dependency ordering well? For example, the ServiceAccount winds up being referenced by the created Deployment, so we clean them up in reverse order in our v1 implementation.

consul-k8s/control-plane/api-gateway/gatekeeper/gatekeeper.go

Lines 60 to 86 in 671675d

	// Delete removes the resources for handling routing of network traffic.
	// This is done in the reverse order of Upsert due to dependencies between resources.
	func (g *Gatekeeper) Delete(ctx context.Context, gatewayName types.NamespacedName) error {
	g.Log.V(1).Info(fmt.Sprintf("Delete Gateway Deployment %s/%s", gatewayName.Namespace, gatewayName.Name))
	if err := g.deleteDeployment(ctx, gatewayName); err != nil {
	return err
	}
	if err := g.deleteService(ctx, gatewayName); err != nil {
	return err
	}
	if err := g.deleteRoleBinding(ctx, gatewayName); err != nil {
	return err
	}
	if err := g.deleteServiceAccount(ctx, gatewayName); err != nil {
	return err
	}
	if err := g.deleteRole(ctx, gatewayName); err != nil {
	return err
	}
	return nil
	}

[nathancoleman]

I've dropped the explicit deletes in favor of allowing the k8s garbage collector to do its thing.
If we discover a reason we need to delete explicitly like we do in the gatekeeper package today, then we can add the explicit deletion back at a later date.

[nathancoleman]

The enterprise control-plane tests are failing because our self-hosted macos runner doesn't have yarn installed. Working to resolve this outside of this PR.