Backport of [NET-10567] Fix namespace normalization on external registration/ACL Setup for Terminating Gateways into release/1.5.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #4224 to be assessed for backporting due to the inclusion of the label backport/1.5.x.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@jm96441n
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.

merge conflict error: POST https://api.github.com/repos/hashicorp/consul-k8s/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

---

Changes proposed in this PR

• Fix namespace normalization on external registration (when namespaces are enabled the empty namespace is converted to "default" for acl rules)
• Fix how acls are setup for terminating gateways -> moved from the registrations controller/cache to the terminating gateway controller to avoid race conditions with persisting the terminating gateway config entry (TODO: clean up extra policies when a terminating gateway is removed/services are removed and are not referenced by any other terminating gateways)
• Fixed bug in cache where values weren't being persisted with the correct key
• updated terminating gateway tests so they would catch this in the future

How I've tested this PR

• wrote tests
• ran in kind cluster

How I expect reviewers to test this PR

• clone this branch and build the container using make docker-dev
• clone down the following gist: https://gist.github.com/jm96441n/1be432dd2b6b5ade7bc1b058539f6095
• modify the CONSUL_K8S_CHART_LOCATION variable in the start.sh file to point to the helm charts in your local version of consul-
• run the start.sh file (this requires you to have kind and yq on your machine, and you'll need to run chmod +x ./start.sh)
• after that completes run the following to set up your consul CLI to talk to consul:

export CONSUL_HTTP_ADDR="127.0.0.1:8501"
export CONSUL_HTTP_TOKEN=$(kubectl get --namespace consul secrets/consul-consul-bootstrap-acl-token --template={{.data.token}} | base64 -d)
export CONSUL_HTTP_SSL=true
export CONSUL_HTTP_SSL_VERIFY=false

• enable a port forward to the consul server on port 8501
• run consul acl role list and you will see both terminating gateways with the term gateway policy and the zoidberg and nibbler policies
• run consul acl policy read -name zoidberg-write-policy to see the policy include the namespace
• shell into the bender service and run curl localhost:1234 to see the request to zoidberg go through, run curl localhost:5678 to see the request to nibbler go through
• now we'll test that acl policies are removed correctly
• edit the termgw.yaml file to no longer reference the zoidberg service and apply the file
• run consul acl role list and see the first terminating gateway no longer references the zoidberg policy
• run consul acl policy list and you'll see the zoidberg policy still exists
• now edit the termgw2.yaml file to no longer reference the zoidberg service and apply that file
• now run consul acl role list and see that none of the terminating gateways reference that policy
• now run consul acl policy list and see that the zoidberg policy no longer exists because no gateway is referencing it

Checklist

• Tests added
• CHANGELOG entry added

---

Overview of commits

• 16b5d81 - b07ae41 - 1dc22bf - 0ec89d1 - d56ad44 - 3d30c81 - 937e39e - 7da17dc - 253002c - 3947e69 - 73bbbe7 - 8d905f3 - 8262074 - c749f90 - 0aff5cd - 6281461 - 76e907c - 2d9bf08 - 6bdaa5a - df14fa3 - ad789e2 - 5737d89 - 2f8e9c7 - b0ac632 - 001b660 - 66b2e0d - 3f3c631 - dd851d5 - a6d937f - 24ccf60 - ff61c01 - 197501d - 5617b2f - 7fe78ce - a8bfe59 - 4953f78 - e77adb4 - ad73acc - e4cdf2c - a2ad236

[hashicorp-cla-app]

All committers have signed the CLA.

[hashicorp-cla-app]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

---

temp seems not to be a GitHub user.
You need a GitHub account to be able to sign the CLA.
If you have already a GitHub account, please add the email address used for this commit to your account.

_{Have you signed the CLA already but the status is still pending? Recheck it.}