Container vault-agent is not valid for pod

[TitanRob16]

I'm following this tutorial:
https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-sidecar

Combined with this:
https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-google-cloud-gke

I've launched the test application and have patched the deployment with the patch-inject-secrets.yaml.

However, the tutorial then goes on to say:
"Wait until the re-deployed orgchart pod reports that it is Running and ready (2/2)."

But running kubectl get pods shows:

NAME                                    READY   STATUS    RESTARTS   AGE
devwebapp-7c89bbcd9b-2hcns              1/1     Running   0          32m
vault-0                                 1/1     Running   0          105m
vault-1                                 1/1     Running   0          105m
vault-2                                 1/1     Running   0          105m
vault-agent-injector-5b5889ffb4-c8kwx   1/1     Running   0          105m

And when running:

kubectl logs \
    $(kubectl get pod -l app=devwebapp -o jsonpath="{.items[0].metadata.name}") \
    --container vault-agent

I get:
error: container vault-agent is not valid for pod devwebapp-7c89bbcd9b-2hcns

There was some conversation under hashicorp/vault-k8s#206 about firewalls, especially if you're using a private GKE cluster (which I am). However, I have a firewall rule that allows connections from the Control Plane to the Nodes via 443.

I wonder if this has been set incorrectly:

vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

I read this: https://www.vaultproject.io/docs/platform/k8s/injector/examples#connectivity
And then changed the above to:

vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:8080"

But that had no effect when I redeployed the test app.

There's not much information from what I can see on troubleshooting this, or how your cluster should be configured within GKE.

1. Is there a way to check if it is a connectivity issue within the cluster itself?
2. Is there anything else I can try to diagnose this?
3. It'd be helpful to have clearer documentation on how the GKE private cluster should be configured in this scenario. As it's written, there's no detail about the GKE set up as such.

Any help with this would be greatly appreciated!

[tvoran]

Hi @TitanRob16, as mentioned in hashicorp/vault-k8s#46 (comment), the firewall rule needs to allow traffic to port 8080, so I'd suggest trying that.