nginx readinessProbe fails on freshly deployed lagoon-examples/drupal-base

[rwunderer]

Describe the bug

We are currently in the process of evaluating lagoon, but are facing an issue with readinessProbes in nginx

To Reproduce

Steps to reproduce the behavior:

1. Set up a Kubernetes cluster with workers having both private and public network interfaces
2. Install Lagoon
3. Deploy a test application based on https://github.com/lagoon-examples/drupal-base
4. Deployment fails because the nginx container never becomes ready.

Logs

k -n lagoon-drupal-example-main logs nginx-6dc76c5c5-64wrb
Defaulted container "nginx" out of: nginx, php
Setting up Healthz routing
Healthz routing - using PHP
2025/01/15 17:06:03 [error] 43#43: *1 access forbidden by rule, client: 91.XX.YY.ZZ, server: , request: "GET /nginx_status HTTP/1.1", host: "10.2.3.39:50000"
2025/01/15 17:06:04 [error] 43#43: *2 access forbidden by rule, client: 91.XX.YY.ZZ, server: , request: "GET /nginx_status HTTP/1.1", host: "10.2.3.39:50000"
2025/01/15 17:06:12 [error] 43#43: *3 access forbidden by rule, client: 91.XX.YY.ZZ, server: , request: "GET /nginx_status HTTP/1.1", host: "10.2.3.39:50000"

91.XX.YY.ZZ is the public ip address of the worker node where the nginx pod is running.

Additional context

In https://github.com/uselagoon/lagoon-images/blob/main/images/nginx/nginx.conf the /nginx_status location is limited to localhost and RFC1918 addresses, however on our servers the Kubelet seems to be using the node's public ip for readiness probes. As this is a managed service it is not possible for us to limit the addresses the Kubelet binds to.

   server {

        listen       50000;

        location /nginx_status {
            stub_status     on;
            access_log      off;
            allow           127.0.0.1;
            allow           10.0.0.0/8;
            allow           172.16.0.0/12;
            allow           192.168.0.0/16;
            deny            all;
        }

    }

Is this even the correct check endpoint?

Also, is it even correct that the deployment tries to use /nginx_status for readinessProbes (and livenessProbes)? Shouldn't it use /.lagoonhealthz that was especially defined for the purpose?
https://github.com/uselagoon/lagoon-images/tree/main/images/nginx/healthcheck

[tobybellwood]

How have you configured cluster ingress?

[rwunderer]

How have you configured cluster ingress?

Yes we did.

I don't think ingress is related to the problem though. Health checks by the kubelet bypass ingress as they need to work for pods that are not exposed outside the cluster as well.

And of course there is no traffic forwarded by the ingress controller as the pod is never becoming ready.

The problem is that the kubelet is running these health checks from an ip address that nginx considers off-limits for /nginx_status.

I see three possible solutions:

1. globally allow access to /nginx_status (not good as it is bad practice to allow external access to this information)
2. force the kubelet somehow to use the internal node ip for health checks which "happens" to be with in the allowed range (this wouldn't work on a cluster without private node ips)
3. use a different (globally allowed) endpoint for health checks

The last solution looks the most promising to me, especially as there seems to exist a dedicated service in lagoon for exactly this reason. (https://github.com/uselagoon/lagoon-images/tree/main/images/nginx/healthcheck)
So I'm wondering why lagoon does not use this service for pod health checks?

Or am I completely missing something here?