[Release-1.27] - Emit events for certificates about to expire

[brandond]

Backport fix for Emit events for certificates about to expire

• Emit events for certificates about to expire #5620

[ShylajaDevadiga]

Validated using latest commit id 191329a on release-1.27 branch
Environment Details
Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:

cat /etc/os-release
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:
1 server 1 agent node

Config.yaml:

cat /etc/rancher/rke2/config.yaml
write-kubeconfig-mode: "0644"
token:
Steps to reproduce the issue and validate the fix

1 Copy config.yaml
2 Set env variable CATTLE_NEW_SIGNED_CERT_EXPIRATION_DAYS=30 in /etc/default/rke2-server file
3 Install rke2
4 Check the warning when the certs are within 90 days of expiring

Validation results:

rke2 -v rke2 version v1.27.12+dev.191329a6 (191329a) go version go1.21.8 X:boringcrypto

kubectl get event|grep cert 5m52s Warning CertificateExpirationWarning node/ip-172-31-1-234 Node certificates require attention - restart rke2 on this node to trigger automatic rotation: api-server/client-kube-apiserver.crt: certificate CN=system:apiserver,O=system:masters will expire within 90 days at 2024-05-16T00:32:31Z, api-server/serving-kube-apiserver.crt: certificate CN=kube-apiserver will expire within 90 days at 2024-05-16T00:32:31Z, admin/client-admin.crt: certificate CN=system:admin,O=system:masters will expire within 90 days at 2024-05-16T00:32:31Z, controller-manager/client-controller.crt: certificate CN=system:kube-controller-manager will expire within 90 days at 2024-05-16T00:32:31Z, kube-proxy/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-16T00:32:31Z, kube-proxy/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-16T00:32:31Z, kubelet/client-kubelet.crt: certificate CN=system:node:ip-172-31-1-234,O=system:nodes will expire within 90 days at 2024-05-16T00:32:34Z, kubelet/serving-kubelet.crt: certificate CN=ip-172-31-1-234 will expire within 90 days at 2024-05-16T00:32:33Z, auth-proxy/client-auth-proxy.crt: certificate CN=system:auth-proxy will expire within 90 days at 2024-05-16T00:32:31Z, cloud-controller/client-rke2-cloud-controller.crt: certificate CN=rke2-cloud-controller-manager will expire within 90 days at 2024-05-16T00:32:31Z, etcd/client.crt: certificate CN=etcd-client will expire within 90 days at 2024-05-16T00:32:31Z, etcd/server-client.crt: certificate CN=etcd-server will expire within 90 days at 2024-05-16T00:32:31Z, etcd/peer-server-client.crt: certificate CN=etcd-peer will expire within 90 days at 2024-05-16T00:32:31Z, scheduler/client-scheduler.crt: certificate CN=system:kube-scheduler will expire within 90 days at 2024-05-16T00:32:31Z, supervisor/client-supervisor.crt: certificate CN=system:rke2-supervisor,O=system:masters will expire within 90 days at 2024-05-16T00:32:31Z, rke2-controller/client-rke2-controller.crt: certificate CN=system:rke2-controller will expire within 90 days at 2024-05-16T00:32:31Z, rke2-controller/client-rke2-controller.crt: certificate CN=system:rke2-controller will expire within 90 days at 2024-05-16T00:32:31Z 19s Warning CertificateExpirationWarning node/ip-172-31-7-80 Node certificates require attention - restart rke2 on this node to trigger automatic rotation: kubelet/client-kubelet.crt: certificate CN=system:node:ip-172-31-7-80,O=system:nodes will expire within 90 days at 2024-05-16T00:38:18Z, kubelet/serving-kubelet.crt: certificate CN=ip-172-31-7-80 will expire within 90 days at 2024-05-16T00:38:18Z, rke2-controller/client-rke2-controller.crt: certificate CN=system:rke2-controller will expire within 90 days at 2024-05-16T00:32:31Z, kube-proxy/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-16T00:32:31Z

kubectl get --raw /api/v1/nodes/ip-172-31-1-234/proxy/metrics|grep expiration # HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request. # TYPE apiserver_client_certificate_expiration_seconds histogram apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="3600"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="7200"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="86400"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="172800"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="345600"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="604800"} 0 apiserver_client_certificate_expiration_seconds_bucket{le="2.592e+06"} 1 apiserver_client_certificate_expiration_seconds_bucket{le="7.776e+06"} 1 apiserver_client_certificate_expiration_seconds_bucket{le="1.5552e+07"} 1 apiserver_client_certificate_expiration_seconds_bucket{le="3.1104e+07"} 1 apiserver_client_certificate_expiration_seconds_bucket{le="+Inf"} 1 apiserver_client_certificate_expiration_seconds_sum 2.591617739953341e+06 apiserver_client_certificate_expiration_seconds_count 1

sudo /usr/local/bin/rke2 certificate check INFO[0000] Server detected, checking agent and server certificates INFO[0000] Checking certificates for auth-proxy WARN[0000] /var/lib/rancher/rke2/server/tls/client-auth-proxy.crt: certificate CN=system:auth-proxy will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-auth-proxy.crt: certificate CN=rke2-request-header-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z INFO[0000] Checking certificates for supervisor WARN[0000] /var/lib/rancher/rke2/server/tls/client-supervisor.crt: certificate CN=system:rke2-supervisor,O=system:masters will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-supervisor.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z INFO[0000] Checking certificates for kube-proxy WARN[0000] /var/lib/rancher/rke2/server/tls/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-kube-proxy.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z WARN[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z INFO[0000] Checking certificates for kubelet WARN[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=system:node:ip-172-31-1-234,O=system:nodes will expire within 90 days at 2024-05-16T00:32:34Z INFO[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z WARN[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=ip-172-31-1-234 will expire within 90 days at 2024-05-16T00:32:33Z INFO[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=rke2-server-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z INFO[0000] Checking certificates for rke2-controller WARN[0000] /var/lib/rancher/rke2/server/tls/client-rke2-controller.crt: certificate CN=system:rke2-controller will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-rke2-controller.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z WARN[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=system:rke2-controller will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z INFO[0000] Checking certificates for api-server WARN[0000] /var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt: certificate CN=system:apiserver,O=system:masters will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z WARN[0000] /var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt: certificate CN=kube-apiserver will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt: certificate CN=rke2-server-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z INFO[0000] Checking certificates for admin WARN[0000] /var/lib/rancher/rke2/server/tls/client-admin.crt: certificate CN=system:admin,O=system:masters will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-admin.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z INFO[0000] Checking certificates for cloud-controller WARN[0000] /var/lib/rancher/rke2/server/tls/client-rke2-cloud-controller.crt: certificate CN=rke2-cloud-controller-manager will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-rke2-cloud-controller.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z INFO[0000] Checking certificates for controller-manager WARN[0000] /var/lib/rancher/rke2/server/tls/client-controller.crt: certificate CN=system:kube-controller-manager will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-controller.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z INFO[0000] Checking certificates for etcd WARN[0000] /var/lib/rancher/rke2/server/tls/etcd/client.crt: certificate CN=etcd-client will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/etcd/client.crt: certificate CN=etcd-server-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z WARN[0000] /var/lib/rancher/rke2/server/tls/etcd/server-client.crt: certificate CN=etcd-server will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/etcd/server-client.crt: certificate CN=etcd-server-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z WARN[0000] /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.crt: certificate CN=etcd-peer will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/etcd/peer-server-client.crt: certificate CN=etcd-peer-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z INFO[0000] Checking certificates for scheduler WARN[0000] /var/lib/rancher/rke2/server/tls/client-scheduler.crt: certificate CN=system:kube-scheduler will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/server/tls/client-scheduler.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z

Agent

sudo /usr/local/bin/rke2 certificate check INFO[0000] Agent detected, checking agent certificates INFO[0000] Checking certificates for kube-proxy WARN[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=system:kube-proxy will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z INFO[0000] Checking certificates for kubelet WARN[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=system:node:ip-172-31-7-80,O=system:nodes will expire within 90 days at 2024-05-16T00:38:18Z INFO[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z WARN[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=ip-172-31-7-80 will expire within 90 days at 2024-05-16T00:38:18Z INFO[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=rke2-server-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z INFO[0000] Checking certificates for rke2-controller WARN[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=system:rke2-controller will expire within 90 days at 2024-05-16T00:32:31Z INFO[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=rke2-client-ca@1713227551 is ok, expires at 2034-04-14T00:32:31Z ec2-user@ip-172-31-7-80:~>