heartbeat pod not working after upgrading kubernetes

[mbth20]

Hi
After I upgraded kubernetes from v1.23.4 to v1.23.5 all linkerd-heartbeat pods are showing error.
on the portal shows me "http error, status code 504 unexpected API response"

I appreciate any clue to fix this issue.

[alpeb]

I have no access to v1.23.5 clusters yet to test this. Can you share the logs from those errored pods?

[mbth20]

Thank you for your response

here is the log from one pod. It is similar log for all those pods:
time="2022-03-22T14:13:55Z" level=info msg="running version stable-2.11.0"
time="2022-03-22T14:14:00Z" level=error msg="Prometheus query failed: unexpected result Prometheus result vector length: 0"
time="2022-03-22T14:14:00Z" level=error msg="Prometheus query failed: unexpected result Prometheus result vector length: 0"
time="2022-03-22T14:14:00Z" level=error msg="Prometheus query failed: unexpected result Prometheus result vector length: 0"
time="2022-03-22T14:14:00Z" level=info msg="Sending heartbeat: https://versioncheck.linkerd.io/version.json?ext-viz=1&install-time=1628087092&k8s-version=v1.23.0&max-mem-destination=33538048&max-mem-linkerd-proxy=17489920&meshed-pods=11&p95-cpu-destination=0.002&p95-cpu-linkerd-proxy=0.005&proxy-injector-injections=11&service-profile-count=3&source=heartbeat&total-rps=7&uuid=ff5f27f8-9875-493c-9324-9217bc84bf9c&version=stable-2.11.0"
time="2022-03-22T14:14:30Z" level=fatal msg="Failed to send heartbeat: Check URL [https://versioncheck.linkerd.io/version.json?ext-viz=1&install-time=1628087092&k8s-version=v1.23.0&max-mem-destination=33538048&max-mem-linkerd-proxy=17489920&meshed-pods=11&p95-cpu-destination=0.002&p95-cpu-linkerd-proxy=0.005&proxy-injector-injections=11&service-profile-count=3&source=heartbeat&total-rps=7&uuid=ff5f27f8-9875-493c-9324-9217bc84bf9c&version=stable-2.11.0] request failed with: Get "https://versioncheck.linkerd.io/version.json?ext-viz=1&install-time=1628087092&k8s-version=v1.23.0&max-mem-destination=33538048&max-mem-linkerd-proxy=17489920&meshed-pods=11&p95-cpu-destination=0.002&p95-cpu-linkerd-proxy=0.005&proxy-injector-injections=11&service-profile-count=3&source=heartbeat&total-rps=7&uuid=ff5f27f8-9875-493c-9324-9217bc84bf9c&version=stable-2.11.0\": dial tcp 162.159.137.85:443: i/o timeout"

[alpeb]

The Prometheus failures are not relevant. The last line shows the url being requested, which attempts to send the data you see there (k8s and linkerd version, memory used by destination and proxies, number of meshed pods, etc) to linkerd.io. This allows us to have a better idea of how linkerd is being used, but it's not needed for a correct operation of linkerd. It's probable that in your case your firewall rules are denying that outgoing request. If you're not able or prefer not to allow that request to go through, you can disable the heartbeat job with the --set disableHeartBeat=false option, and that won't affect the operation of your linkerd instance.

[mbth20]

All nodes are in the same VLAN so there is no firewall rules. you brought to my attention that it may relate to proxy setting. I will check and get back if the issue still persists.
thank you.

[mbth20]

Hi
I am not sure what is the issue but I managed to fix the linkerd by downgraded from 2.11.1 to 2.10.2 and now it is working. 2.11.1 was working before but when I upgraded the kubernetes it crashed. it could be compatibility issue.