Liveness probe for tokman fails often, resulting in container restarts and poor user experience

[nforro]

The liveness probe fails with:
Get "http://${IPADDR}:8000/api/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

This happens on both prod and stage, but on prod it's very frequent, currently the counter says 649 times in the last 12 days, while on stage it has happened only 5 times in the last 3 days.

[nforro]

I tried to debug this as much as I can, however I didn't find find out much. Every now and then when doing the liveness check a kubelet sends a HTTP request that is accepted by a tokman container but then the connection is terminated by the client (before the server has a chance to respond) and that results in the error message. I believe that rules out an issue on tokman side, but that's all I can tell.

There is also an error appearing from time to time on a short-running worker pod:
Liveness probe failed: Ignored keyword arguments: {'type': 'pagure'}
I had a look into that as well and the error message is actually just an unrelated warning (coming from ogr) and the actual error is that celery status (the command the liveness probe runs) on a short-running worker sometimes doesn't produce the expected output (the short-running worker hostname is missing from the list, only long-running workers are there) - this is reproducible even when running the command manually in a terminal, from time to time.

[mfocko]

Opened RITM1766219

[mfocko]

I don't see any of these events in the last 5 days, I will keep an eye on it and close the issue for now

---

OTOH the same issue has popped up with a short-running worker, so we might want to adjust the requests/limits soon™

There are also some outcomes from the ticket above that can be included in our own docs.