Missing a queue name upon receiving a task event #335
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
danihodovic
reviewed
Nov 26, 2024
| # couldn't find the target task by id | ||
| # the dictionary path is missing | ||
| logger.warning(f"Couldn't find the target task by its id: {target.id}") | ||
| return queue_name |
There was a problem hiding this comment.
What happens when an app deployment changes the queue and the celery-exporter doesn't restart?
There was a problem hiding this comment.
It will keep return the old queue name until the cache will have been invalidated.
That was my next question which I missed to include in the header. Pardon for that 🙏
How to not query the broker for each task as it can lead to some performance degradation depending on the producing rate ?
Several strategies comes at once, namely:
- we need to start another worker to invalidate the cache
- remove caching for this lookup method at this point
- decrease the cache size
I've also taken a look into Flower to see how it manages such cases, but couldn't find any relevant parts related to queue name processing.
|
Can you add a test? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi there!
Context
I am facing the issue with missing values for
queuelabel for the Celery tasks metrics. Basically, I don't see any other value except the default one:celerySo I tried to run the exporter locally, and here is what I've found:
track_task_eventonly considersqueueproperty upon receiving a task, but the queue name is not always available in this way ( I assume it's available when a queue name is passed directly to the call).To improve lookup capabilities of the exporter, I've added another way of getting a queue name which must solve the aforementioned issue.