Explicitly cleanup SqlTask on worker when no longer needed #24728
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
There is a separate fetcher for dynamic filters. Are you sure these are not cleaned before coordinator fetches DFs? CC @raunaqmorarka . Consider running benchmarks too |
losipiuk
force-pushed
the
lukaszos/explicitly-cleanup-sqltask-on-worker-when-no-longer-needed-43dc5e
branch
from
275bc50 to
ec05e54
Compare
sopel39
reviewed
Jan 16, 2025
| @@ -124,6 +128,7 @@ public synchronized void updateDynamicFiltersVersionAndFetchIfNecessary(long new | |||
| private synchronized void stop() | |||
| { | |||
| running = false; | |||
| remoteTaskCleaner.markDynamidFilterFetcherStopped(); | |||
losipiuk
force-pushed
the
lukaszos/explicitly-cleanup-sqltask-on-worker-when-no-longer-needed-43dc5e
branch
from
c0dab07 to
ec05e54
Compare
|
Sorry - moved to #24729 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently SqlTask objects are removed from SqlTaskManager.tasks map (cache) after timeout (15 minutes by default). Even though the object is not huge, we observed increased memory pressure up to OOM on busy clusters.
With this PR entries are dropped form SqlTaskManager as soon as they are no longer needed, when coordinator will no longer query for the information
Note: I this code is prone to race condition if last worker to acknowlege task results retransmits acknowlegement. There is a chance that a this point task is already cleaned by cleanup request from coordinator, and entry will get resurected in tasks cache map. This should be extremly rare though, and not cause any problems.