CNDB-12553: ensure that memtable is reclaimed even when notification subscribers throw #1545
+271
−9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Checklist before you submit for review
|
subscribers throw The direct cause of CNDB-12553 is that CNDB-specific subscriber to SSTableAddingNotification throws an error, and Cassandra doesn't handle it properly. In such case the flush is interrupted (despite multiple uses of exception-safe code and accumulating exceptions) after the sstable creation transaction committed, but before the memtable has been reclaimed. As a result the memtable allocator believes more and more memory is being used and being reclaimed eventually stopping writes due to apparent lack of memory in the memtable. This patch changes memtable flushing behaviour so that the memtable is reclaimed iff it has been removed from the View, regardless of whether the flush fails or not.
jakubzytka
force-pushed
the
cndb-12553-ensure-memtable-reclaimed-when-notification-subscriber-throws
branch
from
337bcb0 to
67e28f6
Compare
|
❌ Build ds-cassandra-pr-gate/PR-1545 rejected by Butler1 new test failure(s) in 2 builds Found 1 new test failures
Found 7 known test failures |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the issue
Cassandra doesn't properly support throwing notification subscribers
that fail flushes. In such case the flush is interrupted (despite
multiple uses of exception-safe code and accumulating exceptions)
after the sstable creation transaction committed, but before the
memtable has been reclaimed. As a result the memtable allocator believes
more and more memory is being used and being reclaimed eventually
stopping writes due to apparent lack of memory in the memtable.
What does this PR fix and why was it fixed
This patch changes memtable flushing behaviour so that the memtable
is reclaimed iff it has been removed from the View, regardless
of whether the flush fails or not.