sequencer: Improve visibility of long-running background tasks #981
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bobvawter
requested review from
MattWhelan,
BramGruneir,
sravotto and
Jeremyyang920
BramGruneir
approved these changes
Aug 29, 2024
There was a problem hiding this comment.
Reviewed 4 of 4 files at r1, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @bobvawter, @Jeremyyang920, @MattWhelan, and @sravotto)
internal/sequencer/sequtil/lease_group.go line 72 at r1 (raw file):
// Shut down the nested stopper. entry.Debugf("stopping; waiting for %d tasks to complete", sub.Len()) sub.Stop(time.Minute)
Should this be configurable? Even if hidden?
bobvawter
force-pushed
the
bob_track_tasks
branch
from
b2385f1 to
e7fd574
Compare
This change corrects a context-break, where callbacks executed by the key scheduler were executed using a context from the scheduler's workgroup. In cases where the lease loop recycles and the target database is executing slowly, the old queries were not guaranteed to have been canceled when the next iteration of the loop starts. Replicator will now exit if a stuck task is detected by the lease loop since there is no other way to guarantee that the task would not continue at some arbitrary point in the future. This commit depends on cockroachdb/field-eng-powertools#4
bobvawter
force-pushed
the
bob_track_tasks
branch
from
e7fd574 to
e29d9fd
Compare
|
@BramGruneir PTAL; I think we should fail definitely in this situation. |
BramGruneir
approved these changes
Aug 29, 2024
There was a problem hiding this comment.
Reviewed 6 of 6 files at r2, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @bobvawter, @Jeremyyang920, @MattWhelan, and @sravotto)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change corrects a context-break, where callbacks executed by the key scheduler were executed using a context from the scheduler's workgroup. In cases where the lease loop recycles and the target database is executing slowly, the old queries were not guaranteed to have been canceled when the next iteration of the loop starts.
This commit depends on cockroachdb/field-eng-powertools#4
This change is