HBASE-25749 Improved logging when interrupting active RPC handlers ho… #6789
+15
−9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment has been minimized.
This comment has been minimized.
|
@apurtell Appreciate your thoughts on this PR. |
This comment has been minimized.
This comment has been minimized.
| @@ -8704,6 +8700,8 @@ private void interruptRegionOperations() { | |||
| // eligible for interrupt; if so, we should interrupt it. | |||
| if (entry.getValue().booleanValue()) { | |||
| entry.getKey().interrupt(); | |||
There was a problem hiding this comment.
Sometime even if you have interuppted the thread it will still stay active and eventually after wait timeout RS will get aborted. Can you try to print the stack trace of such threads as well.
We can do that when we have waited 80%(might be different) of the wait time and there are still some Interruptable thread alive.
There was a problem hiding this comment.
@Umeshkumar9414 Please check now. Added the stack trace for thread which are not interruptable.
|
Most of the failures seems to be in |
|
This test is failing even in another PR. Example https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6790/1/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt |
mnpoonia
force-pushed
the
local_HBASE-25749
branch
2 times, most recently
from
6515ef7 to
2b0a7da
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…lding the region close lock We should add the thread to regionlockholders map to make sure we have can track the threads holding a lock even if it is not interuuptable. When calling interrupt we will not interrupt such threads but will print warn with thread name to help with debugging
mnpoonia
force-pushed
the
local_HBASE-25749
branch
from
2b0a7da to
f397308
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
@virajjasani If you could give it a look. Thanks |
virajjasani
reviewed
Mar 20, 2025
| if (entry.getValue().booleanValue()) { | ||
| entry.getKey().interrupt(); |
There was a problem hiding this comment.
How about we keep this change as is, and log whether the thread is interruptible as part of the common log above:
// print all thread stacks which are still holding locks and are the cause of RS abort
for (Map.Entry<Thread, Boolean> rslocks : regionLockHolders.entrySet()) {
LOG.warn("Thread still holding lock: {} , interruptible: {} , Stack trace: {}",
rslocks.getKey(), rslocks.getValue(), Threads.printStackTrace(rslocks.getKey()));
}
This will help make logs appear at common place and make it more easier to debug.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…lding the region close lock
We should add the thread to regionlockholders map to make sure we can track the threads holding a lock even if it is not interruptible. When calling interrupt we will not interrupt such threads but will print warn with thread name to help with debugging