Fix 7875 #7910
Draft
Fix 7875 #7910
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
❌ Your patch check has failed because the patch coverage (0.00%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage.
Additional details and impacted files@@ Coverage Diff @@
## release-13.0 #7910 +/- ##
================================================
- Coverage 89.48% 1.22% -88.26%
================================================
Files 276 276
Lines 60063 59391 -672
Branches 7524 7412 -112
================================================
- Hits 53747 729 -53018
- Misses 4166 58581 +54415
+ Partials 2150 81 -2069 |
…ng Citus upgrade tests" This reverts commit b6b73e2.
…des (citusdata#7875) Currently, RecoverWorkerTransactions() creates a new connection for each worker node and then performs transaction recovery by reading and locking the pg_dist_transaction catalog table until the end of the transaction. When RecoverTwoPhaseCommits() calls RecoverWorkerTransactions() for each worker node, the lock acquisition order between pg_dist_authinfo and pg_dist_transaction can reverse on alternate iterations. This reversal can lead to a deadlock if any concurrent process requires locks on these catalog tables—a situation that has surfaced during the Citus upgrade workflow. To resolve this, we now pre-establish all worker node connections upfront. This change ensures that RecoverWorkerTransactions() operates with a single, consistent distributed catalog table connection, thereby always acquiring locks on pg_dist_authinfo and pg_dist_transaction in the correct order and preventing potential deadlocks during extension updates or similar operations.
Prepared transactions on the local node are initiated and managed by a remote worker in the cluster. As such, we now only call RecoverWorkerTransactions for remote nodes, ensuring that transaction recovery is handled by the appropriate node.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix Deadlock with transaction recovery is possible during Citus upgrades (#7875)
Currently, RecoverWorkerTransactions() creates a new connection for each worker node and then performs transaction recovery by reading and locking the pg_dist_transaction catalog table until the end of the transaction. When RecoverTwoPhaseCommits() calls RecoverWorkerTransactions() for each worker node, the lock acquisition order between pg_dist_authinfo and pg_dist_transaction can reverse on alternate iterations.This reversal can lead to a deadlock if any concurrent process requires locks on these catalog tables—a situation that has surfaced during the Citus upgrade workflow.To resolve this, we now pre-establish all worker node connections upfront.
This change ensures that RecoverWorkerTransactions() operates with a single, consistent distributed catalog table connection, thereby always acquiring locks on pg_dist_authinfo and pg_dist_transaction in the correct order and preventing potential deadlocks during extension updates or similar operations.
The PR also reverts the commit that disabled the relevant test cases,