IGNITE-27678 Same partitions on different nodes can hold different updates if writeThrough is enabled by zstan · Pull Request #12925 · apache/ignite

zstan · 2026-03-23T11:15:09Z

No description provided.

…dates if writeThrough is enabled

sonarqubecloud · 2026-04-10T12:44:23Z

Quality Gate failed

Failed conditions
6 New Code Smells (required ≤ 1)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Copilot

Pull request overview

This PR addresses transaction recovery for write-through caches so surviving nodes do not retain divergent partition updates after a node leaves during transaction completion.

Changes:

Adds a DHT transaction salvage message and handling paths for write-through recovery.
Adjusts transaction manager/future recovery behavior and store-session exception propagation.
Adds and updates tests covering write-through recovery, idle verify consistency, and leftover salvage transaction cleanup.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`modules/core/src/test/java/org/apache/ignite/internal/processors/cache/query/IndexingSpiQueryTxSelfTest.java`	Updates expected transaction exception behavior and waits for futures.
`modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/IgniteTxCacheWriteSynchronizationModesMultithreadedTest.java`	Adds post-test assertion for empty salvage transaction tracking.
`modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/dht/IgniteCacheTxRecoveryRollbackTest.java`	Extends cleanup checks to include salvage transaction state.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxRemoteStateImpl.java`	Makes remote tx state maps final.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxRemoteStateAdapter.java`	Makes active cache ID list final.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxManager.java`	Adds write-through-specific salvage/recovery coordination.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxHandler.java`	Registers handling for DHT salvage requests.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxAdapter.java`	Adjusts nullable annotation on finalization status implementation.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/store/GridCacheStoreManagerAdapter.java`	Logs and propagates runtime store-session listener failures.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/near/GridNearTxFinishFuture.java`	Sends salvage requests to backups during committed checks for write-through txs.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/GridDhtLocalPartition.java`	Improves partition-state assertion diagnostics.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionsExchangeFuture.java`	Suppresses unused warning for retained field.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/GridDhtTxSalvageMessage.java`	Adds a new cache message for transaction salvage.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/GridDhtTxRemote.java`	Simplifies master node ID collection creation.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/GridDhtTxLocal.java`	Salvages local write-through txs on heuristic failure.
`modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/GridDhtTxFinishFuture.java`	Adds node-id-aware salvage handling when a finish target has left.
`modules/control-utility/src/test/java/org/apache/ignite/util/IdleVerifyCheckWithWriteThroughTest.java`	Adds idle-verify regression coverage for write-through tx recovery.
`modules/control-utility/src/test/java/org/apache/ignite/testsuites/IgniteControlUtilityTestSuite2.java`	Includes the new write-through idle-verify test in the suite.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

zstan · 2026-05-18T05:37:55Z


            // Treat heuristic exception as critical.
-            if (X.hasCause(e, IgniteTxHeuristicCheckedException.class))
+            if (X.hasCause(e, IgniteTxHeuristicCheckedException.class)) {


CacheStoreWithIgniteTxFailureTest#testSystemExceptionAfterCacheStoreCommit fail without this fix
IndexingSpiQueryTxSelfTest - waitForCondition appended

ignitetcbot · 2026-05-20T09:02:12Z

TCBot Test Analysis

TeamCity: --> Run :: All Results
Branch: pull/12925/head
Base: master
Tested commit: 88ef5cd

Possible Blockers (0)

No blockers found.

New Tests (5)

Cache 13: 1 tests
- IgniteCacheTestSuite13: SystemViewCacheExpiryPolicyTest.testCacheViewExpiryPolicy[factory=javax.cache.configuration.FactoryBuilder$SingletonFactory@7e7740a5, actual=SingletonFactory [expiryPlc=EternalExpiryPolicy [create=ETERNAL]]] - PASSED
Control Utility 2: 4 tests
- IgniteControlUtilityTestSuite2: IdleVerifyCheckWithWriteThroughTest.testTxCoordinatorLeftClusterWithEnabledReadWriteThrough[cmdHnd=cli, withPersistence=false, concMode=PESSIMISTIC] - PASSED
- IgniteControlUtilityTestSuite2: IdleVerifyCheckWithWriteThroughTest.testTxCoordinatorLeftClusterWithEnabledReadWriteThrough[cmdHnd=cli, withPersistence=true, concMode=PESSIMISTIC] - PASSED
- IgniteControlUtilityTestSuite2: IdleVerifyCheckWithWriteThroughTest.testTxCoordinatorLeftClusterWithEnabledReadWriteThrough[cmdHnd=cli, withPersistence=false, concMode=OPTIMISTIC] - PASSED
- IgniteControlUtilityTestSuite2: IdleVerifyCheckWithWriteThroughTest.testTxCoordinatorLeftClusterWithEnabledReadWriteThrough[cmdHnd=cli, withPersistence=true, concMode=OPTIMISTIC] - PASSED

zstan · 2026-05-21T06:56:35Z

-                    if ((tx.near() && !tx.local() && tx.originatingNodeId().equals(evtNodeId))
+                    Map<UUID, Collection<UUID>> txNodes = tx.transactionNodes();
+
+                    if (tx.storeWriteThrough() && txNodes != null


for reviewer: class that can catch\handle more questions about flags\implementation approaches is : IgniteTxCacheWriteSynchronizationModesMultithreadedTest
it can be speed up by testing only writeThrough scenarios, check:
IgniteTxCacheWriteSynchronizationModesMultithreadedTest#multithreaded - store If {@code true} sets store in cache configuration

zstan · 2026-05-21T06:58:13Z

+                        sendTxSalvage(tx, evtNodeId);
+                    }
+
+                    Supplier<Boolean> fullSyncedOp = () -> tx.writeEntries().stream().map(e ->


Possible we can change it in future but for now only full synced is covered, othervize i expect some failures with IgniteTxCacheWriteSynchronizationModesMultithreadedTest

sonarqubecloud · 2026-05-21T12:17:57Z

Quality Gate failed

Failed conditions
5 New Code Smells (required ≤ 1)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

alex-plekhanov · 2026-06-25T18:50:10Z

+                    if (tx.masterNodeIds().contains(nodeId))
+                        continue;
+
+                    ClusterNode involvedNode = cctx.discovery().node(nodeId);


Let's rename involvedNode to backupNode for clarity and similarity with other GridDhtTxSalvageMessage sending code.

alex-plekhanov · 2026-06-25T18:55:59Z

+
+                    U.awaitQuiet(nodeLeftRegisteredOnBackup);
+
+                    // let`s wait until all discovery events have been processed on backup node.


alex-plekhanov · 2026-06-25T18:57:23Z

+        assertEquals(secondVal, nodeCoord.cache(DEFAULT_CACHE_NAME).get(primaryKey));
+        assertEquals(secondVal, nodeBackup.cache(DEFAULT_CACHE_NAME).get(primaryKey));
+
+        if (withPersistence) {


Let's remove withPersistance condition and check for in-memory mode too. For in-memory mode it cache-store value will be checked.

got it, done

alex-plekhanov · 2026-06-25T19:18:23Z

+                    }
+
+                    Supplier<Boolean> fullSyncedOp = () -> tx.writeEntries().stream().map(e ->
+                        cctx.cacheContext(e.cacheId())).allMatch(GridCacheContext::syncCommit);


According to IgniteTxStateImpl#syncMode mode is FULL_SYNC if any of caches have FULL_SYNC mode. Instead of entries iterator we can iterate over active cache ids (txState().cacheIds()). Maybe it's better ti reuse IgniteTxStateImpl#syncMode somehow, but there is assert false in IgniteTxRemoteStateAdapter

unexpected behavior in this method )

for (int i = 0; i < activeCacheIds.size(); i++) { int cacheId = activeCacheIds.get(i); CacheWriteSynchronizationMode cacheSyncMode = cctx.cacheContext(cacheId).config().getWriteSynchronizationMode(); switch (cacheSyncMode) { case FULL_SYNC: return FULL_SYNC;

What is unexpected? If any of caches have FULL_SYNC - return FULL_SYNC, else if any of caches have PRIMARY_SYNC return PRIMARY_SYNC, else (all caches have FULL_ASYNC) - return FULL_ASYNC

got it, fixed

alex-plekhanov · 2026-06-26T13:07:58Z

     */
    private void multithreadedTests(CacheWriteSynchronizationMode syncMode, boolean restart) throws Exception {
-        multithreaded(syncMode, 0, false, false, restart);
+        //multithreaded(syncMode, 0, false, false, restart);


Please revert these changes

ups .. done

github-actions · 2026-06-26T13:51:21Z

Possible compatibility issues. Please, check rolling upgrade cases

This PR modifies protected classes (with Order annotation).
Changes to these classes can break rolling upgrade compatibility.

Affected files:

modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/GridDhtTxSalvageMessage.java

github-actions Bot added the compatibility label Mar 23, 2026

zstan force-pushed the ignite-27678-nostruct branch 2 times, most recently from c623095 to 2029771 Compare March 23, 2026 11:40

zstan force-pushed the ignite-27678-nostruct branch 3 times, most recently from 0bfd72d to ff5d209 Compare April 6, 2026 13:10

zstan added 7 commits April 9, 2026 15:17

IGNITE-27678 Same partitions on different nodes can hold different up…

b4e7418

…dates if writeThrough is enabled

fix

fb9de7f

no break + event

0585650

fix npe

da2af74

tx manager

8651672

ligthter tests

b47be91

fix

d2bb726

zstan force-pushed the ignite-27678-nostruct branch from ff5d209 to d2bb726 Compare April 9, 2026 12:18

zstan added 4 commits April 9, 2026 18:05

last failed test fix

ef55749

IndexingSpiQueryTxSelfTest fix

911999a

code style

71150a2

code style

3941b80

Merge branch 'master' into ignite-27678-nostruct

53bce06

zstan requested a review from Copilot May 15, 2026 12:13

Copilot started reviewing on behalf of zstan May 15, 2026 12:13 View session

Copilot AI reviewed May 15, 2026

View reviewed changes

zstan added 2 commits May 15, 2026 18:15

improve test

ce3028e

debug needs

d222979

zstan commented May 18, 2026

View reviewed changes

zstan added 2 commits May 18, 2026 08:38

fix

6d6231c

copilot issue

88ef5cd

minor fix

b626bc7

zstan commented May 21, 2026

View reviewed changes

style

74775e8

alex-plekhanov reviewed Jun 23, 2026

View reviewed changes

Comment thread ...al/processors/cache/distributed/IgniteTxCacheWriteSynchronizationModesMultithreadedTest.java Outdated

Comment thread .../java/org/apache/ignite/internal/processors/cache/distributed/dht/GridDhtTxFinishFuture.java

zstan added 2 commits June 24, 2026 08:45

Merge branch 'master' into ignite-27678-nostruct

5ab180a

fix after review

1169f7e

alex-plekhanov reviewed Jun 24, 2026

View reviewed changes

zstan added 2 commits June 25, 2026 16:30

fix after review

6cbbe4c

fix

c8f4022

alex-plekhanov reviewed Jun 25, 2026

View reviewed changes

zstan added 3 commits June 26, 2026 08:28

remove force checkout

3584b4b

remove FULL_SYNC checks

629641a

fix after review

76f8159

alex-plekhanov reviewed Jun 26, 2026

View reviewed changes

fix after review

543130c

alex-plekhanov approved these changes Jun 26, 2026

View reviewed changes


		U.awaitQuiet(nodeLeftRegisteredOnBackup);

		// let`s wait until all discovery events have been processed on backup node.

Uh oh!

Conversation

zstan commented Mar 23, 2026

Uh oh!

sonarqubecloud Bot commented Apr 10, 2026

Quality Gate failed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zstan May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ignitetcbot commented May 20, 2026

TCBot Test Analysis

Possible Blockers (0)

New Tests (5)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented May 21, 2026

Quality Gate failed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 26, 2026

Possible compatibility issues. Please, check rolling upgrade cases

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zstan May 18, 2026 •

edited

Loading