Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IGNITE-23661 Eliminate race in onBeforeApply within StandaloneMetaStorageManager #4723

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

sanpwc
Copy link
Contributor

@sanpwc sanpwc commented Nov 14, 2024

https://issues.apache.org/jira/browse/IGNITE-23661

Long story short, the stack trace of a problem is following:

  1. assertTrue(waitForCondition(() -> newLogicalTopology.equals(finalDistributionZoneManager.logicalTopology()) fails because finalDistributionZoneManager.logicalTopology() is not updated because

  2. notificationFuture in WatchProcessor#notifyWatches() is not completed because notifyUpdateRevisionFuture is not completed because

  3. assert causalityToken > lastCompleteToken in org.apache.ignite.internal.causality.IncrementalVersionedValue#completeInternal isn't matched (but not logged though) because

  4. watch events are reordered by revision despite the fact that they are properly ordered by timestamp, see org.apache.ignite.internal.metastorage.server.NotifyWatchProcessorEvent#compareTo for more details.

  5. All in all, order mismatch is a result of non-thread safe StandaloneMetaStorageManager. Both onBeforeApply and command processing within listener should be thread-safe. That's why there's a race between operation safeTime assignment and revision calculation. onBeforeApply is guarded by group specific monitor, precisely synchronized (groupIdSyncMonitor(request.groupId()))
    See ActionRequestProcessor.handleRequestInternal for more details. Command processing on its turn is expected to be processed under raft umbrella, meaning in single-thread environment.

And corresponding synchronisation was missed in StandaloneMetaStorageManager.

@sanpwc sanpwc changed the title IGNITE-23661 Add some thread-safety to the StandaloneMetaStorageManager IGNITE-23661 Eliminate race in onBeforeApply within StandaloneMetaStorageManager Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants