Why AsyncWriteJournal.Resequencer is needed?

[migesok]

Hi!

During an investigation of a latency issue in our code using classic persistent actors, we found that the "lost" time is spent waiting inside AsyncWriteJournal.Resequencer. As I understood, it reorders journal write responses to be in the same order as the requests. The order is global (per plugin instance). Our problem was gone when we patched the journal plugin logic to remove Resequencer. At the same time no other issues arose during tests.

Why this ordering is needed? Why it has to be global and not, say, per persistence ID? Per actor instance ID?

What I found so far myself:

• no additional pekko-persistence tests fail even if I remove the resequencer (though some test fail for me even without changes because of the outdated leveldb JNI library)
• that piece of code is almost as old as the Akka itself - it was there in the first Akka Persistence commits:
  +per #3641 Storage plugin API akka/akka#1751
  I also found that the logic was added to the original eventsourced project (from which it was ported to Akka) when it added support for "async" journal plugins (not sure what that means 100%):
  krasserm/eventsourced@f200b12
• journal resequencing logic is mentioned in defer and deferAsync implementations so I assume they are related somehow

[pjfanning]

Thanks @migesok for raising this.

I'm not aware of why this Resequencer is used. If, as you suggest, the resequencer doesn't seem to affect the results of any tests, maybe, we could add a config setting that allows it to be skipped. My preference, for backward compatibility, is that the Resequencer is used by default. It might be best not to do anything until other contributors have time to provide their opinions or context.

Have you tried running any of the tests in modules like pekko-persistence-jdbc?

[migesok]

"very funky classic persistent actor implementations" - that's what I'm dealing with, I believe. We have persistAsync and deferAsync in ungodly combinations. But I can't make it fail in a test even without resequencer which makes me think there is some other built-in mechanism which makes it work. Or I don't understand the type of failures resequencer was made to avert and I'm yet to hit it.

My current assumption is that resequencer was meant to keep the order of async callbacks preserved (persist and defer) but they seem to be in order even without it. Even when I deliberately reorder journal actor responses to be in a wrong order.

We already have the fix removing resequencer applied locally for some of our projects. I'm trying to understand whether it is safe to extend it company-wise and whether it should be contributed to Pekko. For this reason I'm trying to understand the consequences of removing resequencer.

[ptrdom]

I think this is what can happen if there is no resequencer:

1. Actor defers two writes and then persists them.
2. No resequencer, so second write might complete first, and first write can compete second or fail:
3. Handlers are executed in order, but writes are completed out of order, so you are executing first write's handler on completion second write.

So if user-level code is sensitive to this behaviour, which I would call a bug, then it can break. The way I see it this is more relevant to failure cases - using the above example, first write might not even complete, but its handler is being triggered by the second write succeeding.

[migesok]

@ptrdom can you help me a bit with translating this into PersistentActor logic? By the first point you mean calling deferAsync twice and then doing persistAsync inside their respective deferAsync callbacks?

I made a small "chaos" test harness for this. If I can put it into a chain of persist and defer calls I can try to confirm it.

So far I wasn't able to make PersistentActor run 2 actual journal asyncWriteMessages operations in parallel. It seems to collect all outstanding writes and dump then in 1 call, no matter what I do. Probably I just don't understand it that well.

[pjfanning]

I googled around and the only real discussion about Akka/Pekko Resequencers that I could find was in this book.

https://github.com/yyccR/papers/blob/master/genetic%20algorithm/Reactive.Messaging.Patterns.with.the.Actor.Model.Applications.and.Integration.in.Scala.and.Akka.pdf

Mainly about Actors but it does discuss Persistent Actors too.

[migesok]

After some experiments and diving into Akka 2.6 and Pekko code, I came to this:

I'm almost certain that neither for a classic persistent actor, nor for a typed, one can observe differences in behaviour between AsyncWriteJournal with and without resequencer. It is because

• persistent actors inside 1 "incarnation" (between restarts) never do 2 asyncWriteMessages calls simultaneously, they are always done 1 after another
• persistent actors discard and ignore asyncWriteMessages result responses from requests initiated by different "incarnations" of the same actor (courtesy of actorInstanceId)

So the only way to observe differences in behaviour with and without resequencer is to stage an interplay between 2 persistent actors, giving them commands to persist events, and observing order in which replies come back. It should never be visible if you look at isolated actors.

On one hand, resequencer can be dropped altogether, on another - could there be systems around which really depend on such intricate ordering guarantees which manifests only when several persistent actors work together? Possible but unlikely.

To be on the safe side, resequencer switch could be a plugin level config option.

My current plan is to roll out the change first inside our codebase. We are still running mostly on Akka 2.6 with Pekko migration in progress, using in-house journal and snapshot plugins. Then we will try to backport it to Pekko, if maintainers agree to take it.