WriteAheadLog (6.x): make a committed write durable across a graceful restart with a batched FlushInterval

[msallin]

DotNext.Net.Cluster 6.3.0, .NET 10. RaftCluster (TCP) with a hand-assigned WriteAheadLog as AuditTrail over a SimpleStateMachine, one cluster per pod. Membership and restart leadership are already solved with a persistent configuration storage (per #207); this question is purely about WAL durability.

I expose a synchronous storage API, so a write must be durable before it returns, or at minimum survive a graceful (SIGTERM) restart. Completion is detected with a waiter completed inside SimpleStateMachine.ApplyAsync: I await ReplicateAsync(...) then await the waiter, so a write returns once the entry is applied.

• With FlushInterval = TimeSpan.Zero (flush on every commit in the background) this is reliable: a node restarted from the same WAL recovers everything via InitializeAsync. But it fsyncs on every commit.
• With a positive FlushInterval (to batch flushes for throughput), a graceful shutdown loses the un-snapshotted tail. A single committed entry written then immediately followed by DisposeAsync is not recovered after a restart, even when I call FlushAsync() before disposing the cluster and the log. A longer run (e.g. 200 writes) is recovered, because the periodic flusher and snapshots cover it.

Questions:

1. Is there a supported way to make a committed write durable on return — awaiting the flush, not just the apply — so the caller knows it is on stable storage?
2. With a hand-assigned WriteAheadLog, what is the correct flush-on-graceful-shutdown sequence so a positive FlushInterval does not lose the most recent committed-but-unflushed entries? Is FlushAsync() the right call, and where must it run relative to RaftCluster.StopAsync() and disposal?
3. Or is FlushInterval = TimeSpan.Zero the only mode that guarantees a committed entry survives a restart, with any positive interval being best-effort (up to one interval lost on a stop)?

Goal: one cluster per pod that can batch flushes for throughput yet never lose a committed write across a graceful restart. Happy to provide a minimal repro.

[sakno]

Not doing fsync on every apply is intentional behavior. Let me describe the motivation. Durability is the responsibility of the whole cluster. Log record becomes durable when it's replicated to the quorum. However, it doesn't mean that this log record is written to the disk. It means that the durability is guaranteed by the quorum.

For performance reasons, applied and committed state of a log record are separated. The leader is not interested in applied state. Log record turns into committed state it is replicated to the majority (quorum). Committed then is flushed in the background, in parallel with Apply action. WriteAheadLog creates a checkpoint that contains the index of the last flushed (and committed) log record. If leader is crashed, and then restarted, it can recover lost log record from a new leader, so your state machine can recover its state normally.

[sakno]

This optimization allows the client to get ACK earlier than fsync, which is fine, because the log record is replicated and committed to the quorum. This doesn't work well with 1-node setup (which is inappropriate for production clusters). In other words, WriteAheadLog guarantee the durability of the last write while the whole cluster alive. Then the whole cluster is crashed, WriteAheadLog can recover from some safe point, which is not the last write.

[sakno]

I added test in the commit aea3ec1 to make sure that FlushAsync behaves correctly. When FlushInterval is set, FlushAsync doesn't flush actually, it just waits for the signal from the background flusher process, which means that FlushAsync might suspend the caller up to FlushInterval. One more thing - if the log record is not committed, even FlushAsync cannot guarantee anything.

[msallin]

Thanks, this fully clarifies it.

As an FYI on the use case: I use WriteAheadLog as the Raft audit trail behind a database-free Hangfire job storage (msallin/hangfire-raft, published as Hangfire.Community.Raft), so the application nodes are the Raft cluster and there is no SQL or Redis. Thanks again.