Raft async storage writes by mjlshen · Pull Request #22223 · etcd-io/etcd

mjlshen · 2026-07-30T01:43:41Z

This PR was written in part with the assistance of generative AI, specifically the tests and designing the preliminary performance experiment whose results are in the description below. I am open to any other guidance around how best to take a benchmark before/after this PR's changes to provide what's expected.

Fixes #16291

What this changes

Integrates Raft asynchronous storage writes into etcdserver behind the alpha RaftAsyncStorageWrites feature gate, disabled by default.

When enabled, etcd:

processes append and apply work in FIFO order;
batches compatible contiguous WAL appends;
responds only after the corresponding write or application completes; and
preserves existing snapshot persistence and crash-recovery ordering.

Preliminary performance

I ran five repetitions before/after the changes using a three-member Docker Desktop cluster on my MacBook

benchmark put \
  --endpoints=etcd0:2379,etcd1:2379,etcd2:2379 \
  --conns=4 \
  --clients=64 \
  --total=100000 \
  --val-size=1024 \
  --key-space-size=1 \
  --report-perfdash

Results:

Metric	Gate off mean	Gate on mean	Change
Requests/second	10,022.8	12,530.1	+25.0%
Average latency	6.44 ms	5.16 ms	-19.9%
p50 latency	4.86 ms	3.84 ms	-21.0%
p90 latency	11.94 ms	9.20 ms	-22.9%
p99 latency	27.50 ms	22.20 ms	-19.3%
WAL fsync count	18,986	14,460	-23.8%
Backend commit count	301.0	247.6	-17.7%
Server process CPU	20.02 s	20.01 s	-0.06%

The preliminary results show an improvement! But emphasizing that I am open to any other guidance around how best to take a benchmark before/after this PR's changes to provide potentially more accurate numbers.

Replace the overloaded notify channel with explicit completion signals for snapshot persistence, Raft storage updates, and state machine application. Signed-off-by: Michael Shen <mishen@umich.edu>

kubernetes-prow · 2026-07-30T01:43:49Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mjlshen
Once this PR has been reviewed and has the lgtm label, please assign siyuanfoundation for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubernetes-prow · 2026-07-30T01:43:52Z

Hi @mjlshen. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mjlshen · 2026-07-30T01:47:00Z

+	// Bound the number of append messages accepted ahead of stable storage.
+	// The append worker coalesces compatible messages into a single WAL sync.
+	// This is a performance tuning value, not a Raft protocol limit.
+	asyncAppendQueueSize = 64


TODO: This value was selected based on local performance experiments and is not otherwise fixed. Once there's an agreed upon way to benchmark this change, this parameter should be re-swept before this PR should be merged.

mjlshen · 2026-07-30T02:13:40Z

+	// RaftAsyncStorageWrites enables Raft log persistence and state machine application
+	// through Raft's asynchronous storage write message interface.
+	// owner: @mjlshen
+	// issue: https://github.com/etcd-io/etcd/issues/16291
+	// alpha: v3.8
+	RaftAsyncStorageWrites featuregate.Feature = "RaftAsyncStorageWrites"


We may or may not want a feature flag (discussion), but it appears that we can't run antithesis tests on code that adds a feature flag and wants to use it immediately. If we choose to keep the feature flag, then I can make a separate MR so that:

The changes to this file are in the MR and merge in

The rest of this PRs changes, importantly tests/antithesis/config/* which try to enable the RaftAsyncStorageWrites feature flag immediately.

serathius · 2026-07-30T09:24:12Z

/ok-to-test

Looks promising but the most important test is the robustness one to confirm correctness of implementation.

serathius · 2026-07-30T09:25:26Z

 	SetMemberLocalAddr:           {Default: false, PreRelease: featuregate.Alpha},
 	FastLeaseKeepAlive:           {Default: true, PreRelease: featuregate.Beta},
 	PriorityRequest:              {Default: false, PreRelease: featuregate.Alpha},
+	RaftAsyncStorageWrites:       {Default: false, PreRelease: featuregate.Alpha},


For test sake, could you flip the flag to true so as we review PR we know that all tests are passing on the new implementation?

Sure thing! I was thinking of opening an issue so that Antithesis can handle this general workflow of adding a featuregate that defaults to false and validating it in the same PR as well if that sounds good to you.

codecov · 2026-07-30T09:53:43Z

Codecov Report

❌ Patch coverage is 86.51685% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.80%. Comparing base (9ff8b78) to head (3a64fae).
⚠️ Report is 24 commits behind head on main.

Files with missing lines	Patch %	Lines
server/etcdserver/raft.go	85.12%	24 Missing and 12 partials ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
server/etcdserver/bootstrap.go	`66.59% <100.00%> (+0.07%)`	⬆️
server/etcdserver/server.go	`83.69% <100.00%> (-0.01%)`	⬇️
server/features/etcd_features.go	`60.00% <ø> (ø)`
server/etcdserver/raft.go	`86.96% <85.12%> (-1.58%)`	⬇️

... and 19 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #22223      +/-   ##
==========================================
+ Coverage   69.70%   69.80%   +0.09%     
==========================================
  Files         449      449              
  Lines       38207    38421     +214     
==========================================
+ Hits        26631    26818     +187     
- Misses      10148    10165      +17     
- Partials     1428     1438      +10

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9ff8b78...3a64fae. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Add an alpha feature gate, reliable append and apply lanes, safe WAL batching, and focused synchronization tests for Raft asynchronous storage writes. Signed-off-by: Michael Shen <mishen@umich.edu>

Cover snapshot catch-up and failpoint-driven robustness with Raft asynchronous storage writes enabled. Signed-off-by: Michael Shen <mishen@umich.edu>

Exercise the alpha Raft asynchronous storage-write path in the one- and three-node Antithesis configurations. Signed-off-by: Michael Shen <mishen@umich.edu>

Signed-off-by: Michael Shen <mishen@umich.edu>

serathius · 2026-07-31T04:55:55Z

Seeing a lot of entry at index 1227 missing from unstable log; ignoring in https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/etcd-io_etcd/22223/pull-etcd-robustness-amd64/2083047905240813568

kubernetes-prow · 2026-07-31T05:03:36Z

@mjlshen: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-etcd-integration-2-cpu-amd64	`3a64fae`	link	true	`/test pull-etcd-integration-2-cpu-amd64`
pull-etcd-e2e-amd64	`3a64fae`	link	true	`/test pull-etcd-e2e-amd64`
pull-etcd-robustness-arm64	`3a64fae`	link	true	`/test pull-etcd-robustness-arm64`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

server: clarify raft apply synchronization

9e9f2c9

Replace the overloaded notify channel with explicit completion signals for snapshot persistence, Raft storage updates, and state machine application. Signed-off-by: Michael Shen <mishen@umich.edu>

kubernetes-prow Bot added area/robustness-testing area/testing labels Jul 30, 2026

kubernetes-prow Bot added the needs-ok-to-test label Jul 30, 2026

kubernetes-prow Bot added the size/XXL label Jul 30, 2026

mjlshen commented Jul 30, 2026

View reviewed changes

mjlshen mentioned this pull request Jul 30, 2026

Integrate raft's new feature (async write) into etcd #16291

Open

mjlshen commented Jul 30, 2026

View reviewed changes

kubernetes-prow Bot added ok-to-test and removed needs-ok-to-test labels Jul 30, 2026

serathius reviewed Jul 30, 2026

View reviewed changes

mjlshen force-pushed the raft-async-storage-writes branch 4 times, most recently from f7e8c91 to a1def08 Compare July 30, 2026 18:36

mjlshen changed the title ~~Raft async storage writes~~ Draft: Raft async storage writes Jul 31, 2026

mjlshen changed the title ~~Draft: Raft async storage writes~~ Raft async storage writes Jul 31, 2026

mjlshen marked this pull request as draft July 31, 2026 03:58

kubernetes-prow Bot added the do-not-merge/work-in-progress label Jul 31, 2026

mjlshen added 4 commits July 30, 2026 21:15

server: integrate raft async storage writes

79a2226

Add an alpha feature gate, reliable append and apply lanes, safe WAL batching, and focused synchronization tests for Raft asynchronous storage writes. Signed-off-by: Michael Shen <mishen@umich.edu>

tests: exercise raft async storage writes

5a9f07e

Cover snapshot catch-up and failpoint-driven robustness with Raft asynchronous storage writes enabled. Signed-off-by: Michael Shen <mishen@umich.edu>

tests: enable raft async writes in Antithesis

6be1538

Exercise the alpha Raft asynchronous storage-write path in the one- and three-node Antithesis configurations. Signed-off-by: Michael Shen <mishen@umich.edu>

Temporarily enable RaftAsyncStorageWrites to validate in CI

3a64fae

Signed-off-by: Michael Shen <mishen@umich.edu>

mjlshen force-pushed the raft-async-storage-writes branch from a1def08 to 3a64fae Compare July 31, 2026 04:24

mjlshen marked this pull request as ready for review July 31, 2026 04:31

kubernetes-prow Bot removed the do-not-merge/work-in-progress label Jul 31, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Raft async storage writes - #22223

Raft async storage writes#22223
mjlshen wants to merge 5 commits into
etcd-io:mainfrom
mjlshen:raft-async-storage-writes

mjlshen commented Jul 30, 2026 •

edited

Loading

Uh oh!

kubernetes-prow Bot commented Jul 30, 2026

Uh oh!

kubernetes-prow Bot commented Jul 30, 2026

Uh oh!

mjlshen Jul 30, 2026

Uh oh!

mjlshen Jul 30, 2026 •

edited

Loading

Uh oh!

serathius commented Jul 30, 2026 •

edited

Loading

Uh oh!

serathius Jul 30, 2026

Uh oh!

mjlshen Jul 30, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

serathius commented Jul 31, 2026 •

edited

Loading

Uh oh!

kubernetes-prow Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mjlshen commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Preliminary performance

Uh oh!

kubernetes-prow Bot commented Jul 30, 2026

Uh oh!

kubernetes-prow Bot commented Jul 30, 2026

Uh oh!

mjlshen Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

mjlshen Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serathius commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serathius Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

mjlshen Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

serathius commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kubernetes-prow Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

mjlshen commented Jul 30, 2026 •

edited

Loading

mjlshen Jul 30, 2026 •

edited

Loading

serathius commented Jul 30, 2026 •

edited

Loading

mjlshen Jul 30, 2026 •

edited

Loading

codecov Bot commented Jul 30, 2026 •

edited

Loading

serathius commented Jul 31, 2026 •

edited

Loading

kubernetes-prow Bot commented Jul 31, 2026 •

edited

Loading