Missing snapshot-directory fsync can make a member fail after a crash

[gyuho]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via the report form.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

When a member receives a database snapshot, SaveDBFrom syncs the snapshot file and renames it into member/snap, but does not sync the containing directory. After a power loss, the *.snap.db directory entry may be lost even though the file was synced.

The WAL can retain the corresponding snapshot record. On restart, etcd finds that record but not the database snapshot and panics:

failed to find database snapshot file (snap: snapshot file doesn't exist)

What did you expect to happen?

etcd should make the renamed database snapshot durable before recording or processing the corresponding Raft snapshot. A directory-sync failure should stop snapshot handling and return an error.

How can we reproduce it (as minimally and precisely as possible)?

1. Start a three-member cluster with aggressive snapshotting.
2. Stop or isolate one follower, write enough data to compact the Raft log, then restore the follower so it must receive a database snapshot.
3. Pause the follower after *.snap.db is renamed but before the snapshot directory is synced.
4. Kill etcd, then abruptly reboot the machine without syncing disks, for example with echo b > /proc/sysrq-trigger.
5. Restart the member. If the directory entry was lost but the WAL snapshot record survived, etcd panics because the referenced database snapshot is missing.

Environment and affected version

• Three-member etcd cluster on Linux/amd64
• Reproduced on AWS EC2 with the member data directory on a dedicated non-journaled ext2 EBS volume
• Affected main revision: 23a4e406a2e70a807486b4c40a9e24da493886bf (the base of PR fix: fsync snap directory when saving a received snapshot db #22314)
• Relevant configuration: aggressive snapshotting and one follower forced far enough behind to receive a full database snapshot

Anything else we need to know?

Similar issues

Issues #11949 and #14569 show the same startup failure but do not identify the missing directory fsync.

Proposed fix and backport

PR #22314 fixes this by syncing the snapshot directory on both the rename and existing-file paths, and propagating sync failures before Raft processing.

We need to backport this fix to the v3.5, v3.6, and v3.7 release branches.

Etcd version (please run commands below)

Details

Affected main revision: 23a4e406a2e70a807486b4c40a9e24da493886bf (the base of PR #22314). The bug predates this revision; the same code path exists in the v3.5, v3.6, and v3.7 release branches.

$ etcd --version
v3.7.1

Etcd configuration (command line flags or environment variables)

Details

Three-member cluster on Linux/amd64, reproduced on AWS EC2 with the member data directory on a dedicated non-journaled ext2 EBS volume. Aggressive snapshotting was used to force snapshot transfer; otherwise defaults.

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Details

Not applicable; the failure is at startup after a power loss, before the member can serve.

Relevant log output

panic: failed to recover v3 backend from snapshot: failed to find database snapshot file (snap: snapshot file doesn't exist)

[gyuho]

/assign