Increase supported storage size to match K8s scalability requirements

[serathius]

What would you like to be added?

A concrete roadmap to scale etcd storage capacity to 30 GB (15 GB active dataset plus 15 GB revision history), enabling etcd to reliably support Kubernetes large-resource scalability requirements.

We propose the following deliverables:

1. Standardized storage benchmarks and correctness tests: Enable experimentation to improve storage performance, by establishing a standard benchmark and test harness validating correctness and compaction/defrag overhead.
2. Storage layer decoupling: Cleanup the etcd storage backend interfaces to allow easier iteration.
3. Prototype optimizations to storage: Prototype and benchmark optimization storage engine including possibility of alternative storage engines.

Why is this needed?

etcd provides a reliable foundation for Kubernetes control planes, successfully powering 5,000-node clusters with 150,000 pods under standard configurations. But the original scalability envelope relied on the assumption that pods are minimal 4 KB objects (kubernetes/kubernetes#134375).

Discrepancy Between Test Assumptions and Real-World Scale

Modern Kubernetes architectures with sidecars, init containers, and rich metadata routinely generate pod objects averaging 10 KB to 20 KB, occasionally reaching over 70 KB.

To close this gap, Kubernetes established Resource Size as a formal scalability dimension in kubernetes/kubernetes#134375 and defined an exemplar pod shape envelope in kubernetes/kubernetes#138415. Upstream findings show that:

• Object metadata and controller field ownership tracked in managedFields often double the effective pod size
• Adding a sidecar or init container adds 1+KB of status data for pod.
• Kubernetes release-blocking tests have raised pod resource size from 800 MB to 1.5 GB.

Single-Resource Storage Limits

While Kubernetes supports sharding etcd clusters across distinct API resources, sharding cannot mitigate the growth of a single resource. Pods alone represent the primary bottleneck:

• Active state: 150,000 pods at an average size of 100 KB require 15 GB of latest-state storage.
• Revision history: At 1,000 write QPS and a 150-second compaction window, history requires an additional 15 GB.
• Total target capacity: 30 GB of sustained database volume.

Observed Failure Modes in etcd at Scale

Testing etcd at this storage scale reveals critical stability limits rooted in bbolt architecture:

• Quorum instability from maintenance locks: File fragmentation and freelist bloat require periodic compaction and defragmentation. Defragmentation acquires an exclusive database lock, stalling read and write transactions and risking Raft heartbeat timeouts.
• Member recovery failure: When an etcd follower falls behind and requires a snapshot, streaming and applying a multi-gigabyte snapshot over the network takes longer than the compaction window. The follower receives an outdated snapshot and enters an unrecoverable catch-up loop.

Micro-Benchmark Findings

Isolated micro-benchmarks on bbolt illustrate the performance cliff:

• Under a baseline workload of 150,000 keys at 10 KB value size, bbolt sustains approximately 20,000 write QPS.
• Compacting that dataset takes over 20-30 seconds, dropping throughput by 50% to roughly 10,000 QPS.
• Running defrag takes over 30-40s and fully locks the database for the duration of the operation, while not providing meaningful db file reduction.

etcd $ TMPDIR=$(pwd) go test --run ^$ --bench BenchmarkStorageWrite ./server/storage/mvcc/ --benchtime=10x
goos: linux
goarch: amd64
pkg: go.etcd.io/etcd/server/v3/storage/mvcc
cpu: AMD Ryzen Threadripper PRO 3945WX 12-Cores     
BenchmarkStorageWrite/FS=ext4/Compaction=false/Defrag=false/Keys=150000/ValueSize=10000-24         	      10	5797744538 ns/op	     16143 disk_mb	        11.28 disk_overhead	     25872 write_qps
BenchmarkStorageWrite/FS=ext4/Compaction=true/Defrag=false/Keys=150000/ValueSize=10000-24          	      10	9393507668 ns/op	        25.22 avg_compaction_duration_s	         5.000 compaction_count	     14675 disk_mb	        10.26 disk_overhead	     15968 write_qps
BenchmarkStorageWrite/FS=ext4/Compaction=false/Defrag=true/Keys=150000/ValueSize=10000-24          	      10	7518997072 ns/op	        40.95 avg_defrag_duration_s	         2.000 defrag_count	     16143 disk_mb	        11.28 disk_overhead
BenchmarkStorageWrite/FS=ext4/Compaction=true/Defrag=true/Keys=150000/ValueSize=10000-24           	      10	9301968190 ns/op	        23.65 avg_compaction_duration_s	        30.57 avg_defrag_duration_s	         5.000 compaction_count	         2.000 defrag_count	     14675 disk_mb	        10.26 disk_overhead

Addressing these storage bottlenecks will allow etcd to scale comfortably alongside modern Kubernetes workload profiles.

Success metrics

1. Database overhead after compacting history, the size on disk divided by size of state. Target: always below 1x.
2. Reduce impact of maintenance operations on write throughput and other operations. Target less than 10%

Storage Performance Comparison: 150k Pods (10 KB payload, 10x writes ~ 15 GB logical data)

This represents 1/10th of the production state target scale.

Dimension	bbolt (B+tree)	badger (LSM Engine)	pebble (LSM Engine)	Comparison
Disk Footprint (Baseline)	14.7 GB (1.01x logical size)	7.3 GB (0.50x logical size)	2.35 GB (0.16x logical size)	Pebble disk footprint is 6.25x smaller (compression)
Write throughput (Baseline)	~30,000 QPS	~13,500 QPS	~34,000 QPS	Pebble write throughput is ~1.7x faster than bbolt
Compaction Duration	2.0 – 7.1 s	4.4 – 26.8 s	0.8 – 2.6 s	Pebble compacts ~2.7x faster than bbolt
Write throughput (with Compaction)	~15,900 QPS (-47% drop)	~10,000 QPS (-26% drop)	~35,700 QPS (5% increase)	Pebble compaction improves performance
Disk Footprint (after Compaction)	2.03x logical size	3.16x logical size	0.62x logical size	Pebble keeps disk footprint under of logical size
Defragmentation Duration	70.8 – 113.6 s	25.1 – 78.1 s	14.2 s
Write throughput (with Defrag)	~100 QPS (-99.7% drop)	~16,300 QPS (20% increase)	~38,400 QPS (12% increase)	For LSM engines defragmenting improves performance
Disk Footprint (after Defrag)	1.1x logical size	3.16x logical size	0.2x logical size	Pebble reclaims space more efficiently
Read Throughput (Get)	260k QPS	90k QPS	40k QPS	Compression slows down reads, but doesn't matter as much
Read Throughput (Range)	200-300 QPS	65-100 QPS	40-60 QPS	Compression slows down reads, but doesn't matter as much
Watch Throughput (Resync)	5-50ms	0.8ms - 2.4s	0.8-100ms	Compression slows down reads, but doesn't matter as much

Note: the badger and pebble implementation is not fully optimized as it was mostly vibed, goal is to show bbolt performance gap to modern storage engine. To ensure validity the correctness was validated via linearizability harness to ensuring compatible response with bbolt implementation.

/cc @ahrtr @fuweid

[ah8ad3]

I am also interested, please tag if there is anything i can do.

[tjungblu]

/cc
/cc @dusk125

[ahrtr]

30GB isn't a problem at all for bbolt, please see etcd-io/bbolt#401 (comment). I thought about adding an option (e.g. --pagesize) to etcd to let users to configure the bbolt page size. My test result proved that it has impact on performance (64KB pagesize is an optimal choice for 1KB value size), but the value size was only 1K. We need to benchmark etcd by tuning the page size and value size. Lots of work.

But the defragmentation (or compact of bbolt) is indeed a hard problem that we need to resolve.

[serathius]

30GB isn't a problem at all for bbolt, please see etcd-io/bbolt#401 (comment). I thought about adding an option (e.g. --pagesize) to etcd to let users to configure the bbolt page size.

Right, but the problem is the effective file size. We also should look on on disk overhead (total/in use size) and look at on disk compression of data if we want to be able to effectively exchange snapshots between members.

[fuweid]

Link etcd-io/bbolt#939 (comment) for compaction

[serathius]

To understand realistic performance target I have created robustness tests inspired correctness testing for MVCC #22339 and used that to vibe replacing bbolt with alternative storage options like pebble and badger.

I find it sad that for other storages, operations like defrag not only don't lock the database, but overall improve write throughput performance. Will see if I can take it as far as replacing bbolt across whole etcd and passing robustness tests.

[vivekpatani]

@serathius I'd be interested in helping out with this, do you think there are any action items I can help out with? Thanks for doing this.