perf(core): block snapshots with layers #2664

edg-l · 2025-05-05T13:19:49Z

Motivation

Adds block snapshots using disk and diff layers. Improving performance by reducing trie database queries.

Description

Heavily inspired by https://github.com/ethereum/go-ethereum/blob/master/core/state/snapshot/

Uses a base disk layer at block X, adding diff layers as blocks come at X + 1, X + 2, until the cap of max layers is reached, when that happens, flattening of layers is done, moving the bottom disk layer closer to the head block like a moving window.

When one queries for example for the account state at HEAD-4 block. The snapshot tree requests the snapshot at HEAD-4, then using a bloom filter which aggregates all accounts from previous diff layers, we can now if we skip to the disk layer directly or traverse the diff layers. Traversing the diff layers is faster than quering the database.

The disk layer has a flat database table mapping hashes to accounts and storages

Missing features:

Save diff layers to the database

Last load tests done:

pr commit: a8ca81f

macOS M3 36gb

pr load-test:

First blocks: 0.58, 0.55, 0.54

Last blocks 0.38, 0.37, 0.32

pr load-test-erc20:

First blocks: 1.04, 0.86, 0.83, 0.99, 0.99

Last blocks: 0.58, 0.58, 0.56

Main:

load-test:

First blocks: 0.38, 0.23, 0.23, 0.30, 0.27
Last blocks: 0.21, 0.20, 0.12

load-test-erc20:

First blocks: 0.60, 0.53, 0.47, 0.48, 0.58, 0.56, 0.54
Last blocks: 0.40, 0.40, 0.36

github-actions · 2025-05-05T13:20:55Z

Lines of code report

Total lines added: 1476
Total lines removed: 0
Total lines changed: 1476

Detailed view

+---------------------------------------------+-------+------+
| File                                        | Lines | Diff |
+---------------------------------------------+-------+------+
| ethrex/crates/blockchain/blockchain.rs      | 513   | +10  |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/api.rs                | 287   | +17  |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/error.rs              | 54    | +3   |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/lib.rs                | 13    | +2   |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/snapshot/cache.rs     | 16    | +16  |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/snapshot/difflayer.rs | 282   | +282 |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/snapshot/disklayer.rs | 105   | +105 |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/snapshot/error.rs     | 24    | +24  |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/snapshot/mod.rs       | 6     | +6   |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/snapshot/tree.rs      | 742   | +742 |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/store.rs              | 1394  | +102 |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/store_db/in_memory.rs | 655   | +31  |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/store_db/libmdbx.rs   | 1474  | +68  |
+---------------------------------------------+-------+------+
| ethrex/crates/storage/store_db/redb.rs      | 1318  | +68  |
+---------------------------------------------+-------+------+

github-actions · 2025-05-07T09:56:17Z

Benchmark for `c144ed8`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	35.8±0.83ms	38.8±1.29ms	+8.38%
Trie/cita-trie insert 1k	3.7±0.05ms	3.9±0.04ms	+5.41%
Trie/ethrex-trie insert 10k	132.1±3.59ms	128.2±3.19ms	-2.95%
Trie/ethrex-trie insert 1k	11.3±0.46ms	11.2±0.39ms	-0.88%

github-actions · 2025-05-07T11:50:17Z

Benchmark Block Execution Results Comparison Against Main

Command	Mean [s]	Min [s]	Max [s]	Relative
`base`	178.872 ± 4.763	175.598	189.596	1.00
`head`	184.060 ± 2.756	180.037	189.569	1.03 ± 0.03

github-actions · 2025-05-07T12:42:25Z

Benchmark for `084ba79`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	33.7±1.29ms	41.0±2.02ms	+21.66%
Trie/cita-trie insert 1k	3.7±0.03ms	3.7±0.11ms	0.00%
Trie/ethrex-trie insert 10k	138.8±3.52ms	134.8±3.78ms	-2.88%
Trie/ethrex-trie insert 1k	11.3±0.13ms	11.3±0.31ms	0.00%

github-actions · 2025-05-07T13:01:29Z

Benchmark for `9cc084f`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	34.8±0.47ms	35.5±0.34ms	+2.01%
Trie/cita-trie insert 1k	3.6±0.09ms	3.7±0.04ms	+2.78%
Trie/ethrex-trie insert 10k	124.1±1.42ms	123.5±1.13ms	-0.48%
Trie/ethrex-trie insert 1k	11.1±0.16ms	11.2±0.36ms	+0.90%

github-actions · 2025-05-07T14:42:22Z

Benchmark for `23432e3`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	35.6±1.00ms	36.7±1.31ms	+3.09%
Trie/cita-trie insert 1k	3.7±0.03ms	3.7±0.05ms	0.00%
Trie/ethrex-trie insert 10k	128.8±2.79ms	128.7±2.03ms	-0.08%
Trie/ethrex-trie insert 1k	11.1±0.06ms	11.2±0.64ms	+0.90%

github-actions · 2025-05-07T14:43:39Z

Benchmark for `c6fa39c`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	36.7±1.94ms	35.8±0.68ms	-2.45%
Trie/cita-trie insert 1k	3.7±0.02ms	3.7±0.04ms	0.00%
Trie/ethrex-trie insert 10k	130.7±2.53ms	130.1±2.32ms	-0.46%
Trie/ethrex-trie insert 1k	11.0±0.03ms	11.4±0.47ms	+3.64%

github-actions · 2025-05-08T13:03:16Z

Benchmark for `3275d89`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	39.5±2.00ms	34.6±1.35ms	-12.41%
Trie/cita-trie insert 1k	3.7±0.05ms	3.7±0.08ms	0.00%
Trie/ethrex-trie insert 10k	134.2±2.33ms	131.9±2.14ms	-1.71%
Trie/ethrex-trie insert 1k	11.3±0.13ms	11.1±0.05ms	-1.77%

github-actions · 2025-05-08T14:20:00Z

Benchmark for `795388a`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	35.7±0.92ms	35.8±0.63ms	+0.28%
Trie/cita-trie insert 1k	3.6±0.04ms	3.7±0.04ms	+2.78%
Trie/ethrex-trie insert 10k	126.8±3.04ms	125.6±4.53ms	-0.95%
Trie/ethrex-trie insert 1k	11.0±0.11ms	11.0±0.07ms	0.00%

github-actions · 2025-05-08T14:24:28Z

Benchmark for `e40a6a8`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	38.3±2.56ms	38.9±2.58ms	+1.57%
Trie/cita-trie insert 1k	3.7±0.10ms	3.7±0.04ms	0.00%
Trie/ethrex-trie insert 10k	135.3±2.41ms	135.1±3.31ms	-0.15%
Trie/ethrex-trie insert 1k	11.3±0.12ms	11.4±0.40ms	+0.88%

github-actions · 2025-05-29T15:39:06Z

Benchmark for `4fe7658`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	37.2±3.59ms	41.8±3.09ms	+12.37%
Trie/cita-trie insert 1k	3.6±0.07ms	3.7±0.10ms	+2.78%
Trie/ethrex-trie insert 10k	62.1±2.48ms	62.3±2.36ms	+0.32%
Trie/ethrex-trie insert 1k	6.6±0.07ms	6.7±0.11ms	+1.52%

pablodeymo · 2025-05-29T18:25:27Z

crates/blockchain/payload.rs

@@ -547,6 +547,7 @@ impl Blockchain {
        context.payload.header.requests_hash = context.requests_hash;
        context.payload.header.gas_used = context.payload.header.gas_limit - context.remaining_gas;
        context.account_updates = account_updates;
+


Try not to include pretifier changes in such a big PR. This makes review more complicated.

github-actions · 2025-05-29T18:40:10Z

Benchmark for `7349e91`

Click to view benchmark

Test	Base	PR	%
Trie/cita-trie insert 10k	38.3±2.53ms	37.6±1.68ms	-1.83%
Trie/cita-trie insert 1k	3.7±0.05ms	3.6±0.14ms	-2.70%
Trie/ethrex-trie insert 10k	60.4±1.63ms	60.0±2.18ms	-0.66%
Trie/ethrex-trie insert 1k	6.7±0.06ms	6.5±0.08ms	-2.99%

pablodeymo · 2025-05-29T18:43:13Z

cmd/ethrex_replay/src/rpc/mod.rs

-    if trie.get(&hash_address(address))?.is_none() {
+    if trie.get(hash_address(address))?.is_none() {


I think it's better to fix clippy warnings in a ad hoc PR and then rebase main to this one.

Thanks for pointing this out 😄.

This wasn't happening in main, I figured out this was due to a change in the signature of Trie::get. Originally, this function received a &PathRLP (type alias to Vec<u8>), and this PR
changed said parameter to impl AsRef<u8>. I don't think this change is worthwile and also affected a lot of tests, incrementing the diff count and diff view unnecessarily. So I went and restored the original parameter.

…o edgl_snapshots_v2

This reverts commit 795fc25.

crates/blockchain/blockchain.rs

edg-l added 2 commits May 5, 2025 13:48

progress

e4cc248

progress 2

2169a9f

edg-l mentioned this pull request May 5, 2025

perf(l1): add block snapshots with account states #2652

Closed

edg-l added 6 commits May 6, 2025 11:30

Merge branch 'main' into edgl_snapshots_v2

8505f24

implement get_storage for disklayer

f6eaaf8

progress

97c3538

progress

e465069

traverse diff layers for account

bf54228

add storage to layers

97acea6

edg-l changed the title ~~WIP: Snapshots~~ feat(core): block snapshots with layers May 7, 2025

edg-l added the performance label May 7, 2025

edg-l changed the title ~~feat(core): block snapshots with layers~~ perf(core): block snapshots with layers May 7, 2025

add snapshot tree

b5dee6e

Merge branch 'main' into edgl_snapshots_v2

6c5a19c

dont use store directly

8ee99e5

edg-l added 2 commits May 7, 2025 16:39

start integrating

0639e73

comments

ee229de

integrate

74a473b

move snapshot creating to its own method and call it from store block

c9a1e5a

Merge branch 'main' into edgl_snapshots_v2

51213cd

Merge branch 'main' into edgl_snapshots_v2

5af46d2

pablodeymo reviewed May 29, 2025

View reviewed changes

lint: clippy

758eaf6

pablodeymo reviewed May 29, 2025

View reviewed changes

fkrause98 and others added 14 commits May 29, 2025 19:16

chore: restore original "PathRLP" parameter

e70124c

lint: clippy

576d4f9

lint: fmt

352885f

fix: snapshot removed storage value

81f8374

Merge branch 'main' into edgl_snapshots_v2

b39c7e5

Merge branch 'main' into edgl_snapshots_v2

e525607

Merge branch 'main' into edgl_snapshots_v2

7617438

Merge remote-tracking branch 'origin/main' into edgl_snapshots_v2

fb40df1

Merge branch 'edgl_snapshots_v2' of github.com:lambdaclass/ethrex int…

e88a1de

…o edgl_snapshots_v2

add some basic tests

9dd88cd

Merge branch 'edgl_snapshots_v2' of github.com:lambdaclass/ethrex int…

0bbf45f

…o edgl_snapshots_v2

Merge branch 'edgl_snapshots_v2' of github.com:lambdaclass/ethrex int…

bcb0069

…o edgl_snapshots_v2

add docs

5e18cfc

lint

f06b9e1

jrchatruc mentioned this pull request Jun 4, 2025

Test PR: Simple key value snapshot #2866

Closed

fkrause98 added 5 commits June 4, 2025 17:13

feat: remove feature snapshots

d4e264f

ci: invalidate cache

795fc25

Revert "ci: invalidate cache"

61945e4

This reverts commit 795fc25.

Merge branch 'main' into edgl_snapshots_v2

6dd086d

fix: build

26da7c7

MegaRedHand reviewed Jun 4, 2025

View reviewed changes

crates/blockchain/blockchain.rs Outdated Show resolved Hide resolved

edg-l added 3 commits June 11, 2025 10:38

return error on error

491d556

conflicts and better error handling

34e00c9

expect

6dc3fbd

		if trie.get(&hash_address(address))?.is_none() {
		if trie.get(hash_address(address))?.is_none() {

perf(core): block snapshots with layers #2664

Are you sure you want to change the base?

perf(core): block snapshots with layers #2664

Uh oh!

Conversation

edg-l commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Lines of code report

Uh oh!

github-actions bot commented May 7, 2025

Benchmark for c144ed8

Uh oh!

github-actions bot commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Block Execution Results Comparison Against Main

Uh oh!

github-actions bot commented May 7, 2025

Benchmark for 084ba79

Uh oh!

github-actions bot commented May 7, 2025

Benchmark for 9cc084f

Uh oh!

github-actions bot commented May 7, 2025

Benchmark for 23432e3

Uh oh!

github-actions bot commented May 7, 2025

Benchmark for c6fa39c

Uh oh!

github-actions bot commented May 8, 2025

Benchmark for 3275d89

Uh oh!

github-actions bot commented May 8, 2025

Benchmark for 795388a

Uh oh!

github-actions bot commented May 8, 2025

Benchmark for e40a6a8

Uh oh!

github-actions bot commented May 29, 2025

Benchmark for 4fe7658

Uh oh!

pablodeymo May 29, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 29, 2025

Benchmark for 7349e91

Uh oh!

pablodeymo May 29, 2025

Choose a reason for hiding this comment

Uh oh!

fkrause98 May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

edg-l commented May 5, 2025 •

edited

Loading

github-actions bot commented May 5, 2025 •

edited

Loading

Benchmark for `c144ed8`

github-actions bot commented May 7, 2025 •

edited

Loading

Benchmark for `084ba79`

Benchmark for `9cc084f`

Benchmark for `23432e3`

Benchmark for `c6fa39c`

Benchmark for `3275d89`

Benchmark for `795388a`

Benchmark for `e40a6a8`

Benchmark for `4fe7658`

Benchmark for `7349e91`

fkrause98 May 29, 2025 •

edited

Loading