Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
e38dad3
Moving merkle.rs to merkle directory
bernard-avalabs Aug 14, 2025
03d5ad1
chore: Moved merkle.rs to mod.rs in merkle directory
bernard-avalabs Aug 14, 2025
a0aeb19
Merge branch 'bernard/insert-worker-pool-rayon' of github.com:ava-lab…
bernard-avalabs Aug 14, 2025
afae01e
Simple version where all operations are offloaded to a single
bernard-avalabs Aug 19, 2025
c9e7651
Basic handoff to offload thread appears to be working.
bernard-avalabs Aug 19, 2025
bd877e6
More testing
bernard-avalabs Aug 19, 2025
7a22038
Working on prepare phase. For the case where the trie is empty,
bernard-avalabs Aug 20, 2025
c3d4100
Working on case for non-empty trie and an non-empty partial path in t…
bernard-avalabs Aug 20, 2025
c2445bc
Implemented all of the prepare phase cases. Appears to be working.
bernard-avalabs Aug 20, 2025
894033c
Added comments detailing steps to fix a temporarily malformed
bernard-avalabs Aug 20, 2025
8fb6da5
Initial code for undoing transform
bernard-avalabs Aug 28, 2025
bff76c1
Minor cleanup
bernard-avalabs Aug 29, 2025
7ef3fe2
Additional comments.
bernard-avalabs Aug 29, 2025
31129e6
Added some nodestore related changes. Fix possible error with the use of
bernard-avalabs Aug 29, 2025
efce980
Extra debug
bernard-avalabs Aug 29, 2025
6628617
Appears to work with an added insert path (needs to be refactored).
bernard-avalabs Sep 1, 2025
954a02d
Added Delete and DeleteRange
bernard-avalabs Sep 1, 2025
bd1e7c2
Cleanup
bernard-avalabs Sep 1, 2025
fcda015
Added code to handle most of the cases where the key is empty.
bernard-avalabs Sep 2, 2025
364f8d7
Improved comments around handling empty key.
bernard-avalabs Sep 2, 2025
b12ac8b
Added deleted from child nodestores to the proposal.
bernard-avalabs Sep 2, 2025
e2fde1a
Moved threadpool to DB.
bernard-avalabs Sep 2, 2025
2cbf542
Support case where trie only has a single leaf with an empty key.
bernard-avalabs Sep 2, 2025
9c4ac97
Added some error handling
bernard-avalabs Sep 2, 2025
660f0f8
Reorganizing code
bernard-avalabs Sep 2, 2025
eb2d0dd
Code cleanup
bernard-avalabs Sep 2, 2025
231c264
More error handling.
bernard-avalabs Sep 3, 2025
5fff34d
More cleanly handled the remove empty prefix case
bernard-avalabs Sep 3, 2025
ee9a488
Small fix to also delete the root value on a remove all.
bernard-avalabs Sep 3, 2025
e5bf520
Clean up. Fixed test.
bernard-avalabs Sep 3, 2025
e5a5757
Removed OneLock since it is not necessary
bernard-avalabs Sep 3, 2025
b670403
Cleanup with additional comments.
bernard-avalabs Sep 3, 2025
e816dad
Added test cases
bernard-avalabs Sep 3, 2025
e6d953e
More cleanup
bernard-avalabs Sep 3, 2025
b3b8c3d
Merge remote-tracking branch 'origin/main' into bernard/insert-worker…
bernard-avalabs Sep 4, 2025
2bfaf34
Cleanup for PR
bernard-avalabs Sep 4, 2025
2c2f3b5
Replaced Option<ThreadPool> with OnceLock<ThreadPool> so propose_para…
bernard-avalabs Sep 4, 2025
57717d9
Addressing most of the PR comments.
bernard-avalabs Sep 10, 2025
320c178
Moved threadpool to RevisionManager.
bernard-avalabs Sep 10, 2025
cfe983e
Made the initial change to the insert/remove/remove_prefix interface …
bernard-avalabs Sep 11, 2025
301243c
Moved take child code from parallel.rs to branch.rs
bernard-avalabs Sep 11, 2025
5effd6e
Added a TODO to describe a possible optimization using scoped threads.
bernard-avalabs Sep 11, 2025
7dd784c
Merge remote-tracking branch 'origin/main' into bernard/insert-worker…
bernard-avalabs Sep 11, 2025
7e12f7b
Initial commit for parallel hashing.
bernard-avalabs Sep 24, 2025
bfae618
Fixed hash computation
bernard-avalabs Sep 24, 2025
d57f8aa
Added error handling.
bernard-avalabs Sep 25, 2025
c8422b0
Cleanup
bernard-avalabs Sep 25, 2025
7c8ff58
More cleanup. Sends vector of deleted nodes instead of child's nodest…
bernard-avalabs Sep 25, 2025
6b07fb3
Cleanup.
bernard-avalabs Sep 25, 2025
42e6c1c
Modify fuzz_checker to randomly use either propose or propose_parallel
bernard-avalabs Sep 25, 2025
acbaca7
Comment fix
bernard-avalabs Sep 25, 2025
89aa167
Comment fix
bernard-avalabs Sep 25, 2025
4e21f71
Removed printlns
bernard-avalabs Sep 25, 2025
885f384
Comment fix
bernard-avalabs Sep 25, 2025
9c0c139
Comment fix
bernard-avalabs Sep 25, 2025
3a090b6
Changes for Eth hash
bernard-avalabs Sep 25, 2025
fbaf3eb
Removed hash_subtrie_with_index and modified hash_helper to take a path.
bernard-avalabs Sep 25, 2025
82d2c59
Updated typed-builder
bernard-avalabs Sep 25, 2025
1850186
Merge branch 'main' into bernard/insert-worker-pool-rayon-part2-merged
bernard-avalabs Sep 25, 2025
6d6c61c
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs Sep 25, 2025
24be026
Updated rayon
bernard-avalabs Sep 25, 2025
0c60b09
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs Sep 25, 2025
f2c464b
Updating CI
bernard-avalabs Sep 25, 2025
8f7cb00
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs Sep 25, 2025
f11bd53
Merge branch 'main' into bernard/insert-worker-pool-rayon-part2-merged
bernard-avalabs Sep 26, 2025
21dec41
Removed extra "with" entries in ci.yaml
bernard-avalabs Sep 26, 2025
82b76aa
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs Sep 26, 2025
4cc16b1
Merge branch 'main' into bernard/insert-worker-pool-rayon-part2-merged
bernard-avalabs Sep 26, 2025
35e12a7
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs Sep 26, 2025
e85d0b0
Fix after merge
bernard-avalabs Sep 26, 2025
4291ed6
Formatting
bernard-avalabs Sep 26, 2025
426868a
Ethhash fix after merge
bernard-avalabs Sep 26, 2025
6645609
Merge branch 'main' into bernard/insert-worker-pool-rayon-part2-merged
bernard-avalabs Sep 26, 2025
5d17d5a
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs Sep 26, 2025
30093de
Separate response channel into sender/receiver
bernard-avalabs Sep 29, 2025
9b70357
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs Sep 29, 2025
a6970e7
Addressed PR feedback
bernard-avalabs Oct 6, 2025
2176b9a
Fix for ethhash
bernard-avalabs Oct 6, 2025
0356df3
Comment update
bernard-avalabs Oct 6, 2025
5516f9d
Additional change for feedback
bernard-avalabs Oct 6, 2025
1e98c9a
Merge branch 'main' into bernard/insert-worker-pool-rayon-part2-merged
bernard-avalabs Oct 6, 2025
d01ca7b
Updating cargo.lock
bernard-avalabs Oct 6, 2025
8a80806
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs Oct 6, 2025
fccb73e
Added config option to turn on or off parallel insert and hashing.
bernard-avalabs Oct 6, 2025
9a59216
Comment fix
bernard-avalabs Oct 6, 2025
99049dd
FFI-related changes
bernard-avalabs Oct 6, 2025
b7474c6
Added testcase for FFI changes
bernard-avalabs Oct 6, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 50 additions & 50 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions ffi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ A `Config` should be provided when creating the database. A default config is pr
FreeListCacheEntries: 40000,
Revisions: 100,
ReadCacheStrategy: OnlyCacheWrites,
Parallel: false,
}
```

Expand All @@ -83,6 +84,12 @@ If set to `true`, an empty database will be created, overriding any existing fil

*Default*: `false`

#### `Parallel` - `bool`

If set to `true`, proposals are created in parallel. This can improve performance by supporting both parallel reads and parallel hashing during proposal creation.

*Default*: `false`

#### `Revisions` - `uint`

Indicates the number of committed roots accessible before the diff layer is compressed. Must be explicitly set if the config is specified to at least 2.
Expand Down
3 changes: 3 additions & 0 deletions ffi/firewood.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ type Config struct {
FreeListCacheEntries uint
Revisions uint
ReadCacheStrategy CacheStrategy
Parallel bool
}

// DefaultConfig returns a sensible default Config.
Expand All @@ -68,6 +69,7 @@ func DefaultConfig() *Config {
FreeListCacheEntries: 40_000,
Revisions: 100,
ReadCacheStrategy: OnlyCacheWrites,
Parallel: false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is no functionality change and is a performance improvement, then why is this available to be set? Just to ensure that it works as expected prior to actually shipping it? It seems like this shouldn't even be available

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some cases (e.g., very small proposals) in which parallel insertion/hashing is slower than serial insertion/hashing. We may want to allow the application using firewood to choose which version they want to use.

We are still waiting on some performance numbers to determine if providing this option is necessary.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should hold off on merging this until we know what the tradeoffs are. It's an easy change we can make, but nobody is going to want to manually configure this variable I think.

}
}

Expand Down Expand Up @@ -113,6 +115,7 @@ func New(filePath string, conf *Config) (*Database, error) {
revisions: C.size_t(conf.Revisions),
strategy: C.uint8_t(conf.ReadCacheStrategy),
truncate: C.bool(conf.Truncate),
parallel: C.bool(conf.Parallel),
}

return getDatabaseFromHandleResult(C.fwd_open_db(args))
Expand Down
4 changes: 4 additions & 0 deletions ffi/firewood.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading