-
Notifications
You must be signed in to change notification settings - Fork 24
feat: Added DB configuration option to turn on/off parallel insert and hashing #1340
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
87 commits
Select commit
Hold shift + click to select a range
e38dad3
Moving merkle.rs to merkle directory
bernard-avalabs 03d5ad1
chore: Moved merkle.rs to mod.rs in merkle directory
bernard-avalabs a0aeb19
Merge branch 'bernard/insert-worker-pool-rayon' of github.com:ava-lab…
bernard-avalabs afae01e
Simple version where all operations are offloaded to a single
bernard-avalabs c9e7651
Basic handoff to offload thread appears to be working.
bernard-avalabs bd877e6
More testing
bernard-avalabs 7a22038
Working on prepare phase. For the case where the trie is empty,
bernard-avalabs c3d4100
Working on case for non-empty trie and an non-empty partial path in t…
bernard-avalabs c2445bc
Implemented all of the prepare phase cases. Appears to be working.
bernard-avalabs 894033c
Added comments detailing steps to fix a temporarily malformed
bernard-avalabs 8fb6da5
Initial code for undoing transform
bernard-avalabs bff76c1
Minor cleanup
bernard-avalabs 7ef3fe2
Additional comments.
bernard-avalabs 31129e6
Added some nodestore related changes. Fix possible error with the use of
bernard-avalabs efce980
Extra debug
bernard-avalabs 6628617
Appears to work with an added insert path (needs to be refactored).
bernard-avalabs 954a02d
Added Delete and DeleteRange
bernard-avalabs bd1e7c2
Cleanup
bernard-avalabs fcda015
Added code to handle most of the cases where the key is empty.
bernard-avalabs 364f8d7
Improved comments around handling empty key.
bernard-avalabs b12ac8b
Added deleted from child nodestores to the proposal.
bernard-avalabs e2fde1a
Moved threadpool to DB.
bernard-avalabs 2cbf542
Support case where trie only has a single leaf with an empty key.
bernard-avalabs 9c4ac97
Added some error handling
bernard-avalabs 660f0f8
Reorganizing code
bernard-avalabs eb2d0dd
Code cleanup
bernard-avalabs 231c264
More error handling.
bernard-avalabs 5fff34d
More cleanly handled the remove empty prefix case
bernard-avalabs ee9a488
Small fix to also delete the root value on a remove all.
bernard-avalabs e5bf520
Clean up. Fixed test.
bernard-avalabs e5a5757
Removed OneLock since it is not necessary
bernard-avalabs b670403
Cleanup with additional comments.
bernard-avalabs e816dad
Added test cases
bernard-avalabs e6d953e
More cleanup
bernard-avalabs b3b8c3d
Merge remote-tracking branch 'origin/main' into bernard/insert-worker…
bernard-avalabs 2bfaf34
Cleanup for PR
bernard-avalabs 2c2f3b5
Replaced Option<ThreadPool> with OnceLock<ThreadPool> so propose_para…
bernard-avalabs 57717d9
Addressing most of the PR comments.
bernard-avalabs 320c178
Moved threadpool to RevisionManager.
bernard-avalabs cfe983e
Made the initial change to the insert/remove/remove_prefix interface …
bernard-avalabs 301243c
Moved take child code from parallel.rs to branch.rs
bernard-avalabs 5effd6e
Added a TODO to describe a possible optimization using scoped threads.
bernard-avalabs 7dd784c
Merge remote-tracking branch 'origin/main' into bernard/insert-worker…
bernard-avalabs 7e12f7b
Initial commit for parallel hashing.
bernard-avalabs bfae618
Fixed hash computation
bernard-avalabs d57f8aa
Added error handling.
bernard-avalabs c8422b0
Cleanup
bernard-avalabs 7c8ff58
More cleanup. Sends vector of deleted nodes instead of child's nodest…
bernard-avalabs 6b07fb3
Cleanup.
bernard-avalabs 42e6c1c
Modify fuzz_checker to randomly use either propose or propose_parallel
bernard-avalabs acbaca7
Comment fix
bernard-avalabs 89aa167
Comment fix
bernard-avalabs 4e21f71
Removed printlns
bernard-avalabs 885f384
Comment fix
bernard-avalabs 9c0c139
Comment fix
bernard-avalabs 3a090b6
Changes for Eth hash
bernard-avalabs fbaf3eb
Removed hash_subtrie_with_index and modified hash_helper to take a path.
bernard-avalabs 82d2c59
Updated typed-builder
bernard-avalabs 1850186
Merge branch 'main' into bernard/insert-worker-pool-rayon-part2-merged
bernard-avalabs 6d6c61c
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs 24be026
Updated rayon
bernard-avalabs 0c60b09
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs f2c464b
Updating CI
bernard-avalabs 8f7cb00
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs f11bd53
Merge branch 'main' into bernard/insert-worker-pool-rayon-part2-merged
bernard-avalabs 21dec41
Removed extra "with" entries in ci.yaml
bernard-avalabs 82b76aa
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs 4cc16b1
Merge branch 'main' into bernard/insert-worker-pool-rayon-part2-merged
bernard-avalabs 35e12a7
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs e85d0b0
Fix after merge
bernard-avalabs 4291ed6
Formatting
bernard-avalabs 426868a
Ethhash fix after merge
bernard-avalabs 6645609
Merge branch 'main' into bernard/insert-worker-pool-rayon-part2-merged
bernard-avalabs 5d17d5a
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs 30093de
Separate response channel into sender/receiver
bernard-avalabs 9b70357
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs a6970e7
Addressed PR feedback
bernard-avalabs 2176b9a
Fix for ethhash
bernard-avalabs 0356df3
Comment update
bernard-avalabs 5516f9d
Additional change for feedback
bernard-avalabs 1e98c9a
Merge branch 'main' into bernard/insert-worker-pool-rayon-part2-merged
bernard-avalabs d01ca7b
Updating cargo.lock
bernard-avalabs 8a80806
Merge branch 'bernard/insert-worker-pool-rayon-part2-merged' into ber…
bernard-avalabs fccb73e
Added config option to turn on or off parallel insert and hashing.
bernard-avalabs 9a59216
Comment fix
bernard-avalabs 99049dd
FFI-related changes
bernard-avalabs b7474c6
Added testcase for FFI changes
bernard-avalabs File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there is no functionality change and is a performance improvement, then why is this available to be set? Just to ensure that it works as expected prior to actually shipping it? It seems like this shouldn't even be available
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are some cases (e.g., very small proposals) in which parallel insertion/hashing is slower than serial insertion/hashing. We may want to allow the application using firewood to choose which version they want to use.
We are still waiting on some performance numbers to determine if providing this option is necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should hold off on merging this until we know what the tradeoffs are. It's an easy change we can make, but nobody is going to want to manually configure this variable I think.