feat: Block exchange optimizations #1325
Merged
+1,360
−526
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cnanakos
force-pushed
the
feat/simple-optimizations-rebase
branch
from
46cc4ad to
372151e
Compare
cnanakos
force-pushed
the
feat/simple-optimizations-rebase
branch
13 times, most recently
from
6b9ebc7 to
7c4e862
Compare
cnanakos
force-pushed
the
feat/simple-optimizations-rebase
branch
16 times, most recently
from
85eb620 to
5cf5f00
Compare
Use probabilistic distribution based on peer quality scores, giving all peers opportunity while favoring better-performing ones. Selection probability is inversely proportional to score. Part of #974
- Reduce DefaultFetchBatch to prevent blockData explosion - Add chunked onBatch processing to prevent memory accumulation - Implement memory buffer reuse to reduce GC allocation churn - Fix short-circuit evaluation Part of #974
This prevents unbounded peer accumulation while maintaining redundancy. When peer count exceeds the maximum, the least active peers are removed from tracking to free resources. Part of #974
Achieves significant memory reduction with equivalent network performance. The reduced batch size prevents memory pressure while preserving transfer efficiency, improving overall system resource utilization. Part of #974
- Add codex_block_exchange_discovery_requests_total counter to track peer discovery frequency - Add codex_block_exchange_peer_timeouts_total counter to monitor peer reliability issues - Add codex_block_exchange_requests_failed_total counter to track request failure rates Part of #974
This prevents peers from becoming permanently invisible to block discovery when they fail to respond to WantHave requests. Part of #974
Returns false on duplicate marking attempts instead of logging errors, eliminating duplicate marking loop in blockPresenceHandler and preventing duplicate block requests across concurrent flows. Part of #974
Part of #974 Signed-off-by: Chrysostomos Nanakos <[email protected]>
Part of #974 Signed-off-by: Chrysostomos Nanakos <[email protected]>
Remove makeRandomDataset in favor of direct composition with makeRandomBlocks and makeDataset. Part of #974 Signed-off-by: Chrysostomos Nanakos <[email protected]>
Part of #974 Signed-off-by: Chrysostomos Nanakos <[email protected]>
Implements delta-based WantList updates to reduce network traffic during block exchange. Only sends newly added blocks instead of resending the entire WantList on every refresh. Also some network related fixes: - Add TCP_NODELAY flag to prevent Nagle's algorithm delays - Clear sendConn on stream reset to allow garbage collection - Improve error handling in NetworkPeer.send() Part of #974 Signed-off-by: Chrysostomos Nanakos <[email protected]>
Now fetchBatched maintains a sliding window of batchSize blocks in-flight. When 75% complete, adds next chunk to maintain constant window size. This ensures blocks are already pending or have been fetched when StoreStream needs them. Part of #974 Signed-off-by: Chrysostomos Nanakos <[email protected]>
Part of #974 Signed-off-by: Chrysostomos Nanakos <[email protected]>
cnanakos
force-pushed
the
feat/simple-optimizations-rebase
branch
from
82ed5ef to
ff7ac82
Compare
…mory leak Stale entries in peer.lastSentWants were not removed when blocks were resolved or cancelled, causing unbounded memory growth. This adds incremental cleanup during refreshBlockKnowledge, removing up to 2048 stale entries per refresh cycle with proper event loop yielding. Part of #974 Signed-off-by: Chrysostomos Nanakos <[email protected]>
…proc Part of #974 Signed-off-by: Chrysostomos Nanakos <[email protected]>
cnanakos
force-pushed
the
feat/simple-optimizations-rebase
branch
3 times, most recently
from
d67a8c8 to
8257406
Compare
…with no peers (#1343) There is a limitation in `downloadInternal` that infinitely loops for peers when attempting to download a block if the node does not have peers. Currently, the block retry counter is only decremented if there exists peers and a download attempt is made, otherwise the loop will continue until peers are found, ad infinium. This fix decrements the retry counter on each loop that no peers are found. Additionally, it waits for the `DiscoveryRateLimit` interval before retrying, as otherwise looking for peers would not produce new results.
…ning Part of #974 Signed-off-by: Chrysostomos Nanakos <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.