Skip to content

Fullnodes to publish data columns from EL getBlobs #7258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 8, 2025
Merged

Fullnodes to publish data columns from EL getBlobs #7258

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d4e9d6a
Have Fullnodes publish data columns from EL `getBlobs`.
jimmygchen Apr 4, 2025
0c60a36
Address review comment.
jimmygchen Apr 7, 2025
ff98957
Merge branch 'unstable' into distributed-publishing-full-nodes
jimmygchen Apr 7, 2025
12ea5dd
Use `retain` instead of collecting into a new `Vec` when filtering.
jimmygchen Apr 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions beacon_node/network/src/network_beacon_processor/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -927,20 +927,20 @@ impl<T: BeaconChainTypes> NetworkBeaconProcessor<T> {
block_root: Hash256,
publish_blobs: bool,
) {
let is_supernode = self.network_globals.is_supernode();

let custody_columns = self.network_globals.sampling_columns.clone();
let self_cloned = self.clone();
let publish_fn = move |blobs_or_data_column| {
// At the moment non supernodes are not required to publish any columns.
// TODO(das): we could experiment with having full nodes publish their custodied
// columns here.
if publish_blobs && is_supernode {
if publish_blobs {
match blobs_or_data_column {
BlobsOrDataColumns::Blobs(blobs) => {
self_cloned.publish_blobs_gradually(blobs, block_root);
}
BlobsOrDataColumns::DataColumns(columns) => {
self_cloned.publish_data_columns_gradually(columns, block_root);
let columns_to_publish = columns
.into_iter()
.filter(|c| custody_columns.contains(&c.index))
.collect();
self_cloned.publish_data_columns_gradually(columns_to_publish, block_root);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small optimization here might be to skip the filtering if self.network_globals.is_supernode()?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 0c60a36

I was a bit tempted to remove the is_supernode function - with the introduction of validator custody, some full nodes now performs the same tasks as supernodes - including distributed blob publishing and reconstruction, but I'll leave this change out of this PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With validator custody, this becomes a bit complicated - the node should be publishing the columns that it imported into its database - so we need to be using the custody_group_count tied to the Rpc or gossip verified beacon block instead of using the NetworkGlobals, because CGC is now dynamic:

However, there are a few moving pieces and this wiring hasn't been fully set up, I think it's best to wait until the following PR is completed:

}
};
}
Expand Down Expand Up @@ -1139,7 +1139,7 @@ impl<T: BeaconChainTypes> NetworkBeaconProcessor<T> {
///
/// This is an optimisation to reduce outbound bandwidth and ensures each column is published
/// by some nodes on the network as soon as possible. Our hope is that some columns arrive from
/// other supernodes in the meantime, obviating the need for us to publish them. If no other
/// other nodes in the meantime, obviating the need for us to publish them. If no other
/// publisher exists for a column, it will eventually get published here.
fn publish_data_columns_gradually(
self: &Arc<Self>,
Expand All @@ -1164,9 +1164,9 @@ impl<T: BeaconChainTypes> NetworkBeaconProcessor<T> {
});
};

// If this node is a super node, permute the columns and split them into batches.
// Permute the columns and split them into batches.
// The hope is that we won't need to publish some columns because we will receive them
// on gossip from other supernodes.
// on gossip from other nodes.
data_columns_to_publish.shuffle(&mut rand::thread_rng());

let blob_publication_batch_interval = chain.config.blob_publication_batch_interval;
Expand Down
Loading