Committee aggregation

docs/client/networking.md

@@ -33,6 +33,7 @@ Each node entry contains an ENR. This is an Ethereum Node Record. It includes:
 - The node's public key
 - Network address
 - Port numbers
+- Committee assignments (for aggregators)
 - Other metadata
 
 In production, dynamic discovery would replace static configuration.
@@ -62,15 +63,35 @@ Messages are organized by topic. Topic names follow a pattern that includes:
 
 This structure lets clients subscribe to relevant messages and ignore others.
 
+The payload carried in the gossipsub message is the SSZ-encoded,
+Snappy-compressed message, which type is identified by the topic:
+
+| Topic Name | Message Type | Encoding |
+|------------------------------------------------------------|-----------------------------|--------------|
+| /leanconsensus/devnet3/blocks/ssz_snappy | SignedBlockWithAttestation | SSZ + Snappy |
+| /leanconsensus/devnet3/attestations/ssz_snappy | SignedAttestation | SSZ + Snappy |
+| /leanconsensus/devnet3/attestation\_{subnet_id}/ssz_snappy | SignedAttestation | SSZ + Snappy |
+| /leanconsensus/devnet3/aggregation/ssz_snappy | SignedAggregatedAttestation | SSZ + Snappy |
+
 ### Message Types
 
-Two main message types exist:
+Three main message types exist:
+
+- _Blocks_, defined by the `SignedBlockWithAttestation` type, are proposed by
+  validators and propagated on the block topic. Every node needs to see blocks
+  quickly.
 
-Blocks are proposed by validators. They propagate on the block topic. Every
-node needs to see blocks quickly.
+- _Attestations_, defined by the `SignedAttestation` type, come from all
+  validators. They propagate on the global attestation topic. Additionally,
+  each committee has its own attestation topic. Validators publish to their
+  committee's attestation topic and global attestation topic. Non-aggregating
+  validators subscribe only to the global attestation topic, while aggregators
+  subscribe to both the global and their committee's attestation topic.
 
-Attestations come from all validators. They propagate on the attestation topic. High volume
-but small messages.
+- _Committee aggregations_, defined by the `SignedAggregatedAttestation` type,
+  created by committee aggregators. These combine attestations from committee
+  members. Aggregations propagate on the aggregation topic to which every
+  validator subscribes.
 
 ### Encoding
 

docs/client/validator.md

@@ -2,8 +2,9 @@
 
 ## Overview
 
-Validators participate in consensus by proposing blocks and producing attestations. This
-document describes what honest validators do.
+Validators participate in consensus by proposing blocks and producing attestations.
+Optionally validators can opt-in to behave as aggregators in their committee.
+This document describes what honest validators do.
 
 ## Validator Assignment
 
@@ -16,6 +17,32 @@ diversity helps test interoperability.
 In production, validator assignment will work differently. The current approach
 is temporary for devnet testing.
 
+## Attestation Committees and Subnets
+
+Attestation committee is a group of validators contributing to the common
+aggregated attestations. Subnets are network channels dedicated to specific committees.
+
+In the devnet-3 design, however, there is one global subnet for signed
+attestations propagation, in addition to publishing into per committee subnets.
+This is due to 3SF-mini consensus design, that requires 2/3+ of all
+attestations to be observed by any validator to compute safe target correctly.
+
+Note that non-aggregating validators do not need to subscribe to committee
+attestation subnets. They only need to subscribe to the global attestation
+subnet.
+
+Every validator is assigned to a single committee. Number of committees is
+defined in config.yaml. Each committee maps to a subnet ID. Validator's
+subnet ID is derived using their validator index modulo number of committees.
+This is to simplify debugging and testing. In the future, validator's subnet ID
+will be assigned randomly per epoch.
+
+## Aggregator assignment
+
+Some validators are self-assigned as aggregators. Aggregators collect and combine
+attestations from other validators in their committee. To become an aggregator,
+a validator sets `is_aggregator` flag to true as ENR record field.
+
 ## Proposing Blocks
 
 Each slot has exactly one designated proposer. The proposer is determined by
@@ -52,7 +79,7 @@ receive and validate it.
 
 ## Attesting
 
-Every validator attestations in every slot. Attesting happens in the second interval,
+Every validator attests in every slot. Attesting happens in the second interval,
 after proposals are made.
 
 ### What to Attest For
@@ -78,8 +105,8 @@ compute the head.
 
 ### Broadcasting Attestations
 
-Validators sign their attestations and broadcast them. The network uses a single topic
-for all attestations. No subnets or committees in the current design.
+Validators sign their attestations and broadcast them into the global
+attestation topic and its corresponding subnet topic.
 
 ## Timing
 
@@ -98,11 +125,7 @@ blocks and attestations.
 Attestation aggregation combines multiple attestations into one. This saves bandwidth and
 block space.
 
-Devnet 0 has no aggregation. Each attestation is separate. Future devnets will add
-aggregation.
-
-When aggregation is added, aggregators will collect attestations and combine them.
-Aggregated attestations will be broadcast separately.
+Devnet-3 introduces signatures aggregation. Aggregators will collect attestations and combine them. Aggregated attestations will be broadcast separately.
 
 ## Signature Handling
 

packages/testing/src/consensus_testing/test_fixtures/fork_choice.py

@@ -15,6 +15,7 @@
 from lean_spec.subspecs.containers.attestation import (
     Attestation,
     AttestationData,
+    SignedAttestation,
 )
 from lean_spec.subspecs.containers.block import (
     Block,
@@ -50,6 +51,8 @@
 )
 from .base import BaseConsensusFixture
 
+DEFAULT_VALIDATOR_ID = ValidatorIndex(0)
+
 
 class ForkChoiceTest(BaseConsensusFixture):
     """
@@ -210,8 +213,9 @@ def make_fixture(self) -> Self:
         # The Store is the node's local view of the chain.
         # It starts from a trusted anchor (usually genesis).
         store = Store.get_forkchoice_store(
-            state=self.anchor_state,
+            anchor_state=self.anchor_state,
             anchor_block=self.anchor_block,
+            validator_id=DEFAULT_VALIDATOR_ID,
         )
 
         # Block registry for fork creation
@@ -261,7 +265,11 @@ def make_fixture(self) -> Self:
 
                     # Process the block through Store.
                     # This validates, applies state transition, and updates head.
-                    store = store.on_block(signed_block, LEAN_ENV_TO_SCHEMES[self.lean_env])
+                    store = store.on_block(
+                        signed_block,
+                        current_validator=DEFAULT_VALIDATOR_ID,
+                        scheme=LEAN_ENV_TO_SCHEMES[self.lean_env],
+                    )
 
                 elif isinstance(step, AttestationStep):
                     # Process a gossip attestation.
@@ -356,33 +364,85 @@ def _build_block_from_spec(
         #
         # Attestations vote for blocks and influence fork choice weight.
         # The spec may include attestations to include in this block.
-        attestations, attestation_signatures = self._build_attestations_from_spec(
-            spec, store, block_registry, parent_root, key_manager
+        attestations, attestation_signatures, valid_signature_keys = (
+            self._build_attestations_from_spec(
+                spec, store, block_registry, parent_root, key_manager
+            )
         )
 
-        # Merge new attestation signatures with existing gossip signatures.
-        # These are needed for signature aggregation later.
-        gossip_signatures = dict(store.gossip_signatures)
-        gossip_signatures.update(attestation_signatures)
+        # Merge per-attestation signatures into the Store's gossip signature cache.
+        # Required so the Store can aggregate committee signatures later when building payloads.
+        working_store = store
+        for attestation in attestations:
+            sig_key = SignatureKey(attestation.validator_id, attestation.data.data_root_bytes())
+            if sig_key not in valid_signature_keys:
+                continue
+            signature = attestation_signatures.get(sig_key)
+            if signature is None:
+                continue
+            signed_attestation = SignedAttestation(
+                validator_id=attestation.validator_id,
+                message=attestation.data,
+                signature=signature,
+            )
+            working_store = working_store.on_gossip_attestation(
+                signed_attestation,
+                scheme=LEAN_ENV_TO_SCHEMES[self.lean_env],
+                is_aggregator=True,
+            )
 
-        # Collect attestations from the store if requested.
+        # Prepare attestations and aggregated payloads for block construction.
         #
-        # Previous proposers' attestations become available for inclusion.
-        # This makes test vectors more realistic.
-        available_attestations: list[Attestation] | None = None
+        # Two sources of attestations:
+        # 1. Explicit attestations from the spec (always included)
+        # 2. Store attestations (only if include_store_attestations is True)
+        #
+        # For all attestations, we need to create aggregated proofs
+        # so build_block can include them in the block body.
+        # Attestations with the same data should be merged into a single proof.
+        available_attestations: list[Attestation]
         known_block_roots: set[Bytes32] | None = None
 
+        aggregated_payloads = dict(store.aggregated_payloads) if store.aggregated_payloads else {}
+
+        # Collect all attestations that need aggregated proofs
+        all_attestations_for_proofs: list[Attestation] = list(attestations)
+
         if spec.include_store_attestations:
             # Gather all attestations: both active and recently received.
-            available_attestations = [
+            store_attestations = [
                 Attestation(validator_id=vid, data=data)
                 for vid, data in store.latest_known_attestations.items()
             ]
-            available_attestations.extend(
+            store_attestations.extend(
                 Attestation(validator_id=vid, data=data)
                 for vid, data in store.latest_new_attestations.items()
             )
+
+            # Add store attestations to the list for proof creation
+            all_attestations_for_proofs.extend(store_attestations)
+
+            # Combine for block construction
+            available_attestations = store_attestations + attestations
             known_block_roots = set(store.blocks.keys())
+        else:
+            # Use only explicit attestations from the spec
+            available_attestations = attestations
+
+        # Build aggregated proofs via Store aggregation logic.
+        attestation_map = {
+            attestation.validator_id: attestation.data
+            for attestation in all_attestations_for_proofs
+        }
+        aggregation_store = working_store.model_copy(
+            update={
+                "head": parent_root,
+                "latest_new_attestations": attestation_map,
+                "aggregated_payloads": aggregated_payloads,
+            }
+        )
+        aggregation_store = aggregation_store.aggregate_committee_signatures()
+        aggregated_payloads = aggregation_store.aggregated_payloads
 
         # Build the block using spec logic
         #
@@ -393,11 +453,10 @@ def _build_block_from_spec(
             slot=spec.slot,
             proposer_index=proposer_index,
             parent_root=parent_root,
-            attestations=attestations,
+            attestations=available_attestations,
             available_attestations=available_attestations,
             known_block_roots=known_block_roots,
-            gossip_signatures=gossip_signatures,
-            aggregated_payloads=store.aggregated_payloads,
+            aggregated_payloads=aggregated_payloads,
         )
 
         # Create proposer attestation
@@ -505,7 +564,7 @@ def _build_attestations_from_spec(
         block_registry: dict[str, Block],
         parent_root: Bytes32,
         key_manager: XmssKeyManager,
-    ) -> tuple[list[Attestation], dict[SignatureKey, Signature]]:
+    ) -> tuple[list[Attestation], dict[SignatureKey, Signature], set[SignatureKey]]:
         """
         Build attestations and signatures from block specification.
 
@@ -521,15 +580,16 @@ def _build_attestations_from_spec(
             key_manager: Key manager for signing.
 
         Returns:
-            Tuple of (attestations list, signature lookup dict).
+            Tuple of (attestations list, signature lookup dict, valid signature keys).
         """
         # No attestations specified means empty block body.
         if spec.attestations is None:
-            return [], {}
+            return [], {}, set()
 
         parent_state = store.states[parent_root]
         attestations = []
         signature_lookup: dict[SignatureKey, Signature] = {}
+        valid_signature_keys: set[SignatureKey] = set()
 
         for aggregated_spec in spec.attestations:
             # Build attestation data once.
@@ -567,8 +627,10 @@ def _build_attestations_from_spec(
                 # This enables lookup during signature aggregation.
                 sig_key = SignatureKey(validator_id, attestation_data.data_root_bytes())
                 signature_lookup[sig_key] = signature
+                if aggregated_spec.valid_signature:
+                    valid_signature_keys.add(sig_key)
 
-        return attestations, signature_lookup
+        return attestations, signature_lookup, valid_signature_keys
 
     def _build_attestation_data_from_spec(
         self,

packages/testing/src/consensus_testing/test_fixtures/state_transition.py

@@ -10,10 +10,8 @@
 from lean_spec.subspecs.containers.state.state import State
 from lean_spec.subspecs.containers.validator import ValidatorIndex
 from lean_spec.subspecs.ssz.hash import hash_tree_root
-from lean_spec.subspecs.xmss.aggregation import SignatureKey
 from lean_spec.types import Bytes32
 
-from ..keys import get_shared_key_manager
 from ..test_types import BlockSpec, StateExpectation
 from .base import BaseConsensusFixture
 
@@ -263,26 +261,11 @@ def _build_block_from_spec(self, spec: BlockSpec, state: State) -> tuple[Block,
             for vid in agg.aggregation_bits.to_validator_indices()
         ]
 
-        if plain_attestations:
-            key_manager = get_shared_key_manager(max_slot=spec.slot)
-            gossip_signatures = {
-                SignatureKey(
-                    att.validator_id, att.data.data_root_bytes()
-                ): key_manager.sign_attestation_data(
-                    att.validator_id,
-                    att.data,
-                )
-                for att in plain_attestations
-            }
-        else:
-            gossip_signatures = {}
-
         block, post_state, _, _ = state.build_block(
             slot=spec.slot,
             proposer_index=proposer_index,
             parent_root=parent_root,
             attestations=plain_attestations,
-            gossip_signatures=gossip_signatures,
             aggregated_payloads={},
         )
         return block, post_state

packages/testing/src/consensus_testing/test_fixtures/verify_signatures.py

@@ -26,7 +26,7 @@
 from lean_spec.subspecs.containers.validator import ValidatorIndex
 from lean_spec.subspecs.koalabear import Fp
 from lean_spec.subspecs.ssz import hash_tree_root
-from lean_spec.subspecs.xmss.aggregation import AggregatedSignatureProof, SignatureKey
+from lean_spec.subspecs.xmss.aggregation import AggregatedSignatureProof
 from lean_spec.subspecs.xmss.constants import TARGET_CONFIG
 from lean_spec.subspecs.xmss.containers import Signature
 from lean_spec.subspecs.xmss.types import (
@@ -233,19 +233,12 @@ def _build_block_from_spec(
             spec, state, key_manager
         )
 
-        # Provide signatures to State.build_block for valid attestations
-        gossip_signatures = {
-            SignatureKey(att.validator_id, att.data.data_root_bytes()): sig
-            for att, sig in zip(valid_attestations, valid_signatures, strict=True)
-        }
-
         # Use State.build_block for valid attestations (pure spec logic)
         final_block, _, _, aggregated_signatures = state.build_block(
             slot=spec.slot,
             proposer_index=proposer_index,
             parent_root=parent_root,
             attestations=valid_attestations,
-            gossip_signatures=gossip_signatures,
             aggregated_payloads={},
         )