Add CIDR-based collapsing for NetworkNeighbors

docs/features/networkneighbors-collapsing.md

@@ -0,0 +1,87 @@
+# NetworkNeighbors CIDR-based collapsing
+
+## Summary
+
+When a node-agent observes external traffic from a workload, it records each distinct destination IP as a separate `NetworkNeighbor` entry. Since the `Identifier` includes the IP address, high-traffic profiles targeting a range of IPs (e.g. a cloud provider's IP block) can explode into thousands of entries — a real-world case hit **8,687 entries** for a single prefix.
+
+The collapsing pass groups `NetworkNeighbor` entries that differ only by IP (same `Type`, `DNS`, namespace selector, pod selector) and replaces them with a small number of **CIDR-bearing entries** when a group exceeds a configurable threshold. CIDR aggregation is bounded by a configurable floor (minimum prefix length / maximum breadth), so output size is predictable even for scattered IP distributions.
+
+Two new fields in the `CollapseConfiguration` CRD control this:
+- `NetworkIPGroupThreshold` (default 50): group size threshold above which IP collapsing is triggered.
+- `NetworkCIDRFloorBits` (default 16): minimum CIDR prefix length — no emitted block is ever broader than `/<floorBits>`.
+
+The pass is a **fixpoint**: running it twice on the same input is guaranteed to produce the same output, so collapsed entries remain stable across successive saves without oscillation or duplication.
+
+## Why it matters
+
+Before collapsing, external-traffic profiles with 8,687 individual IP entries consumed excessive storage, slowed queries, and produced NetworkPolicies with thousands of rules — most of which could be expressed as a handful of CIDR blocks.
+
+Collapsing reduces entry count by orders of magnitude for traffic aimed at cloud IP blocks (e.g. `100.68.24.0/22`, `16.15.183.0/24`) while preserving policy correctness: generated NetworkPolicies still include the same destination ranges, just expressed more efficiently as CIDR blocks instead of `/32` host routes.
+
+## How it works
+
+**Grouping**: Entries are grouped by a deterministic key over `(Type, DNS, NamespaceSelector, PodSelector)`. Entries in the same group differ only by IP address.
+
+**Classification**: Each IP value is classified as:
+- **IPv4 host literal** (e.g., `192.168.1.5`): aggregatable, fed to the CIDR algorithm.
+- **CIDR block** (e.g., `192.168.0.0/24`): treated as an already-covering range, held stable (not re-parsed or re-tightened) to preserve idempotency.
+- **`"*"` sentinel or IPv6**: pass-through, never aggregated.
+
+**Aggregation**: If a group's count of aggregatable IPv4 host addresses exceeds `NetworkIPGroupThreshold`:
+1. Compute the smallest covering prefix (common leading bits across all hosts).
+2. If that prefix's length is at least `NetworkCIDRFloorBits` (e.g., `/16` or longer), emit it as-is.
+3. If it would be broader than `NetworkCIDRFloorBits`, split the group into floor-length buckets (e.g., `/16` buckets) and emit one CIDR entry per non-empty bucket.
+
+**Output entries**: Each emitted CIDR entry carries:
+- The computed CIDR block(s) in `IPAddresses`.
+- The group's **full merged/deduped DNS names and ports** replicated onto every bucket (so each output entry is independently correlated with its ports and DNS for policy generation).
+- The group's shared singular `DNS` value (constant across the group by construction, used for policy metadata).
+- An `Identifier` derived from the group key plus sorted CIDR list — so the identifier-merge pass on a subsequent save recognizes and re-merges the same collapsed entry instead of duplicating it.
+- Empty `IPAddress` (singular field), non-empty `IPAddresses` (plural field).
+
+**Policy generation**: `GenerateNetworkPolicy`'s rule generators (`generateEgressRule`, `generateIngressRule`) now consume the plural `IPAddresses` field:
+- **CIDR block** (e.g., `192.168.0.0/16`): becomes an `IPBlock` peer with empty `OriginalIP` (no single original IP for a range).
+- **Bare IPv4** (e.g., `192.168.1.5`): mirrored through the existing singular-path logic exactly, including known-server enrichment and `/32` formatting.
+- **`"*"` sentinel**: becomes `0.0.0.0/0`.
+- **IPv6**: skipped (out of scope for v1).
+
+## Scope / limitations
+
+**Held-stable CIDRs do not retroactively re-narrow when floor is tightened**: If an operator later changes `NetworkCIDRFloorBits` from 16 to 24 (smaller blocks, higher precision), already-emitted `/16` blocks are held stable for idempotency and won't be split retroactively. They persist until that group naturally re-collapses (e.g., new IPs arrive, triggering re-aggregation). This is an intentional trade-off: predictable idempotency wins over floor freshness for held entries.
+
+**New host IPs inside an already-held CIDR are not immediately absorbed**: If a `/16` block covers `192.168.0.0/16` and new traffic arrives to `192.168.5.100` (which falls inside that CIDR), the new IP persists as a separate entry until its own group independently exceeds the threshold. Entry-count creep is bounded by `NetworkIPGroupThreshold` and the existing merge logic, so this is not unbounded.
+
+**CIDR/`"*"` peers skip known-server enrichment**: A CIDR block is a range, not a single IP, so it cannot be looked up in the known-servers registry. CIDR and `"*"` entries produce bare `IPBlock` peers without the `PolicyRef` name/server enrichment that singular IPs enjoy. Bare-IP elements of the plural `IPAddresses` field retain full known-server matching identical to the singular-field path.
+
+**IPv4 only for v1**: IPv6 addresses pass through uncollapsed; future work may add IPv6 support.
+
+## Configuration
+
+Both fields are part of the existing `CollapseConfiguration` CRD singleton (`default`), using the same zero-means-default semantics as the existing path-collapsing thresholds:
+
+```yaml
+apiVersion: spdx.softwarecomposition.kubescape.io/v1beta1
+kind: CollapseConfiguration
+metadata:
+  name: default
+spec:
+  # ... existing fields (OpenDynamicThreshold, EndpointDynamicThreshold, CollapseConfigs) ...
+
+  # IP collapsing thresholds (optional; omit or set to 0 for defaults)
+  networkIPGroupThreshold: 50     # Collapse groups of 50+ hosts
+  networkCIDRFloorBits: 16        # No block narrower than /16
+```
+
+Zero or omitted values use the compiled-in defaults (50 and 16 respectively). No operator restart is required; the provider reads the singleton at each request.
+
+## Verifying
+
+**Entry count**: Compare entry counts before and after enabling the feature on a real profile with external traffic. A profile with 8,687 external-traffic entries should drop to a small number (typically hundreds or fewer CIDR entries, depending on traffic distribution).
+
+**CIDR breadth**: Inspect emitted `NetworkNeighbor` entries; no single `IPAddresses` CIDR block should exceed the configured floor (default `/16`).
+
+**Idempotency**: Run the collapse pass twice in succession (e.g., two sequential saves) on the same profile and assert the second pass's output is byte-identical to the first (fixpoint).
+
+**Policy generation**: Feed a collapsed `NetworkNeighborhood` through `GenerateNetworkPolicy` and verify the generated NetworkPolicy includes `IPBlock` rules covering the collapsed CIDRs (test CIDR, bare-IP, and `"*"` cases). Confirm no rule has ports without at least one corresponding peer.
+
+**Known-server enrichment**: Verify that bare-IP entries of `IPAddresses` still receive known-server `PolicyRef` enrichment (identical to singular-field behavior), while CIDR and `"*"` entries produce bare `IPBlock` peers without enrichment.

pkg/apis/softwarecomposition/collapse_types.go

@@ -57,6 +57,15 @@ type CollapseConfigurationSpec struct {
 	// longest-prefix-wins. It REPLACES the compiled-in defaults wholesale
 	// (no merge); include any default prefix you want to keep.
 	CollapseConfigs []CollapseConfigEntry
+	// NetworkIPGroupThreshold is the count threshold above which a group of
+	// NetworkNeighbor entries (sharing Type/DNS/selectors, differing only by
+	// IP) gets CIDR-collapsed. Omitted or 0 means "use the compiled-in
+	// default" (a literal 0 would collapse every group of size 1).
+	NetworkIPGroupThreshold int32
+	// NetworkCIDRFloorBits is the minimum CIDR prefix length (maximum
+	// breadth) a single aggregated block may have. Omitted or 0 means "use
+	// the compiled-in default".
+	NetworkCIDRFloorBits int32
 }
 
 // CollapseConfigEntry is one per-prefix threshold override.

pkg/apis/softwarecomposition/networkpolicy/v2/networkpolicy.go

@@ -1,18 +1,19 @@
 package networkpolicy
 
 import (
-	"bytes"
 	"crypto/sha256"
-	"encoding/gob"
 	"encoding/hex"
+	"encoding/json"
 	"fmt"
 	"net"
+	"net/netip"
 	"sort"
 	"strings"
 
 	helpersv1 "github.com/kubescape/k8s-interface/instanceidhandler/v1/helpers"
 	"github.com/kubescape/storage/pkg/apis/softwarecomposition"
 	"github.com/kubescape/storage/pkg/apis/softwarecomposition/networkpolicy"
+	"github.com/kubescape/storage/pkg/registry/file/networkmatch"
 
 	"github.com/kubescape/go-logger"
 	"github.com/kubescape/go-logger/helpers"
@@ -161,6 +162,18 @@ func listEgressNetworkNeighbors(nn *softwarecomposition.NetworkNeighborhood) []s
 
 }
 
+// containsIPBlockPeer reports whether peers already contains an entry with the given CIDR.
+// Used to avoid duplicate peer entries when merging rules that reference the same IP
+// from multiple distinct NetworkNeighbor entries (e.g. the same peer seen across containers).
+func containsIPBlockPeer(peers []softwarecomposition.NetworkPolicyPeer, cidr string) bool {
+	for _, existing := range peers {
+		if existing.IPBlock != nil && existing.IPBlock.CIDR == cidr {
+			return true
+		}
+	}
+	return false
+}
+
 func mergeIngressRulesByPorts(rules []softwarecomposition.NetworkPolicyIngressRule) []softwarecomposition.NetworkPolicyIngressRule {
 	type PortProtocolKey struct {
 		Port     int32
@@ -194,7 +207,10 @@ func mergeIngressRulesByPorts(rules []softwarecomposition.NetworkPolicyIngressRu
 				keys = append(keys, key)
 			}
 			for _, peer := range rule.From {
-				if peer.IPBlock != nil {
+				if peer.IPBlock == nil {
+					continue
+				}
+				if !containsIPBlockPeer(merged[key], peer.IPBlock.CIDR) {
 					merged[key] = append(merged[key], peer)
 				}
 			}
@@ -277,7 +293,10 @@ func mergeEgressRulesByPorts(rules []softwarecomposition.NetworkPolicyEgressRule
 				keys = append(keys, key)
 			}
 			for _, peer := range rule.To {
-				if peer.IPBlock != nil {
+				if peer.IPBlock == nil {
+					continue
+				}
+				if !containsIPBlockPeer(merged[key], peer.IPBlock.CIDR) {
 					merged[key] = append(merged[key], peer)
 				}
 			}
@@ -337,7 +356,15 @@ func generateEgressRule(neighbor softwarecomposition.NetworkNeighbor, knownServe
 		}
 	}
 
-	if neighbor.IPAddress != "" {
+	skipPorts := false
+	if len(neighbor.IPAddresses) > 0 {
+		peers, refs := buildIPAddressesPeers(neighbor.IPAddresses, neighbor.DNS, knownServers)
+		egressRule.To = append(egressRule.To, peers...)
+		policyRefs = append(policyRefs, refs...)
+		if len(peers) == 0 && neighbor.PodSelector == nil && neighbor.NamespaceSelector == nil {
+			skipPorts = true
+		}
+	} else if neighbor.IPAddress != "" {
 		// look if this IP is part of any known server
 		if entries, contains := knownServers.Contains(net.ParseIP(neighbor.IPAddress)); contains {
 			for _, entry := range entries {
@@ -377,6 +404,10 @@ func generateEgressRule(neighbor softwarecomposition.NetworkNeighbor, knownServe
 		}
 	}
 
+	if skipPorts {
+		return egressRule, policyRefs
+	}
+
 	portMap := make(map[PortProtocolKey]bool)
 	for _, networkPort := range neighbor.Ports {
 		protocol := v1.Protocol(strings.ToUpper(string(networkPort.Protocol)))
@@ -416,7 +447,15 @@ func generateIngressRule(neighbor softwarecomposition.NetworkNeighbor, knownServ
 		}
 	}
 
-	if neighbor.IPAddress != "" {
+	skipPorts := false
+	if len(neighbor.IPAddresses) > 0 {
+		peers, refs := buildIPAddressesPeers(neighbor.IPAddresses, neighbor.DNS, knownServers)
+		ingressRule.From = append(ingressRule.From, peers...)
+		policyRefs = append(policyRefs, refs...)
+		if len(peers) == 0 && neighbor.PodSelector == nil && neighbor.NamespaceSelector == nil {
+			skipPorts = true
+		}
+	} else if neighbor.IPAddress != "" {
 		// look if this IP is part of any known server
 		if entries, ok := knownServers.Contains(net.ParseIP(neighbor.IPAddress)); ok {
 			for _, entry := range entries {
@@ -455,6 +494,10 @@ func generateIngressRule(neighbor softwarecomposition.NetworkNeighbor, knownServ
 		}
 	}
 
+	if skipPorts {
+		return ingressRule, policyRefs
+	}
+
 	portMap := make(map[PortProtocolKey]bool)
 	for _, networkPort := range neighbor.Ports {
 		protocol := v1.Protocol(strings.ToUpper(string(networkPort.Protocol)))
@@ -473,6 +516,86 @@ func generateIngressRule(neighbor softwarecomposition.NetworkNeighbor, knownServ
 	return ingressRule, policyRefs
 }
 
+// buildIPAddressesPeers builds NetworkPolicyPeer/PolicyRef pairs from the plural
+// NetworkNeighbor.IPAddresses field, shared by generateEgressRule/generateIngressRule.
+func buildIPAddressesPeers(ipAddresses []string, dns string, knownServers softwarecomposition.IKnownServersFinder) ([]softwarecomposition.NetworkPolicyPeer, []softwarecomposition.PolicyRef) {
+	var peers []softwarecomposition.NetworkPolicyPeer
+	var policyRefs []softwarecomposition.PolicyRef
+
+	for _, entry := range ipAddresses {
+		if prefix, err := netip.ParsePrefix(entry); err == nil {
+			if !prefix.Addr().Is4() {
+				continue // IPv6 CIDR, out of scope (AC9)
+			}
+			peers = append(peers, softwarecomposition.NetworkPolicyPeer{
+				IPBlock: &softwarecomposition.IPBlock{CIDR: entry},
+			})
+			if dns != "" {
+				// no single original IP for a CIDR range
+				policyRefs = append(policyRefs, softwarecomposition.PolicyRef{
+					DNS:        dns,
+					IPBlock:    entry,
+					OriginalIP: "",
+				})
+			}
+			continue
+		}
+
+		if entry == networkmatch.AnyIPSentinel {
+			const anyCIDR = "0.0.0.0/0"
+			peers = append(peers, softwarecomposition.NetworkPolicyPeer{
+				IPBlock: &softwarecomposition.IPBlock{CIDR: anyCIDR},
+			})
+			if dns != "" {
+				policyRefs = append(policyRefs, softwarecomposition.PolicyRef{
+					DNS:        dns,
+					IPBlock:    anyCIDR,
+					OriginalIP: "",
+				})
+			}
+			continue
+		}
+
+		addr, err := netip.ParseAddr(entry)
+		if err != nil || !addr.Is4() {
+			continue // IPv6 or unparseable, out of scope (AC9)
+		}
+
+		// bare IPv4: mirror the singular IPAddress path exactly, including known-server enrichment
+		if entries, contains := knownServers.Contains(net.ParseIP(entry)); contains {
+			for _, ks := range entries {
+				peers = append(peers, softwarecomposition.NetworkPolicyPeer{
+					IPBlock: &softwarecomposition.IPBlock{CIDR: ks.GetIPBlock()},
+				})
+
+				policyRef := softwarecomposition.PolicyRef{
+					Name:       ks.GetName(),
+					OriginalIP: entry,
+					IPBlock:    ks.GetIPBlock(),
+					Server:     ks.GetServer(),
+				}
+				if dns != "" {
+					policyRef.DNS = dns
+				}
+				policyRefs = append(policyRefs, policyRef)
+			}
+		} else {
+			ipBlock := getSingleIP(entry)
+			peers = append(peers, softwarecomposition.NetworkPolicyPeer{IPBlock: ipBlock})
+
+			if dns != "" {
+				policyRefs = append(policyRefs, softwarecomposition.PolicyRef{
+					DNS:        dns,
+					IPBlock:    ipBlock.CIDR,
+					OriginalIP: entry,
+				})
+			}
+		}
+	}
+
+	return peers, policyRefs
+}
+
 func getSingleIP(ipAddress string) *softwarecomposition.IPBlock {
 	ipBlock := &softwarecomposition.IPBlock{CIDR: ipAddress + "/32"}
 	return ipBlock
@@ -498,12 +621,16 @@ func IsAvailable(nn *softwarecomposition.NetworkNeighborhood) bool {
 	}
 }
 
+// hash must be a deterministic function of s's contents: gob's map encoding follows Go's
+// randomized map iteration order, so gob-encoding a struct containing a map (e.g. a
+// LabelSelector's MatchLabels) previously produced a different byte sequence - and thus a
+// different hash - across otherwise-identical calls. json.Marshal sorts map keys, giving a
+// stable encoding regardless of map iteration order.
 func hash(s any) (string, error) {
-
-	var b bytes.Buffer
-	if err := gob.NewEncoder(&b).Encode(s); err != nil {
+	b, err := json.Marshal(s)
+	if err != nil {
 		return "", err
 	}
-	vv := sha256.Sum256(b.Bytes())
+	vv := sha256.Sum256(b)
 	return hex.EncodeToString(vv[:]), nil
 }