eks-recon: harden node-subnet, PDB, and per-namespace workload facts (#184)

Skill (skills/eks-recon) + DevOps Agent twin (devops-agent/eks-recon):

- networking §2a: add node_subnets AZ-resolution fact (EC2 kubelet nodes
  only; Fargate/hybrid contribute none by design). Handle the 3-segment
  Fargate providerID; mark node_subnets unconfirmed (never count: 0) when
  the node list is obtained but EC2 resolution fails.
- workloads §8: record PDB status.disruptionsAllowed and
  spec.unhealthyPodEvictionPolicy; max_unavailable is int|string.
- workloads summary/by_namespace: per-namespace hpas/pdbs counts;
  kube-*-scoped deployments/services are int|null (null, not false-0).
- Twin mirrors the parent facts via Coverage enumeration (no MCP, per the
  DevOps Agent runtime constraints).

Website doc copies regenerated via misc/update-pages.sh.

Known limitation (#182): the MCP path documents list_k8s_resources for
spec/status fields, which that tool summarizes out. CLI path unaffected;
fixed separately.

Author: utkarpun

devops-agent/eks-recon/references/compute.md

@@ -366,6 +366,10 @@ for n in v1.list_node().items:
     })
 ```
 
+> **Node subnets / AZ:** the subnets EC2 nodes run in (aggregated per subnet, with node counts)
+> are a networking fact — see `references/networking.md` §2a (`networking.node_subnets`).
+> Neither schema carries a per-node subnet/AZ mapping.
+
 **Example output:**
 ```json
 {

devops-agent/eks-recon/references/networking.md

@@ -10,6 +10,7 @@
 - [Detection Capabilities](#detection-capabilities)
   - [1. VPC Identifiers & Endpoint Access](#1-vpc-identifiers--endpoint-access)
   - [2. Subnets & IP Address Availability](#2-subnets--ip-address-availability)
+  - [2a. Node Subnets & AZ Resolution](#2a-node-subnets--az-resolution)
   - [3. CNI Vendor & VPC CNI Configuration](#3-cni-vendor--vpc-cni-configuration)
   - [3a. kube-proxy Mode](#3a-kube-proxy-mode)
   - [4. Ingress Controllers & Gateway API](#4-ingress-controllers--gateway-api)
@@ -47,8 +48,9 @@ This module reads facts from two sources, both read-only:
 If the Kubernetes API is unreachable (access entry absent), report the AWS-API facts and mark
 every K8s-dependent sub-fact (`cni.type`/`cni.vpc_cni` env facts, `kube_proxy.*`, `ingress.*`,
 `gateway_api.*`, `load_balancers.services`/`target_group_bindings`, `service_mesh.*`,
-`dns.coredns` deployment facts / `nodelocal_dns`, `network_policies.*`, `external_dns.*`) as
-`unconfirmed` in the report's Coverage section — never as `false`/`count: 0`.
+`dns.coredns` deployment facts / `nodelocal_dns`, `network_policies.*`, `external_dns.*`, and
+`node_subnets` — its step-1 node list is a K8s-API read) as `unconfirmed` in the report's
+Coverage section — never as `false`/`count: 0`.
 
 > **Reference pseudocode note.** Code blocks labeled *reference pseudocode (kubernetes client)*
 > below illustrate the resource, fields, and RBAC verbs for each K8s-API read. They are **not
@@ -64,6 +66,7 @@ Network configuration spans multiple layers:
 ```
 1. VPC identifiers & endpoint access -> Which VPC, subnets, SGs; how the API server is reached
 2. Subnets & IP availability          -> Per-subnet free IPs, secondary CIDRs
+2a. Node subnets & AZ                  -> Subnets nodes actually run in, AZ resolved for all
 3. CNI vendor & VPC CNI config         -> Pod networking vendor, mode, env vars
 4. Ingress & Gateway API               -> How external traffic enters the cluster
 5. Load balancers                      -> Provisioned ELBs and target group bindings
@@ -162,6 +165,94 @@ aws ec2 describe-vpcs --vpc-ids <vpc-id> --region <region> \
 ]
 ```
 
+### 2a. Node Subnets & AZ Resolution
+
+**Why check this:** The `subnets` list in detection 2 resolves AZ/CIDR only for the subnets in
+`cluster.resourcesVpcConfig.subnetIds` (the cluster-registered list). EKS permits node groups to be
+deployed into subnets that were not specified at cluster creation (network-reqs, "Subnet
+requirements for nodes"), so nodes can run in subnets absent from that list — those node subnets
+otherwise carry no AZ, CIDR, or free-IP fact anywhere in recon. This detection resolves them.
+
+**Scope:** EC2 (kubelet) node subnets only. Fargate and Hybrid nodes are not EC2 instances and
+by design contribute no `node_subnets` entry. ENIConfig (custom-networking) *pod* subnets are
+also out of scope — this is node placement, not pod-ENI placement.
+
+**Via Kubernetes API (step 1)** — read the node list and extract EC2 instance ids from providerIDs:
+
+- **Resource:** `Node`, group/version `v1` (core).
+- **Fields to extract:** `spec.providerID` for each node.
+- **Filter:** providerIDs use several schemes:
+  - EC2:     `aws:///<az>/<instance-id>`
+  - Fargate: `aws:///<az>/<profile-or-task-id>/fargate-ip-<a-b-c-d>.<region>.compute.internal`
+    (3-segment path after `aws:///`, no EC2 instance; observed example in containers-roadmap#1976
+    — no authoritative doc format exists)
+  - Hybrid:  `eks-hybrid:///<region>/<cluster>/<node-name>` (no EC2 instance)
+
+  Keep ONLY `aws:///` providerIDs whose last segment is a real instance id (`^i-`). The
+  filter is MANDATORY: a Fargate/hybrid/empty id makes the whole `describe-instances
+  --instance-ids` call abort (Malformed/NotFound) and yields zero node_subnets facts. The
+  last-segment `^i-` match is unaffected by the Fargate shape (its last segment is a hostname,
+  not `i-*`); positional parsing that assumed the old 2-segment Fargate scheme would misfire.
+- **RBAC verbs:** `get`, `list` on `nodes`.
+
+*Reference pseudocode (kubernetes client), not executable:*
+```python
+v1 = client.CoreV1Api()
+instance_ids = []
+for n in v1.list_node().items:
+    pid = n.spec.provider_id or ""
+    if pid.startswith("aws:///"):
+        last = pid.split("/")[-1]
+        if last.startswith("i-"):
+            instance_ids.append(last)
+```
+
+**Via AWS API** — map each running node instance to its subnet, then resolve AZ/CIDR/free-IP for
+every distinct node subnet:
+
+```bash
+# step 2: Map each RUNNING node instance to its subnet id; group by SubnetId for node_count.
+# Uses --instance-ids (simpler/idiomatic); one fully-purged id aborts the whole batch
+# (InvalidInstanceID.NotFound) — see the residual step-2 breaker in Edge Cases.
+# The running-state filter drops a node terminated mid-recon (SubnetId null) that would
+# otherwise poison the batch.
+aws ec2 describe-instances --instance-ids <instance-ids> --region <region> \
+  --filters Name=instance-state-name,Values=running \
+  --query 'Reservations[].Instances[].{instance:InstanceId,subnet:SubnetId}'
+
+# step 3: Resolve AZ + CIDR + free-IP for EVERY distinct node subnet id. describe-subnets is not
+# limited to the cluster-registered list — it resolves any subnet visible to the caller's
+# credentials (owned or RAM-shared).
+aws ec2 describe-subnets --subnet-ids <distinct-node-subnet-ids> --region <region> \
+  --query 'Subnets[].{id:SubnetId,az:AvailabilityZone,az_id:AvailabilityZoneId,cidr:CidrBlock,free:AvailableIpAddressCount,vpc_id:VpcId}'
+```
+
+- `node_subnets` = count+list of `{id, az, az_id, cidr, free, vpc_id, node_count, in_cluster_subnet_list}`,
+  one entry per distinct subnet an EC2 node runs in. Empty list is a valid state (Fargate-only or
+  zero EC2-node cluster): `count: 0, list: []`. When the node list is unobtainable, OR the node list
+  IS obtained but the EC2 describe calls fail (describe-instances / describe-subnets: AccessDenied /
+  unreachable / mid-flow abort), leave the value null AND mark `node_subnets` **unconfirmed** in the
+  report's Coverage section — NEVER emit `count: 0` (a failed EC2 resolve is not a Fargate-only
+  cluster; `count: 0` would read as one). The bare `count: 0, list: []` is reserved for the genuine
+  "no EC2 nodes" case above.
+- `az` / `az_id` / `cidr` / `free` = `AvailabilityZone` / `AvailabilityZoneId` / `CidrBlock` /
+  `AvailableIpAddressCount` from the describe-subnets call (step 3), resolved for every node subnet
+  including those outside the cluster-registered list. `az_id` is the cross-account-stable zone
+  identifier (AZ *names* are account-relative — matters for shared / cross-account subnets).
+- `vpc_id` = the subnet's `VpcId` (ties an unregistered entry back to its VPC).
+- `node_count` = number of running nodes in that subnet (from the describe-instances grouping).
+- `in_cluster_subnet_list` = `true` when the subnet id is present in `subnet_ids`
+  (`cluster.resourcesVpcConfig.subnetIds`), else `false`. A `false` entry is a node subnet absent
+  from `resourcesVpcConfig.subnetIds` — e.g. a node group launched into an unregistered subnet.
+
+**Example output:**
+```json
+[
+  {"id": "subnet-0aaa111", "az": "us-west-2a", "az_id": "usw2-az1", "cidr": "10.0.1.0/24", "free": 210, "vpc_id": "vpc-0abc123", "node_count": 3, "in_cluster_subnet_list": true},
+  {"id": "subnet-0ddd444", "az": "us-west-2c", "az_id": "usw2-az3", "cidr": "10.4.0.0/20", "free": 4051, "vpc_id": "vpc-0abc123", "node_count": 5, "in_cluster_subnet_list": false}
+]
+```
+
 ### 3. CNI Vendor & VPC CNI Configuration
 
 **Why check this:** The primary CNI vendor determines pod networking behavior. VPC CNI mode
@@ -511,6 +602,23 @@ networking:
         free: int                   # AvailableIpAddressCount
   vpc_secondary_cidrs: list         # aws ec2 describe-vpcs CidrBlockAssociationSet (beyond primary)
 
+  # --- Node subnets (subnets nodes actually run in; AZ resolved for ALL, registered or not) ---
+  node_subnets:                     # aws ec2 describe-subnets over EC2 node instance SubnetIds
+                                    # EC2 nodes only (Fargate/hybrid contribute none).
+                                    # node list unobtainable OR EC2 describe fails => value null + mark unconfirmed
+                                    # in Coverage w/ distinguishing reason ("node list unobtainable — K8s API read
+                                    # failed" vs "EC2 describe failed: <detail>"), never count:0 (see Access Model + §2a)
+    count: int
+    list:
+      - id: string                  # SubnetId
+        az: string                  # AvailabilityZone (resolved for every node subnet)
+        az_id: string               # AvailabilityZoneId — cross-account-stable zone id
+        cidr: string                # CidrBlock
+        free: int                   # AvailableIpAddressCount
+        vpc_id: string              # VpcId (ties an unregistered subnet to its VPC)
+        node_count: int             # running nodes in this subnet
+        in_cluster_subnet_list: bool # true if id is in subnet_ids (resourcesVpcConfig.subnetIds)
+
   # --- CNI ---
   cni:
     type: string                    # aws-vpc-cni | calico | cilium | auto-mode | other (detected, not assumed)
@@ -643,3 +751,28 @@ Non-default settings surface through the aws-node env vars (Security Groups for
 Per-subnet free-IP counts are recorded as facts in `subnets.list[].free` (from
 `AvailableIpAddressCount`). Secondary VPC CIDRs appear in `vpc_secondary_cidrs`. Report the
 numbers; draw no conclusion.
+
+### Node Subnets (§2a) Edge Cases
+
+- **Zero EC2 nodes / Fargate-only cluster:** the step-1 filter yields an empty id list. Do not
+  call `describe-instances` with no ids (region-wide fallback would return unrelated instances).
+  Emit `node_subnets: {count: 0, list: []}` — an empty list is a valid fact.
+- **Fargate / Hybrid nodes present:** their providerIDs are dropped by the `^i-` filter by design;
+  they contribute no `node_subnets` entry (not EC2 instances).
+- **Node churn mid-recon:** a node terminated between steps 1 and 2 is excluded by the
+  `instance-state-name=running` filter, so its `null` SubnetId can't break the step-2
+  group-by-SubnetId or the subnet-id dedup feeding step 3.
+  (A Node object stale >~1h whose instance is fully purged is the one residual step-2 breaker —
+  `InvalidInstanceID.NotFound` aborts the describe-instances call; rare, not caught by `--filters`.)
+- **Large clusters:** `describe-instances --instance-ids` is unpaginated — the EC2 API guidance
+  warns unpaginated requests are throttling- and timeout-prone — so chunk the instance ids across
+  calls and merge, then dedup subnet ids for step 3 and chunk that deduped `--subnet-ids` list into
+  batches the same way (the same batching approach used for the describe-instances `--instance-ids`
+  step, since `describe-subnets --subnet-ids` is likewise unpaginated).
+- **Node list unobtainable:** value stays null (module-level null rule) AND mark `node_subnets`
+  unconfirmed in the report's Coverage section (its step-1 node list is a K8s-API read), with
+  `reason: "node list unobtainable — K8s API read failed"`.
+- **Node list obtained but EC2 describe calls fail** (AccessDenied / unreachable / mid-flow abort):
+  likewise leave the value null and mark `node_subnets` unconfirmed in the Coverage section, with
+  `reason: "EC2 describe failed: <detail>"` — do NOT emit `count: 0` (a failed EC2 resolve is not a
+  Fargate-only cluster; `count: 0` reads as one).

devops-agent/eks-recon/references/workloads.md

@@ -288,25 +288,34 @@ workloads scale automatically and their scaling thresholds.
 Enumerate PodDisruptionBudgets (PDBs) across namespaces. A PDB records the minimum
 availability guarantee for the pods matched by its selector. Record existence, the
 `minAvailable` / `maxUnavailable` value, and the label selector (the workloads it covers).
+Also record `status.disruptionsAllowed` — the point-in-time count of voluntary evictions the
+Eviction API currently permits for the pods this PDB selects. `0` means it currently rejects
+voluntary eviction of covered healthy pods, which can stall a drain of the nodes hosting them
+(a PDB selecting zero pods also reports `0`). Record `spec.unhealthyPodEvictionPolicy` too — it
+governs whether unhealthy covered pods are evictable at `0`.
 
 **Via Kubernetes API** — list PDBs across all namespaces (VERIFIED live: PDBs expose
 MIN AVAILABLE / MAX UNAVAILABLE):
 
 - **Resource:** `PodDisruptionBudget`, group/version `policy/v1`, all namespaces.
 - **Fields to extract:** `metadata.namespace`, `metadata.name`, `spec.minAvailable`,
-  `spec.maxUnavailable`, `spec.selector.matchLabels` (→ `selector`).
+  `spec.maxUnavailable`, `status.disruptionsAllowed` (→ `disruptions_allowed`),
+  `spec.unhealthyPodEvictionPolicy` (→ `unhealthy_pod_eviction_policy`),
+  `spec.selector.matchLabels` (→ `selector`).
 - **RBAC verbs:** `get`, `list` on `poddisruptionbudgets.policy`.
 
-Exactly one of `min_available` / `max_unavailable` is set per PDB; the other is `null`.
-The `selector` matchLabels identify the covered workloads.
+At most one of `min_available` / `max_unavailable` is set per PDB (both may be null for a
+selector-only PDB). The `selector` matchLabels identify the covered workloads.
 
 **Example (one PDB):**
 ```json
 {
   "namespace": "production",
   "name": "api-gateway-pdb",
-  "min_available": "1",
+  "min_available": 1,
   "max_unavailable": null,
+  "disruptions_allowed": 2,
+  "unhealthy_pod_eviction_policy": null,
   "selector": {"app": "api-gateway"}
 }
 ```
@@ -441,23 +450,32 @@ detected; never omit a key. Aggregate containers use the `{count, list}` wrapper
 ```yaml
 workloads:
   summary:
-    deployments: int
+    deployments: int               # kube-*-scoped total (§1 excludes kube-*); a true 0 is a valid fact
     statefulsets: int
     daemonsets: int
     cronjobs: int
     jobs: int
-    services: int
+    services: int                  # kube-*-scoped total (§5 excludes kube-*); a true 0 is a valid fact
     ingresses: int
     hpas: int
     pdbs: int
     namespaces_with_workloads: int
 
+  # kube-* scope split across these columns: ONLY deployments and services exclude kube-*
+  # namespaces (their §1/§5 listings filter kube-* out). statefulsets, ingresses, hpas, and pdbs
+  # INCLUDE kube-* (their listings apply no namespace filter). A namespace row exists wherever ANY
+  # column has a count, so rows may be partial. For a scoped-out column emit `null`, NOT 0 (schema
+  # header rule: null = fact not detected/collected) — e.g. a kube-system row shows `null` for
+  # deployments and services (column scoped out, not "zero found"; kube-system always runs CoreDNS
+  # Deployments + kube-dns Services) but real counts for statefulsets/ingresses/hpas/pdbs.
   by_namespace:
     - namespace: string
-      deployments: int
+      deployments: int|null          # null for kube-* rows: §1 scopes kube-* out, so not counted here (never 0)
       statefulsets: int
-      services: int
+      services: int|null             # null for kube-* rows: §5 scopes kube-* out, so not counted here (never 0)
       ingresses: int
+      hpas: int                    # per-namespace HPA count (rolled up from the hpas.list below)
+      pdbs: int                    # per-namespace PDB count (rolled up from the pdbs.list below)
 
   deployments:
     count: int
@@ -545,8 +563,11 @@ workloads:
     list:
       - namespace: string
         name: string
-        min_available: string      # spec.minAvailable (one of min/max set, other null)
-        max_unavailable: string    # spec.maxUnavailable
+        min_available: int|string  # spec.minAvailable — plain int or "N%" string (K8s IntOrString); at most one is set (both may be null for a selector-only PDB)
+        max_unavailable: int|string  # spec.maxUnavailable — plain int or "N%" string (K8s IntOrString)
+        disruptions_allowed: int   # status.disruptionsAllowed — voluntary evictions the Eviction API currently permits for
+                                   # covered pods; 0 rejects eviction of covered healthy pods (also 0 when it selects no pods); null if status unpopulated
+        unhealthy_pod_eviction_policy: string  # spec.unhealthyPodEvictionPolicy — emit verbatim; null when unset (API defaults behavior to IfHealthyBudget)
         selector: object           # spec.selector.matchLabels — covered workloads
 
   priority_classes:

misc/website/docs/devops-agent/eks-recon/references/compute.md

@@ -377,6 +377,10 @@ for n in v1.list_node().items:
     })
 ```
 
+> **Node subnets / AZ:** the subnets EC2 nodes run in (aggregated per subnet, with node counts)
+> are a networking fact — see `references/networking.md` §2a (`networking.node_subnets`).
+> Neither schema carries a per-node subnet/AZ mapping.
+
 **Example output:**
 ```json
 {