GoogleCloudPlatform · mikesson · Jul 10, 2026 · Jul 10, 2026 · Jul 10, 2026
@@ -62,3 +62,8 @@ spec:
       overviewServiceNamespace: {{ .Values.namespace | quote }}
       hardDependencies:
         - workloads-common
+    - name: abfs
+      path: gitops/modules/abfs-module
+      overviewPath: portal/overview.html
+      overviewService: mod-abfs-overview
+      overviewServiceNamespace: abfs
@@ -0,0 +1,102 @@
+# Horizon-SDV: ABFS Deployment Learnings & Troubleshooting Knowledge Base
+
+This document lists and captures all critical technical insights, root-cause analyses, and troubleshooting workflows gained during the end-to-end deployment and tuning of the Android Build File System (ABFS) on GKE.
+
+---
+
+## 1. GKE Node Pool Taints & DNS Blackout
+
+> [!WARNING]
+> **Technical Issue:**
+> Standard GKE system-critical pods (such as `kube-dns`, `calico-node`, `metrics-server`, and GKE storage CSI driver controllers) are scheduled dynamically to support the cluster. However, these core system daemonsets **do not carry custom tolerations** for application-specific taints.
+>
+> In our initial configuration, the single node pool `sdv-abfs-build-node-pool` was provisioned with the taint `workloadType=android-abfs`. Because this was the only node pool in the dedicated cluster, system-critical pods were stuck in `Pending` indefinitely. This resulted in a **total DNS blackout inside the cluster**, blocking `abfs-server` from resolving GCP endpoints and causing licensing entitlement validation to fail.
+
+### Root-Cause Analysis
+- GKE Autopilot avoids this by managing system taints natively, but on GKE Standard, custom taints applied to the sole node pool of a cluster will starve system daemonsets.
+- Since `kube-dns` was unschedulable, the `abfs-server` pods could not perform public HTTP lookups to verify the licensing entitlement keys, leaving the server in an unready state.
+
+### Resolution & Actionable Guidance
+1. **Disable Taints on Dedicated Clusters**: Modified the Terraform infrastructure logic to make node-pool taints conditional. If the cluster is entirely dedicated to ABFS (`var.enable_dedicated_abfs_cluster = true`), we omit the `workloadType` taint, as workload isolation is unnecessary on a dedicated cluster.
+2. **Manual Remediation**: Cleared the live node taints using:
+   ```bash
+   kubectl taint nodes <node-name> workloadType:NoSchedule-
+   ```
+3. **Key Learning**: Never apply exclusive custom taints to node pools on GKE Standard unless you have a separate untainted system node pool, or have manually patched system DaemonSets (like `kube-dns`) with corresponding tolerations.
+
+---
+
+## 2. GCP CPU & SSD Storage Quota Exhaustion (QIRs)
+
+> [!IMPORTANT]
+> **Technical Issue:**
+> High-performance developer platform modules require extensive cloud compute and storage resources. Standard GCP project initialization quotas are highly restrictive and will silently block GKE cluster autoscaling.
+
+### The Quota Bottlenecks
+- **CPU Quota**: The project's global CPU quota (`CPUS_ALL_REGIONS`) was capped at **32.0**. Since the main platform cluster (`sdv-cluster`) was already consuming 8 CPUs, GKE was unable to provision the required `n2-highcpu-32` (32 vCPUs) build node, leaving the cluster autoscaler blocked.
+- **Storage Quota**: The initial request for 2,000 GB of Regional SSD (`premium-rwo`) storage in `europe-west3` was denied due to standard regional threshold limits.
+
+### Resolution & Resource Sizing Validation
+1. **CPU Quota Increase**: Guided the user to submit a Quota Increase Request (QIR) for **64 CPUs**, which was successfully approved. This provides comfortable headroom to run the `n2-highcpu-32` compute node alongside core system services.
+2. **SSD Storage Quota Tuning**: Requested and secured a **1,000 GB Regional SSD quota** in `europe-west3`.
+3. **Rigorous Capacity Planning**:
+   - Calculated the storage profile: Gerrit uploader StatefulSet replicas are configured at **270 GiB** each.
+   - With 2 active replicas, the total storage consumed is **540 GiB** (\(2 \times 270\text{ GiB}\)), fitting safely within the approved 1,000 GB regional SSD quota limit.
+   - Disabled the server-side SSD cache (`server.cache.enabled = false`) to eliminate any extra storage footprint on the server.
+
+```
+[Total Approved Quota: 1000 GiB]
+├── Allocated: 540 GiB (Uploader 0 + Uploader 1)  [==================== 54%]
+└── Free Margin: 460 GiB                         [================== 46%]
+```
+
+---
+
+## 3. Spanner IAM Policy & Workload Identity Alignment
+
+> [!CAUTION]
+> **Technical Issue:**
+> The Config Connector (KCC) operator running in the core cluster was blocked from updating the project's IAM policy, resulting in `UpdateFailed` errors for `iampolicymember` resources. This left the Google Service Account (GSA) `abfs-runtime` without the necessary permissions, and the Kubernetes Service Accounts (KSAs) on the dedicated cluster had no Workload Identity bindings or annotations.
+
+### Root-Cause Analysis
+- GKE Workload Identity requires precise bidirectional bindings:
+  1. The KSA must be annotated with the GSA email.
+  2. The GSA must carry an IAM policy binding granting `roles/iam.workloadIdentityUser` to the KSA's identity string: `serviceAccount:<project-id>.svc.id.goog[<namespace>/<ksa-name>]`.
+- If KCC lacks broad project-level IAM permission or is located on a separate control-plane cluster, these bindings will fail to reconcile automatically, leaving pods unable to authenticate with Spanner or GCS.
+
+### Resolution & Automation Script
+Developed and executed an automated terminal script to bypass KCC's IAM limitation and bind the identities securely:
+1. **Grant Project-Level Roles to GSA**:
+   - `roles/spanner.databaseUser` (creates sessions/accesses data)
+   - `roles/storage.objectAdmin` (manages AOSP storage)
+   - `roles/secretmanager.secretAccessor` (accesses pusher-config git credential secrets)
+   - `roles/logging.logWriter` & `roles/monitoring.metricWriter`
+2. **Bind GSA to cluster KSAs via Workload Identity**:
+   ```bash
+   gcloud iam service-accounts add-iam-policy-binding "abfs-runtime@horizon-sdv-deploy-3.iam.gserviceaccount.com" \
+     --role="roles/iam.workloadIdentityUser" \
+     --member="serviceAccount:horizon-sdv-deploy-3.svc.id.goog[abfs/abfs-server]"
+   ```
+3. **Annotate KSAs**: Annotated the KSAs `abfs-server` and `abfs-uploader` with the GSA email in the `abfs` namespace.
+
+---
+
+## 4. CASFS Kernel Module Compilation & Host OS Pinning
+
+### Technical Insight
+The `abfs-casfs-installer` DaemonSet compiles and loads the `casfs` virtual filesystem kernel module directly into the GKE node's host operating system kernel.
+- **Kernel Header Tight-Coupling**: Because compilation is tightly coupled with host Linux kernel headers, GKE nodes running standard Container-Optimized OS (COS) must have their versions strictly pinned.
+- **GKE Management Policy**: Under standard platform configurations, enabling `auto_upgrade` will trigger automatic cluster upgrades, updating the underlying kernel. If the CASFS compilation script does not contain immediate support for the new kernel version, GKE node scaling will succeed but the CASFS installer will crash loop, breaking the file system mounts on Gerrit uploaders.
+- **Best Practice**: For clusters employing custom compiled kernel modules, **always disable node-pool auto-upgrade** (`auto_upgrade = false`) and manage cluster updates through scheduled, validated maintenance windows.
+
+---
+
+## 5. Quick-Reference Troubleshooting Checklist
+
+| Symptom | Probable Cause | Diagnostic Command | Remediation Action |
+|:---|:---|:---|:---|
+| **`kube-dns` pods in `Pending`** | Custom node taints applied to the sole node pool. | `kubectl get pods -n kube-system -o wide` | Untaint the GKE node pool or make the taint conditional in Terraform. |
+| **GKE Nodes fail to scale up** | Global GCE CPU or Regional SSD quota exceeded. | `gcloud compute project-info describe --format="yaml(quotas)"` | Submit a Quota Increase Request (QIR) via the GCP Console. |
+| **ABFS Server fails licensing check** | Cluster DNS failure or metadata server blocked. | `kubectl logs deployment/abfs-server -n abfs` | Verify `kube-dns` is running and the node pool is in GCE-metadata mode. |
+| **`UpdateFailed` on `iampolicymember`** | Config Connector lacks IAM project ownership. | `kubectl describe iampolicymember -n abfs` | Apply bindings manually using the `apply_iam_bindings.sh` script. |
+| **`abfs-gerrit-uploader` crash looping** | `casfs` kernel module not loaded on the host node. | `kubectl logs daemonset/abfs-casfs-installer -n abfs` | Verify host kernel version and ensure node pool allows signed/unsigned custom modules. |
@@ -0,0 +1,107 @@
+# Horizon-SDV: Android Build File System (ABFS) Module Integration Guide
+
+This guide documents in full technical detail how the Android Build File System (ABFS) data-plane module is integrated into the Horizon-SDV platform repository (`https://github.com/mikesson/horizon-sdv`) and deployed via GitOps.
+
+---
+
+## 1. Executive Summary & Architecture Overview
+
+The **Android Build File System (ABFS)** is integrated directly as a native platform application module inside the central GitOps framework of **Horizon-SDV**. It is managed, scaled, and synced declaratively under the core GitOps structures.
+
+```mermaid
+graph TD
+    subgraph GitOps Source of Truth - github.com/mikesson/horizon-sdv
+        GOP[gitops/modules/abfs-module] -->|Child Apps| APP[abfs Application]
+        GOP -->|Validation Workflows| WF[argo-workflows Application]
+        GOP -->|Overview Portal| PORT[mod-abfs-overview Service]
+    end
+
+    subgraph GCP GKE Data Plane - sdv-abfs-cluster
+        APP -->|Deploys workloads| SERVER[abfs-server Deployment]
+        APP -->|Deploys uploaders| UPLOAD[abfs-gerrit-uploader StatefulSet]
+        APP -->|Compiles casfs.ko| DK[abfs-casfs-installer DaemonSet]
+        WF -->|Runs health checks| SMOKE[abfs-smoke-test Workflow]
+        PORT -->|Hosts dashboard| nginx[mod-abfs-overview Pod]
+    end
+
+    subgraph Cloud Infrastructure - horizon-sdv-deploy-3
+        SERVER -->|Read/Write Metadata| SP[(Cloud Spanner)]
+        SERVER -->|Read/Write Blobs| GCS[(Cloud Storage)]
+    end
+```
+
+---
+
+## 2. GitOps Integration Strategy: The Dedicated Module (`/gitops/modules/abfs-module`)
+
+The ABFS module is structured as a native, self-contained **Argo CD App-of-Apps parent chart** located at:
+`gitops/modules/abfs-module/`
+
+This parent module orchestrates the delivery of three major sub-components across your dedicated GKE cluster:
+1. **The Core Data Plane (`abfs/` sub-chart)**: Deploys the main `abfs-server`, Gerrit uploaders, and kernel installers.
+2. **The Validation Engine (`argo-workflows/` sub-chart)**: Deploys automated test suites and smoke-test templates (`abfs-smoke-test`) to run health assertions after deployment.
+3. **The Interactive UI Portal (`portal/` + `templates/module-overview-http.yaml`)**: Spins up a static dashboard (`overview.html`) exposed via a GKE gateway HTTPRoute, providing real-time operational state metrics and link portals for developers.
+
+---
+
+## 3. Module File Layout & Structural Breakdown
+
+The integrated module within your repository is structured as follows:
+
+```
+gitops/modules/abfs-module/
+├── Chart.yaml              # App-of-Apps helm module chart definition (v0.2.0)
+├── values.yaml             # Module-manager context parameters and namespacing config
+├── abfs/                   # Core Data Plane sub-chart
+│   ├── Chart.yaml          # Pinned dependency definitions
+│   ├── values.yaml         # Resource caps, replication overrides, and sharding limits
+│   └── templates/          # Kubernetes manifests (deployments, statefulsets, daemonsets, secrets)
+├── argo-workflows/         # Smoke-test validation sub-chart
+│   ├── Chart.yaml          # Argo-workflows helper definition
+│   ├── values.yaml         # Test execution intervals and parameter overrides
+│   └── templates/          # Declarative Argo WorkflowTemplates for automated testing
+├── templates/              # Parent app-of-apps template rendering manifests
+│   ├── application-abfs.yaml             # Declares the ArgoCD Child App for the abfs workloads
+│   ├── application-argo-workflows.yaml    # Declares the ArgoCD Child App for automated validations
+│   └── module-overview-http.yaml         # Declares HTTPRoutes and Services for the UI Portal
+└── portal/
+    └── overview.html       # HTML5 code for the static status and operational overview page
+```
+
+---
+
+## 4. Key Workload Definitions & Helm Orchestration
+
+### A. Child Applications (`templates/application-*.yaml`)
+The parent chart leverages Argo CD's App-of-Apps pattern to declare child applications:
+- **`application-abfs.yaml`**: Mounts the core ABFS workloads, pinning the source repository URL and targeting the dedicated `sdv-abfs-cluster`.
+- **`application-argo-workflows.yaml`**: Schedules automated `Workflow` resources that execute validation tests against GKE.
+
+### B. The Interactive Dashboard (`portal/overview.html`)
+Spins up a lightweight static Nginx web service (`mod-abfs-overview`) which renders a beautiful, modern CSS status board. It displays connection strings, repository sync statuses, active Gerrit sharding configurations, and diagnostics URLs for developers.
+
+### C. GKE HTTPRoute Gateway Routing (`templates/module-overview-http.yaml`)
+Ensures that the dashboard is automatically exposed under the platform domain namespace via an Envoy-backed GKE Gateway.
+
+---
+
+## 5. Architectural Rationale: HOW and WHY
+
+### A. Why an App-of-Apps Module inside `/gitops/modules/`?
+* **HOW**: Integrated the ABFS templates as a sub-chart under `gitops/modules/abfs-module/`.
+* **WHY**:
+  1. **Platform Unified Lifecycle**: It allows the Horizon DevOps Module Manager to dynamically toggle, upgrade, and configure ABFS alongside other modules (like `gerrit` or `mtk-connect`).
+  2. **Argo CD Synchronization**: By packing workloads into an App-of-Apps pattern, Argo CD automatically tracks and reconciles application state drift on the dedicated GKE cluster while pulling direct updates from `github.com/mikesson/horizon-sdv`.
+
+### B. Why is the GCS/Spanner Config Connector (KCC) Mapped to Helm Values?
+* **HOW**: Parameterized all IAM, GCS, and Spanner manifests into clean Go Helm template statements in the sub-charts:
+  ```yaml
+  projectID: {{ .Values.config.projectID }}
+  ```
+* **WHY**:
+  - **Environment Portability**: Eliminates hardcoded environment values (like project IDs, GCS bucket names, and DNS values). This ensures the identical module can be deployed dynamically across staging, production, and sandbox environments without modifying the base git branch.
+
+### C. Why are CASFS Host-Kernel Settings Isolated?
+* **HOW**: Pinned the host-level DaemonSet and privilege contexts (`privileged: true`, `hostPID: true`) within the dedicated `abfs` sub-chart templates.
+* **WHY**:
+  - **Security Bound Isolation**: Separating the host-privileged `casfs` compilation workloads from standard, non-privileged cluster applications keeps the platform core workspace completely secure and secure.