Feature/agent platform by allamand · Pull Request #529 · aws-samples/appmod-blueprints

allamand · 2026-03-13T08:10:13Z

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

allamand · 2026-03-13T08:16:55Z

+
+## CI/CD Integration
+
+### GitLab CI Pipeline


You want to use gitlab for cicd ?

allamand · 2026-03-13T08:18:52Z

For backstage we was wondering if we scope this out if the project and just use the one in CNOE and using existing OSS plugins for kro, gitlab integration…

allamand · 2026-03-13T08:24:20Z

+
+## ApplicationSet Patterns
+
+### List Generator Pattern


We should include cluster generator here as it is the one we are using most in the project

allamand · 2026-03-13T08:31:43Z

+- Integration tests with AWS
+- Deployment validation
+
+## Backstage Integration


I disagree on this. Really backstage should not create things in Kubernetes êtes directly but only create things through gitops : PR, push

This can be fixed.

allamand · 2026-03-13T08:40:10Z

+
+### Cluster Design Principles
+1. **Multi-AZ Deployment**: Spread across 3 availability zones
+2. **Managed Node Groups**: Use EKS managed node groups


We don’t want to use managed nodes groupes but Eks auto mode

This can be fixed.###

allamand · 2026-03-13T08:43:01Z

+}
+```
+
+### IRSA Configuration


Remove this

allamand · 2026-03-13T08:46:38Z

+
+#### ModelConfig CRD
+
+```yaml


Can we add another example pointing to a ray endpoint in the cluster instead of bedrock. ?

This can be fixed.

allamand · 2026-03-13T08:48:34Z

+    model: anthropic.claude-3-5-sonnet-20241022-v2:0
+    region: us-east-1
+
+  # Service account with IRSA


Can we use pod identity instead of IRSA ?

This can be fixed.

allamand · 2026-03-13T08:53:47Z

+
+### Overview
+
+Agent Core Components provision AWS Bedrock Agent Core capabilities (Memory, Browser, Code Interpreter) using Tofu Controller.


Can we deploy this using kro/ACK instead of open tofu/terrzform ? To have consistency with the platform

There is no support for ACK support for Agent core components. So this is the only approach we did POC.

allamand · 2026-03-13T09:02:18Z

+
+> **Note**: There is no Terraform in the `appmod-blueprints` solution repo. Initial EKS cluster creation (via Terraform, CDK, eksctl, etc.) lives in the customer's own infra repo or the workshop repo (`platform-engineering-on-eks`). Once the hub cluster exists, it self-manages via Kro+ACK/CrossPlane compositions and ArgoCD.
+
+### Changes in `sample-agent-platform-on-eks` Repository


What are the need to create another git repo for that ? We should only use appmod-blueprints Also for agents

It is absolutely required for extending the platform to agent platform work we are doing separate. Thats the core reason for this refactor. This is our core tenet

allamand · 2026-03-13T09:48:36Z

+3. **Config-external**: `hub-config.yaml` lives outside the repo; customers pass their own config. The config drives Kro/CrossPlane compositions and ArgoCD bootstrap.
+4. **Provider-agnostic**: Git provider (GitHub vs GitLab vs CodeCommit), OIDC provider, and CI/CD provider are swappable via configuration
+5. **GitOpsy spokes**: Spoke clusters provisioned and managed via CrossPlane/Kro through the hub cluster — same mechanism as hub self-management
+6. **Workshop as a pattern, not a fork**: Workshop-specific code lives in `patterns/workshop/` within the main repo (alongside other consumption patterns like `patterns/hub-only/`, `patterns/full-platform/`). The workshop pattern includes CloudFront, GitLab integration, Identity Center setup, and workshop-specific configurations. Heavy workshop orchestration (Terraform for cluster creation, deploy scripts) lives in the internal `platform-engineering-on-eks` GitLab repo.


platform specific will be in the pattern repo. there is no terraform code to be in internal platform-engineering-on-eks, we just reuse the generic code to create hub. specifics scripts will also leave in tha workshop pattern scrips dir as they only apply on how we deploy the platform and ca be reference by users wanting to use other patterns as well

allamand · 2026-03-13T09:52:47Z

+│       └── README.md                 #   Workshop deployment guide (references platform-engineering-on-eks)
+├── applications/                     # UNCHANGED: Sample apps
+├── backstage/                        # UNCHANGED: Backstage IDP
+├── gitops/                           # REFACTORED: GitOps configurations


What is refactor here ? looks UNCHANGED for me

allamand · 2026-03-13T09:54:48Z

+```
+
+> **Key changes**:
+> - The `platform/infra/terraform/` directory is removed from `appmod-blueprints`. All Terraform code for cluster creation, GitLab PATs, and workshop-specific infra moves to the `platform-engineering-on-eks` internal GitLab repo. The solution repo is purely GitOps-native for ongoing management.


I think we need to keep this terraform cluster to create hub cluster in this repo. it is optional. workshop pattern and full will use it, while other patterns may use existing clusters. there is no point/advantages moving this to gitlab

Terraform module to create hub cluster stays in this repo.

allamand · 2026-03-13T09:55:47Z

+> **Key changes**:
+> - The `platform/infra/terraform/` directory is removed from `appmod-blueprints`. All Terraform code for cluster creation, GitLab PATs, and workshop-specific infra moves to the `platform-engineering-on-eks` internal GitLab repo. The solution repo is purely GitOps-native for ongoing management.
+> - `modules/hub-provisioning/` provides a turnkey Terraform module that customers can `source` from GitHub to provision the hub cluster and bootstrap the platform. After bootstrap, the platform is self-managing.
+> - `examples/` is renamed to `patterns/` to better reflect that these are consumption patterns, not just examples. The `workshop/` pattern is a first-class citizen alongside other patterns. Workshop-specific configuration (CloudFront, GitLab, Identity Center) lives in `patterns/workshop/`; heavy workshop orchestration (Terraform, deploy scripts) lives in `platform-engineering-on-eks`.


There is no examples/ folder in current setup, what are you refering to ?

There is no examples/ folder in current setup, what are you refering to ?

@allamand the proposal is to have a folder called patterns or blueprints - see discussion on the slack channel. Under patterns we can have a workshop folder that will contain workshop specific content. This is to address the fact that we cannot move all workshop specific content to the gitlab for workshop content.

allamand · 2026-03-13T09:56:33Z

+> **Key changes**:
+> - The `platform/infra/terraform/` directory is removed from `appmod-blueprints`. All Terraform code for cluster creation, GitLab PATs, and workshop-specific infra moves to the `platform-engineering-on-eks` internal GitLab repo. The solution repo is purely GitOps-native for ongoing management.
+> - `modules/hub-provisioning/` provides a turnkey Terraform module that customers can `source` from GitHub to provision the hub cluster and bootstrap the platform. After bootstrap, the platform is self-managing.
+> - `examples/` is renamed to `patterns/` to better reflect that these are consumption patterns, not just examples. The `workshop/` pattern is a first-class citizen alongside other patterns. Workshop-specific configuration (CloudFront, GitLab, Identity Center) lives in `patterns/workshop/`; heavy workshop orchestration (Terraform, deploy scripts) lives in `platform-engineering-on-eks`.


all workshop specific patterns should be there, so user can see full picture on how we did it, don't hide things in internal repo

allamand · 2026-03-13T10:31:05Z

+eksctl create cluster --name hub --region us-west-2
+
+# 2. Install ArgoCD
+helm install argocd argo/argo-cd -n argocd --create-namespace


activte EKS capabilities

allamand · 2026-03-13T10:33:41Z

+
+### 3.4 Phase 4: Workshop Isolation
+
+**Goal**: Move all workshop-specific code (including ALL Terraform) to the internal `platform-engineering-on-eks` GitLab repo. The `appmod-blueprints` repo becomes a clean, customer-facing, GitOps-native solution with zero Terraform and zero workshop concerns.


again, just in pattern/workshop/ folder, not internal gitlab

allamand · 2026-03-13T10:35:52Z

+
+#### 3.5.2 Target State
+
+- Hub cluster creation is done once by any tool (eksctl, CDK, TF, CLI) — this is outside `appmod-blueprints`


but we provide few options there who people that don't have existing clusters

allamand · 2026-03-13T10:38:00Z

+  - CloudFront via ACK CloudFront controller (optional)
+  - Observability via ACK Grafana/Prometheus (optional)
+  - Pod Identity via native K8s resources
+- Spoke clusters are provisioned exclusively via Kro RGDs or CrossPlane compositions from the hub


I would say, this does not matter, users will have choice to provision them using any tools, we privide a way to do it with kro/ACK, that could also be crossplane, pullumi, terraform, eksctl, console... that does not matters, we just need to show them how to create/register the cluster secret, and which IAM Rolke to add in the EKS Access entries, then the platform will register it automatically with Argo and bootstrap it as a fleet member

allamand · 2026-03-13T10:39:11Z

+  - Pod Identity via native K8s resources
+- Spoke clusters are provisioned exclusively via Kro RGDs or CrossPlane compositions from the hub
+- ArgoCD ApplicationSets auto-discover and bootstrap new spokes
+- Backstage templates allow self-service spoke creation


I would like to add also here an Agentic way to add clusters using our solution that uses agentic.

so users can either :

use basckstage

use agent

use native gitops integration

to create new spoke clusters, or any apps

elamaran11 · 2026-03-13T15:00:07Z

@allamand Thanks for the feedback. We will be implementing some of these but rest is not part of the tenets we decided upon for this approach. Im happy to have you as part of this refactor effort. Following feedback is incorporated:

EKS Auto Mode — replaced "node groups" with "EKS Auto Mode (no managed node groups)" throughout; added to design principles, executive summary, hub-config example, cluster stack description, hub-provisioning module, bootstrap guide, and test steps.

Pod Identity, not IRSA — added as design principle #10, added pod_identity: true to hub-config example with explicit "not IRSA" comments. IRSA was not previously referenced in this doc, so the additions make the Pod Identity preference explicit.

Backstage GitOps-only — updated all Backstage template references to emphasize PR/push through GitOps, not direct kubectl apply. Updated Task 3.6, Task 5.3, the Asana tables, and the spoke creation flow.

Cluster generator for ApplicationSets — added "cluster generator pattern" to fleet descriptions in both repo structures, the ApplicationSets change table, and the spoke auto-discovery section.

No examples/ folder — fixed the incorrect "renamed from examples/" references to clarify patterns/ is a new directory.

Activate EKS capabilities — updated the customer bootstrap flow to include aws eks update-cluster-config for enabling Auto Mode capabilities (compute, networking, storage, load balancing) after cluster creation.

…tOps-only, cluster generator, activate EKS capabilities, fix examples/ reference

Addresses AL2023 base image CVEs flagged by AppSec: - ALAS2023-2026-1721 (libcap, High, 13d SLA): 2.73-1.amzn2023.0.6 -> 0.7 - ALAS2023-2026-1680 (krb5-libs, Medium): 1.21.3-6 -> 1.21.3-7 - ALAS2023-2026-1699 (curl/libcurl, CVE-2026-3805): 8.17.0-1.amzn2023.0.2 -> 0.3 3.4.1 is the latest pinned 3.x release (AL2023 / fluent-bit v5.0.5). Image tag verified by inspecting the RPM database in the published 3.4.1 image; libssh advisory ALAS2023-2026-1759 (1-month SLA) is still pending an upstream rebuild and tracked separately. Affects spokes (dev, prod) where aws_for_fluentbit is enabled. Hub overlay keeps the addon disabled; the registry change is a no-op for it until enabled. Refs: https://github.com/aws/aws-for-fluent-bit/releases

Addresses Debian 13 base image CVEs flagged by AppSec on public.ecr.aws/docker/library/haproxy:3.4-dev12: - CVE-2026-34182 (openssl, Critical, 26d SLA) - CVE-2026-34180 (openssl, High, 26d SLA) - CVE-2026-34181 (openssl, High, 26d SLA) HAProxy 3.4.0 went GA on 2026-06-11. Switching from a transient 3.4-dev tag to the stable 3.4.0 release. Verified the image ships openssl 3.5.6-1~deb13u2, which is the version the advisory requires. Image pulled from the ECR Public mirror to avoid Docker Hub rate limits. Both files updated: - gitops/addons/configs/argo-cd/values.yaml — runtime values consumed by ArgoCD on every sync - platform/infra/terraform/common/manifests/argocd-initial-values.yaml — bootstrap values used by Terraform on initial install Refs: https://hub.docker.com/_/haproxy/tags

These files are leftover from a prior self-managed ArgoCD path and are not consumed by the current setup, which uses EKS Capabilities for managed ArgoCD: - platform/infra/terraform/common/argocd.tf sets install = false on the gitops-bridge module — Terraform never installs ArgoCD - hub-config.yaml: enable_argocd: false # EKS Managed capability - 0-init.sh comment: 'For EKS capabilities, ArgoCD runs as managed service' No Terraform resource references argocd-initial-values.yaml. No ApplicationSet references gitops/addons/configs/argo-cd/values.yaml. Removing them prevents confusion about where ArgoCD configuration takes effect — for EKS Capabilities clusters, AWS owns the bundle including the redis-ha haproxy sub-component. This also reverts commit 23ec47c, which attempted to pin the haproxy image to 3.4.0 in these dead files. That CVE remediation needs to go through AWS Support for the managed ArgoCD capability. Files removed: - gitops/addons/configs/argo-cd/values.yaml - platform/infra/terraform/common/manifests/argocd-initial-values.yaml

The KubeVela chart ships built-in ComponentDefinitions (e.g. appmod-service) whose CUE templates render to argoproj.io/v1alpha1 Rollout resources. The KubeVela admission webhook does pre-render validation when registering ComponentDefinitions and rejects them with 'no matches for kind "Rollout"' when the Rollout CRD is not present. With both kubevela and argo-rollouts at wave 3, ArgoCD applies them in parallel, creating a race: registration of the bundled ComponentDefinitions can fail until argo-rollouts converges and the Rollout CRD is reachable. Subsequent reconciles eventually succeed, but the customer-visible install transitions through OutOfSync / Degraded states with admission-webhook errors. The taxonomy rule, now documented in registry/README.md, is that a chart's wave reflects the highest wave it consumes — not the resources it emits. KubeVela consumes the Rollout workload kind from wave 3, so it belongs at wave 4. New ComponentDefinitions or Traits added to the chart do not change its wave; only changes to the set of consumed addons do. Charts that ship their own ComponentDefinitions referencing wave-3 resource kinds (e.g. oam-agent-components in sample-open-agentic-platform) sit at wave 5.

…r-rollouts fix(registry): move kubevela to wave 4 after argo-rollouts

Ingress templates now auto-detect CloudFront vs custom domain: - If ingress_domain_name contains 'cloudfront.net': ALB listens on HTTP:80 only (CloudFront terminates TLS) - Otherwise: ALB listens on HTTPS:443 with ssl-redirect (ACM cert required for custom domains) This removes the need for ACM certificates when using CloudFront default domains and eliminates the LBC 'no certificate found' error.

…anges feat(ingress): Conditional ALB listen-ports based on CloudFront domain

Brings in: - a59bfcf feat(ingress): Conditional ALB listen-ports based on CloudFront domain Auto-detects CloudFront vs custom domain on argo-workflows and keycloak ingresses; eliminates the LBC 'no certificate found' error in CloudFront mode. - Plus the merge commits for PR #708 (already in shapirov's ancestry) and PR #752. Conflict-free merge: zero file overlap between the two sides since common ancestor c754631. Validated with git merge-tree dry-run before merging.

feat(platform): multi-cluster auth design + sync-wave taxonomy + CVE bumps + cleanup

The previous D1 wording offered "wildcard ACM cert (*.peeks.dev.<base-domain>)" as an option without noting that DNS-1034 wildcards match exactly one label, so the wildcard alone covers the spoke names but not the hub's bare parent name (peeks.dev.<base-domain>). Replaces with three explicit options (single SAN cert recommended, two-cert variant, per-env variant) and adds a CloudFront-mode disclaimer noting no ACM cert is provisioned in that mode.

…od Identity Adds an Operational Invariants section to .kiro/steering/project.md documenting facts that have been confirmed multiple times and should be treated as ground truth by the agent (i.e., do not re-investigate): - ArgoCD on the hub is the EKS managed Capability — control-plane components run inside the AWS-managed control plane and are not visible via 'kubectl get pods -n argocd'. Health is verified by Application reconcile status, not pod presence. Self-managed ArgoCD values files (e.g., gitops/addons/configs/argo-cd/values.yaml, platform/infra/terraform/common/manifests/argocd-initial-values.yaml) are dead code and have been removed. - Cluster secrets in the hub's argocd namespace use EKS ARNs as the cluster server value; spokes do not run ArgoCD. - Platform uses EKS Pod Identity (not IRSA) for pod-level AWS credentials on EKS Auto Mode.

D1 update: The platform treats config.local.yaml's 'domain' as an opaque customer input — it does not care whether the customer obtained the hostname from Route 53, third-party DNS, a CDN distribution, or anywhere else. Removes the previous CloudFront-mode-specific disclaimer that bled implementation detail into the design contract. Adds a cross-reference to the new HUB_NETWORKING.md for the orthogonal VPC-ownership question. D7 (new): Insecure origin mode — for exploration deployments only. When TLS terminates upstream of the cluster (CDN, WAF, corporate proxy) and the link to the cluster is plain HTTP, an opt-in 'oidc_insecure_origin: "true"' cluster-secret label flips JWT- validating workloads to skip TLS verification on JWKS fetch. Default false. Includes explicit threat-model statement, exploration-only warning, mechanism, and alternatives.

Phase-1 design for letting customers run the hub on their own VPC instead of the platform-created one. Selected by presence of 'hub.vpcId' (and 'hub.subnets.{private,public}') in config.local.yaml. Sections: - Modes: platform-managed (default) vs. customer-supplied - config.local.yaml extension shape - Hard requirements (2+ AZs private + public, NAT gateway, ALB- discovery tags) and soft ones (CIDR sizing, no overlap, custom route-table content) - Pre-install verification checklist (4 aws ec2 describe-* commands) - Implementation plan: composition gating, validation task, docs - Explicit out-of-scope list (spokes, customer Route 53, private- only platform, hub-spoke connectivity) Decoupled from MULTI_CLUSTER_AUTH.md so authentication and network topology stay independent concerns. Cross-referenced both ways.

The platform's existing convention is the 'environment' label on fleet-member values.yaml (consumed by the fleet-secrets ApplicationSet to look up overlays/environments/<environment>/enabled-addons.yaml). The original auth design used '<env>' as a parallel placeholder, which would have created either a redundant second label or a divergence between auth and the rest of the platform. Renames: - Secrets Manager path: 'peeks/<env>/oidc/<client-name>' -> 'peeks/<environment>/oidc/<client-name>' - Client declaration field name: 'env: dev' -> 'environment: dev' - Provisioner uniqueness key: '(name, env)' -> '(name, environment)' - Helm template variable in ExternalSecret example: '{{ .Values.env }}' -> '{{ .Values.environment }}' - Hostname placeholder: '<env>.peeks.dev.<base-domain>' -> '<environment>.peeks.dev.<base-domain>' - Example client names in OAP consumer reference: 'agentgateway-<env>' / 'agent-runtime-<env>' -> 'agentgateway-<environment>' / 'agent-runtime-<environment>' Phase 1 implementation step 2 rewritten: the existing 'environment: dev/prod' labels on spoke fleet-member values.yaml are sufficient — no fleet-member edits needed for Phase 1. Pure naming change. No code or schema breaking change since the implementation has not yet been written.

…to spoke cluster secrets Adds two new annotations to every spoke cluster secret emitted by the fleet-secret chart (direct mode): oidc_issuer_url — HTTPS issuer base URL for the cluster's IdP realm oidc_insecure_origin — "true" only in dev/lab when TLS terminates at the LB and the in-cluster hop is plain HTTP (see D7) Values are plumbed from the hub cluster secret annotations through the fleet-secrets ApplicationSet valuesObject using the index/or pattern: '{{ or (index .metadata.annotations "oidc_issuer_url") "" }}' The index/or pattern is required because both ApplicationSets set goTemplateOptions: ["missingkey=error"]; dot-notation access would crash before the hub cluster secret carries these annotations (Batch B work). Defaults are safe: empty issuer URL means OIDC not yet configured for this cluster; consumer charts treat that as "no auth" and skip setup. The hub cluster secret is managed by the externalSecret mode path (Terraform/Crossplane seeds its annotations directly — no chart changes needed there). Platform convention maintained: free-form config values go in annotations, enable_* toggles go in labels.

Step-by-step how-to for adding OIDC-secured workloads to the platform: 1. Declare OIDC clients (provisioner values fragment) 2. Pull contract via ExternalSecret from peeks/<environment>/oidc/<client> 3. Configure JWT validation (normal + insecure-origin dev mode) 4. Construct ingress hostname from ingress_domain_name annotation 5. Read oidc_* cluster secret annotations directly in chart templates Per-IdP notes: Keycloak, Okta, Azure AD/Entra, Auth0.

…derivation keycloak-client-provisioner chart (platform-charts/): - PostSync Job loops clients[] from values, creates Keycloak realm+clients idempotently, writes contract JSON to Secrets Manager at <secretPathPrefix>/<environment>/oidc/<client.name> - Uses aws-cli image + Pod Identity (no IRSA) for Secrets Manager writes - realmName defaults to environment value (e.g. 'dev', 'prod'); overridable via realmPrefix or explicit realmName - clients: [] default means the Job is a no-op until consumers declare clients client-provisioner ApplicationSet (gitops/bootstrap/): - Matrix: hub cluster x spoke fleet-member values files - Selector: matchExpressions client_provisioner key Exists - Routes to platform-charts/<type>-client-provisioner — customers replace the provisioner by changing client_provisioner label to their IdP type (e.g. 'okta') and providing a matching chart fleet-secret chart: derive oidc_issuer_url in template (no seeding needed): - Removes hub.oidcIssuerUrl (was never seedable without a Terraform/Taskfile step per environment) - Adds oidc.issuerUrlTemplate — a Go tpl string evaluated with .domain and .environment; default: 'https://{{ .domain }}/keycloak/realms/{{ .environment }}' - Customers override per cluster via overlays/clusters/<name>/addon-overrides.yaml - oidc_insecure_origin still configurable via oidc.insecureOrigin (D7) - fleet-secrets ApplicationSet updated: drops oidcIssuerUrl, keeps oidc.insecureOrigin passthrough from hub annotation Fleet members: - tenant: workshop -> tenant: default (workshop is a lab artifact) - client_provisioner: keycloak label added to spoke-dev and spoke-prod

Grants the keycloak-client-provisioner ServiceAccount (keycloak namespace, hub cluster) permission to write OIDC contract entries to Secrets Manager. IAM policy scope: secretsmanager:CreateSecret + PutSecretValue + DescribeSecret on resources matching arn:...:secret:*/oidc/* — covers all paths written by the provisioner Job (<secretPathPrefix>/<environment>/oidc/<client>). Uses the existing crossplane-pod-identity chart pattern (Orphan deletion policy on IAM Role/Policy, same Crossplane upbound providers). Enabled via control-plane pod-identities overlay, not spoke overlays — the Job always runs on the hub where Keycloak lives.

… hub Problem 1 — crossplane-base spoke deployments failing with 'namespaces argocd not found' (attempt #705): - provider-kubernetes.yaml unconditionally created Role/RoleBinding in namespace argocd whenever kubernetes.version was set - Spokes have no argocd namespace (ArgoCD is a managed hub Capability) Fix: gate behind providers.kubernetes.createArgoCDRBAC flag (default false) enabled only in control-plane/crossplane-base/values.yaml override Problem 2 — Crossplane IAM/EKS providers on spoke clusters have no AWS credentials (attempt #4122 on agentcore, PodIdentityAssociation failures): - crossplane-base deploys Crossplane provider CRs to spoke clusters - Those CRs (iam.Role, eks.PodIdentityAssociation) need the IAM provider to reconcile them — but the IAM provider has no Pod Identity on spokes - Chicken-and-egg: can't create credentials without credentials Fix: add crossplane-ec2, crossplane-amp, crossplane-grafana identities to dev and prod pod-identities overlays (deployed by hub's Crossplane, which has working IAM provider credentials)

…gress insecure flag 1. Bump aws-for-fluent-bit from 3.4.8 to 3.4.9 to resolve high-severity CVEs (ALAS2023-2026-1986, ALAS2023-2026-1942, CVE-2026-11850). 2. Fix keycloak and argo-workflows ingress templates: the insecure flag was evaluated as a bare Go template conditional, but the ApplicationSet passes the string "false" which is truthy. Changed to explicit eq comparison so insecure=false correctly renders HTTPS listen-ports with the host field set, placing the ingress rules on the 443 listener alongside Langfuse and AgentGateway.

fix(platform): bump fluent-bit 3.4.9 + fix keycloak/argo-workflows in…

…+ AccessEntry Every eks.aws.upbound.io managed resource requires spec.forProvider.region; the PodIdentityAssociation and AccessEntry templates omitted it, so both failed admission at sync: spec.forProvider.region: Required value .Values.aws.region is already injected (the controller policy uses it throughout) — wire it into both forProvider blocks. The iam.aws.upbound.io MRs (Role/Policy/RolePolicyAttachment) are global and unaffected.

LiteLLM is no longer used — Bifrost is the sole LLM gateway. Updated dashboard title, tags, panels, and PromQL queries to reference only the bifrost namespace. Removed duplicate LiteLLM CPU/Memory panels.

fix(karpenter): set forProvider.region on EKS PodIdentityAssociation …

fix(dashboards): rename LiteLLM Gateway dashboard to Bifrost LLM Gateway

The node AccessEntry used principalArnRef, which is NOT a field on the eks.aws.upbound.io AccessEntry CRD (its cross-resource ref field is principalArnFromRoleRef). The API server prunes the unknown field on apply, so the live MR has no ref and never resolves principalArn: InvalidParameterException: The principalArn parameter format is not valid [] (reading EKS Access Entry (spoke-dev:<no value>)) Rename to principalArnFromRoleRef so the ref resolves to the node Role's ARN. Note this differs from PodIdentityAssociation, whose ref field IS roleArnRef (which is why the controller assoc synced fine).

…f-field fix(karpenter): AccessEntry ref field is principalArnFromRoleRef

refactor(nodepools): decouple workshop-tuned pools; default to EKS Auto Mode built-ins

New docs/EKS-Capabilities-KRO-ACK-Setup.md (mirrors the ArgoCD capability doc): how to enable the Managed KRO + Managed ACK EKS Capabilities via the existing capabilities.kro/ack.enabled config toggle (Capability MRs already in the platform-cluster Composition, gated by function-cel-filter). Documents what Managed ACK bundles (GA controllers only) and the explicit rule that pre-GA controllers like lambdamicrovms must be self-managed downstream. Cross-links the downstream consumer, Flow D (Lambda MicroVM Agent Sandbox) in sample-open-agentic-platform.

Add a commented capabilities: example to the hub cluster block in config.yaml so users can discover the Managed KRO + Managed ACK toggle (the Capability MRs already exist in the platform-cluster Composition, default disabled). Points to the new setup doc and the Flow D consumer.

The manageAddons-gated vpc-cni/kube-proxy Addon MRs were missing spec.forProvider.region, which the eks.aws.upbound.io Addon CRD lists in forProvider.required — so enabling node.manageAddons failed admission with "spec.forProvider.region: Required value" and neither addon was ever created. Also set enableNetworkPolicy explicitly in the vpc-cni configurationValues. These MRs adopt in place with resolveConflictsOnUpdate=OVERWRITE, which takes field ownership: adopting a cluster that already had NetworkPolicy enforcement enabled would otherwise silently turn it off, rendering every NetworkPolicy and ClusterNetworkPolicy inert. Defaults to "true" (fail-secure) and is overridable via node.addons.vpcCni.enableNetworkPolicy. configurationValues is now built with dict|toJson instead of hand-written JSON so quoting/escaping is always well-formed. The value is read with hasKey+get rather than `| default`, because Helm's default treats bool false as empty and would flip an explicit false back to true; an overlay may now write either false or "false". It serializes as the string "true"/"false", matching the format EKS accepts in practice. node.manageAddons remains false by default — unchanged.

The appset-chart already supports an external overlay repo — bootstrap/addons.yaml reads overlay_repo_url / _revision / _basepath into overlayRepoURLGit*, which appends a $overlay source plus three per-addon valueFiles that layer AFTER the in-repo ones. But nothing ever emitted those annotations, so the feature was unreachable: overlayRepoURLGit has no fallback, so the appset-chart's `if $overlayRepoURLGit` guard always skipped the per-addon overlay layer. Add an optional `overlay: {repoURL, revision, basepath}` block that renders them. This lets a CONSUMER repo override Helm values for addons whose chart + registry entry live in this repo (where $defaults resolves here, so layers 3-4 are read from this repo) without opening a PR against it. Previously such a file placed in the consumer repo was skipped silently — every overlay path carries ignoreMissingValueFiles: true — so the setting just never took effect. Rendered in BOTH templates: cluster-secret.yaml (mode=direct, spokes) and external-secret.yaml (mode=externalSecret, hub). Doing only one would have looked correct while silently no-opping for the other cluster type. Default OFF and gated on `if .repoURL`, so behaviour is unchanged for every existing cluster: $overlay becomes a REAL ArgoCD source, and an unreachable or unauthenticated repo would break every overlay-aware app on that cluster. README documents the layering model, a step-by-step walkthrough with verification commands, and the caveats — including that this also repoints the appset-level overrides.yaml reads (which take the registry-entry schema, not chart values).

The tolerations entry in configurationValues was unnecessary: the aws-node and kube-proxy DaemonSets both already ship `{operator: Exists}`, which tolerates every taint, so neither needs help reaching a tainted self-managed node. What actually governs aws-node placement is nodeAffinity `eks.amazonaws.com/compute-type NotIn [fargate,hybrid,auto]`. Auto Mode nodes carry compute-type=auto and are excluded; a self-managed node leaves it unset and is included. Verified live: aws-node runs on the self-managed kata node and on none of the seven Auto Mode nodes. That also retracts the concern noted on this PR that enabling manageAddons would schedule aws-node onto existing Auto Mode nodes — affinity excludes them regardless of tolerations. configurationValues now renders {"enableNetworkPolicy":"true"}, matching the config a live cluster already runs with enforcement working.

Make PR A actually ENABLE the capabilities (not just document a toggle): - config.yaml: hub.capabilities.kro.enabled + ack.enabled = true (was commented). - config.schema.json: add the hub.capabilities schema (kro/ack.enabled booleans, default false) so the config validates. - docs/EKS-Capabilities-KRO-ACK-Setup.md: scope to pure capability enablement — removed the Flow D / lambdamicrovms / pre-GA-vs-GA specifics and the ACK GA-list link (those move to the consumer, PR B in sample-open-agentic-platform). Keeps a generic 'non-GA controllers are a consumer's self-managed concern' scope note. KRO serves ResourceGraphDefinitions; ACK manages GA AWS resources as K8s CRs. Spokes stay opt-in (XRD default false).

…ities-flow-d feat: enable Managed KRO + ACK capabilities on the hub

Kargo 1.11.0: - Native api.basePath support — Kargo UI/API served at /kargo - TLS terminated upstream by ALB - Ingress with platform IngressClass + URL rewrite to strip prefix - No more catch-all '/' conflict with Langfuse Fluent Bit 3.4.9 → 3.4.11: - Picks up latest AL2023 base image patches for CVEs (libacl, glib2, krb5-libs)

Mirror claims/argocd-capability-role.yaml for the KRO + ACK capabilities so a hub bootstrap creates them declaratively (Crossplane IAM Role MRs, adopted by the platform-cluster Composition's Capability MRs via <cluster>-{KRO,ACK}CapabilityRole external-names): - claims/kro-capability-role.yaml: capabilities.eks trust only (KRO orchestrates in-cluster resources; no AWS perms). - claims/ack-capability-role.yaml: trust + a LEAST-PRIVILEGE inline RolePolicy for Flow D's GA resources — iam (roles matching *-microvm-build/-exec) + s3 (buckets matching *-microvm-artifacts) only. - Taskfile: apply both roles after the argocd role in hub:seed, and delete them in teardown (mirrors the existing argocd wiring). Closes the drift where these roles otherwise only exist as manual create-role state.

feat(platform): upgrade Kargo to 1.11.0 + bump Fluent Bit to 3.4.11

The multi-line pipe literal with escaped backslashes in the ALB transforms annotation was being mangled through the ApplicationSet Go template → Helm values pipeline, causing a YAML parse error in the Kargo chart template rendering. Switch to a single-line quoted string with unescaped regex.

…roles feat: codify KRO + ACK capability IAM roles for the hub

fix(kargo): use single-line annotation to avoid YAML parse error

fix(karpenter): region + explicit enableNetworkPolicy on EKS Addon MRs

feat(fleet-secret): optional external overlay repo annotations (layer 5)

Root cause: argocd.urls changed from a list to a map in Kargo 1.11.0. The chart renders urls as key=value pairs in the configmap and a list caused Go fmt to emit invalid YAML. Verified with helm template --version 1.11.0 (199 resources, clean).

fix(kargo): fix argocd.urls format (map not list) and restore URL rew…

…argo 1. Keycloak kargo client: update redirectUris from /login to /kargo/login (Kargo 1.11.0 with basePath sends redirect_uri with the prefix) 2. OIDC admins claim: change /admin to admin (without leading slash) The Keycloak groups mapper has full.path=false, so group names in the token are 'admin' not '/admin'.

fix(kargo): update redirect URI and admin group claim for basePath /k…

Upgrade from 3.4.11 to 3.4.13 to resolve: - CVE-2026-16118 (glib2, High severity) - CVE-2026-11979 (libxml2, Medium severity) New base image: AL2023 2023.12.20260803.3

fix(observability): Bump aws-for-fluent-bit to 3.4.13 for CVE-2026-16118

allamand commented Mar 13, 2026

View reviewed changes

elamaran11 marked this pull request as draft March 13, 2026 15:00

elamaran11 added a commit that referenced this pull request Mar 13, 2026

Integrate PR #529 feedback: EKS Auto Mode, Pod Identity, Backstage Gi…

0b496b6

…tOps-only, cluster generator, activate EKS capabilities, fix examples/ reference

punkwalker pushed a commit that referenced this pull request Apr 29, 2026

Integrate PR #529 feedback: EKS Auto Mode, Pod Identity, Backstage Gi…

ae25c86

…tOps-only, cluster generator, activate EKS capabilities, fix examples/ reference

punkwalker force-pushed the feature/agent-platform branch from cd912af to 35fe079 Compare April 29, 2026 16:02

hmuthusamy force-pushed the feature/agent-platform branch 2 times, most recently from 49c597f to 8394166 Compare May 11, 2026 23:19

hmuthusamy force-pushed the feature/agent-platform branch from 3827097 to 6046408 Compare May 19, 2026 16:26

user1 and others added 20 commits May 27, 2026 18:12

removed agent platform unrelated aspects from the platform design doc

b5bacea

Merge pull request #751 from aws-samples/fix/sync-waves-kubevela-afte…

0b015d6

…r-rollouts fix(registry): move kubevela to wave 4 after argo-rollouts

Merge pull request #752 from aws-samples/feature/cloudfront-domain-ch…

df86ef0

…anges feat(ingress): Conditional ALB listen-ports based on CloudFront domain

Merge pull request #710 from aws-samples/feature/agent-platform-shapirov

3b9f271

feat(platform): multi-cluster auth design + sync-wave taxonomy + CVE bumps + cleanup

hmuthusamy and others added 30 commits July 24, 2026 14:01

Merge pull request #788 from aws-samples/fix/fluent-bit-3.4.9-cve

9ea7347

fix(platform): bump fluent-bit 3.4.9 + fix keycloak/argo-workflows in…

fix(dashboards): rename LiteLLM Gateway dashboard to Bifrost LLM Gateway

2333b45

LiteLLM is no longer used — Bifrost is the sole LLM gateway. Updated dashboard title, tags, panels, and PromQL queries to reference only the bifrost namespace. Removed duplicate LiteLLM CPU/Memory panels.

Merge pull request #790 from aws-samples/fix/karpenter-eks-mr-region

a644706

fix(karpenter): set forProvider.region on EKS PodIdentityAssociation …

Merge pull request #789 from aws-samples/fix/rename-litellm-dashboard

f2656f2

fix(dashboards): rename LiteLLM Gateway dashboard to Bifrost LLM Gateway

Merge pull request #791 from aws-samples/fix/karpenter-accessentry-re…

7792e97

…f-field fix(karpenter): AccessEntry ref field is principalArnFromRoleRef

Merge pull request #795 from aws-samples/feature/agent-platform-shapirov

e3a6f37

refactor(nodepools): decouple workshop-tuned pools; default to EKS Auto Mode built-ins

Merge pull request #796 from aws-samples/docs/managed-kro-ack-capabil…

f75128c

…ities-flow-d feat: enable Managed KRO + ACK capabilities on the hub

Merge pull request #799 from aws-samples/fix/kargo-1.11-basepath

30f0e73

feat(platform): upgrade Kargo to 1.11.0 + bump Fluent Bit to 3.4.11

Merge pull request #800 from aws-samples/feat/hub-kro-ack-capability-…

01addbc

…roles feat: codify KRO + ACK capability IAM roles for the hub

Merge pull request #801 from aws-samples/fix/kargo-ingress-annotation

f5178ca

fix(kargo): use single-line annotation to avoid YAML parse error

Merge pull request #797 from aws-samples/fix/karpenter-eks-addon-mrs

1b54747

fix(karpenter): region + explicit enableNetworkPolicy on EKS Addon MRs

Merge pull request #798 from aws-samples/feat/overlay-repo-annotations

96a695a

feat(fleet-secret): optional external overlay repo annotations (layer 5)

Merge pull request #802 from aws-samples/fix/kargo-ingress-annotation

6fcddb4

fix(kargo): fix argocd.urls format (map not list) and restore URL rew…

Merge pull request #803 from aws-samples/fix/kargo-redirect-uri

263065d

fix(kargo): update redirect URI and admin group claim for basePath /k…

fix(observability): Bump aws-for-fluent-bit to 3.4.13 for CVE-2026-16118

0ff34cc

Upgrade from 3.4.11 to 3.4.13 to resolve: - CVE-2026-16118 (glib2, High severity) - CVE-2026-11979 (libxml2, Medium severity) New base image: AL2023 2023.12.20260803.3

Merge pull request #809 from aws-samples/fix/fluent-bit-cve-2026-16118

d78d656

fix(observability): Bump aws-for-fluent-bit to 3.4.13 for CVE-2026-16118


		### Overview

		Agent Core Components provision AWS Bedrock Agent Core capabilities (Memory, Browser, Code Interpreter) using Tofu Controller.


		> Note: There is no Terraform in the `appmod-blueprints` solution repo. Initial EKS cluster creation (via Terraform, CDK, eksctl, etc.) lives in the customer's own infra repo or the workshop repo (`platform-engineering-on-eks`). Once the hub cluster exists, it self-manages via Kro+ACK/CrossPlane compositions and ArgoCD.

		### Changes in `sample-agent-platform-on-eks` Repository


		### 3.4 Phase 4: Workshop Isolation

		Goal: Move all workshop-specific code (including ALL Terraform) to the internal `platform-engineering-on-eks` GitLab repo. The `appmod-blueprints` repo becomes a clean, customer-facing, GitOps-native solution with zero Terraform and zero workshop concerns.


		#### 3.5.2 Target State

		- Hub cluster creation is done once by any tool (eksctl, CDK, TF, CLI) — this is outside `appmod-blueprints`

Uh oh!

Conversation

allamand commented Mar 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elamaran11 commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

elamaran11 commented Mar 13, 2026 •

edited

Loading