Skip to content

Blue/Green active service selector becomes empty during CrashLoopBackOff Progressing state, bypassing prePromotionAnalysis and autoPromotionEnabled gate (EKS + Fargate + Flux GitOps) #4750

Description

@gprabuheins

Environment

Argo Rollouts version: v1.9.0 (Helm chart 2.40.9)
Kubernetes: EKS + Fargate
GitOps: Flux v2
Strategy: Blue/Green (autoPromotionEnabled: false)
Load Balancer: AWS NLB (internal, nlb-target-type: ip)
Application: Spring Boot with /actuator/health probes

Summary
During UAT validation of blue/green deployments, we observed a failure pattern where the active service selector (rollouts-pod-template-hash) became unstable or empty during prolonged Progressing states triggered by bad image deployments (CrashLoopBackOff / readiness failures).
This resulted in a complete traffic blackout on the active service, requiring manual intervention to restore stable routing.

Observed Behavior
Trigger Condition
Deploying a bad image (CrashLoopBackOff) as a second or later failed revision, with a prior successful revision already stable and active.
Failure Sequence
Bad image deploys → new ReplicaSet spins up
→ Pods enter CrashLoopBackOff
→ Rollout enters Progressing state
→ Active service selector (rollouts-pod-template-hash) becomes empty
→ NLB has no valid endpoint target
→ Full traffic blackout on active service
→ progressDeadlineAbort does not fire (referenced in #3988)
→ Rollout remains stuck in Progressing indefinitely
→ Manual intervention required:

  1. Suspend Flux reconciliation
  2. Manually patch active service selector
  3. Abort rollout
  4. Restore Git state with last known good image
  5. Resume Flux
    Observed Events
    SwitchService: Switched selector for service 'my-service' from '' to ''
    RolloutCompleted: Initial deploy ← bypass of prePromotionAnalysis
    SwitchService: Switched selector for service 'my-service' from '' to ''
    Key Observation
    When the active service selector was empty (''), subsequent deploys were classified as "Initial deploy" by the controller, bypassing:

prePromotionAnalysis
autoPromotionEnabled: false gate
All configured safety gates

This caused the bad image to be promoted directly to stable,active without any analysis or manual approval.

Configuration
yamlstrategy:
blueGreen:
activeService: my-service
previewService: my-service-preview
autoPromotionEnabled: false
scaleDownDelaySeconds: 3600
abortScaleDownDelaySeconds: 30
prePromotionAnalysis:
templates:
- templateName: my-health-check
args:
- name: service-name
value: my-service-preview
spec:
progressDeadlineSeconds: 1200
progressDeadlineAbort: true

Questions

  1. Is this known/expected behavior?
    Is the active service selector becoming empty ('') during prolonged Progressing states a known condition? We believe this may be related to issue Rollout stuck in infinite "Progressing" even after progressDeadlineSeconds exceeds #3988 where progressDeadlineAbort does not fire when CrashLoopBackOff restarts reset the progress deadline timer.
  2. Recommended safeguards?
    Are there any recommended configurations or patterns for preserving stable traffic routing during startup-failure scenarios in blue/green deployments? Specifically around ensuring the active service selector is never left in an empty state during controller recovery.
  3. "Initial deploy" classification
    Is the "Initial deploy" classification triggered intentionally when the active service selector is empty? If so, is there a way to prevent this from bypassing prePromotionAnalysis and autoPromotionEnabled: false when a stable revision already exists?
  4. Upstream improvements
    Are there any planned improvements around:

Selector stability during prolonged Progressing states?
Recovery behavior when progressDeadlineAbort fails to fire?
Related to issues #3988 and #4453?

Additional Context
We applied SSA merge semantics (kustomize.toolkit.fluxcd.io/ssa: merge) to the active and preview services to prevent Flux from overwriting Argo-managed selector fields. This resolved the Flux-side selector conflict but did not address the Argo-internal selector clearing behavior.
We are currently evaluating whether Kubernetes-native rolling deployments with strict capacity-preservation settings (maxUnavailable: 0, minReadySeconds, progressDeadlineSeconds) may provide a simpler and more deterministic operational recovery model for these specific failure modes, while we await upstream resolution.
Any guidance from the maintainers or community on recommended patterns for this scenario would be greatly appreciated.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions