You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Summary
During UAT validation of blue/green deployments, we observed a failure pattern where the active service selector (rollouts-pod-template-hash) became unstable or empty during prolonged Progressing states triggered by bad image deployments (CrashLoopBackOff / readiness failures).
This resulted in a complete traffic blackout on the active service, requiring manual intervention to restore stable routing.
Observed Behavior
Trigger Condition
Deploying a bad image (CrashLoopBackOff) as a second or later failed revision, with a prior successful revision already stable and active.
Failure Sequence
Bad image deploys → new ReplicaSet spins up
→ Pods enter CrashLoopBackOff
→ Rollout enters Progressing state
→ Active service selector (rollouts-pod-template-hash) becomes empty
→ NLB has no valid endpoint target
→ Full traffic blackout on active service
→ progressDeadlineAbort does not fire (referenced in #3988)
→ Rollout remains stuck in Progressing indefinitely
→ Manual intervention required:
Suspend Flux reconciliation
Manually patch active service selector
Abort rollout
Restore Git state with last known good image
Resume Flux
Observed Events
SwitchService: Switched selector for service 'my-service' from '' to ''
RolloutCompleted: Initial deploy ← bypass of prePromotionAnalysis
SwitchService: Switched selector for service 'my-service' from '' to ''
Key Observation
When the active service selector was empty (''), subsequent deploys were classified as "Initial deploy" by the controller, bypassing:
prePromotionAnalysis
autoPromotionEnabled: false gate
All configured safety gates
This caused the bad image to be promoted directly to stable,active without any analysis or manual approval.
Is this known/expected behavior?
Is the active service selector becoming empty ('') during prolonged Progressing states a known condition? We believe this may be related to issue Rollout stuck in infinite "Progressing" even after progressDeadlineSeconds exceeds #3988 where progressDeadlineAbort does not fire when CrashLoopBackOff restarts reset the progress deadline timer.
Recommended safeguards?
Are there any recommended configurations or patterns for preserving stable traffic routing during startup-failure scenarios in blue/green deployments? Specifically around ensuring the active service selector is never left in an empty state during controller recovery.
"Initial deploy" classification
Is the "Initial deploy" classification triggered intentionally when the active service selector is empty? If so, is there a way to prevent this from bypassing prePromotionAnalysis and autoPromotionEnabled: false when a stable revision already exists?
Upstream improvements
Are there any planned improvements around:
Selector stability during prolonged Progressing states?
Recovery behavior when progressDeadlineAbort fails to fire?
Related to issues #3988 and #4453?
Additional Context
We applied SSA merge semantics (kustomize.toolkit.fluxcd.io/ssa: merge) to the active and preview services to prevent Flux from overwriting Argo-managed selector fields. This resolved the Flux-side selector conflict but did not address the Argo-internal selector clearing behavior.
We are currently evaluating whether Kubernetes-native rolling deployments with strict capacity-preservation settings (maxUnavailable: 0, minReadySeconds, progressDeadlineSeconds) may provide a simpler and more deterministic operational recovery model for these specific failure modes, while we await upstream resolution.
Any guidance from the maintainers or community on recommended patterns for this scenario would be greatly appreciated.
Environment
Argo Rollouts version: v1.9.0 (Helm chart 2.40.9)
Kubernetes: EKS + Fargate
GitOps: Flux v2
Strategy: Blue/Green (autoPromotionEnabled: false)
Load Balancer: AWS NLB (internal, nlb-target-type: ip)
Application: Spring Boot with /actuator/health probes
Summary
During UAT validation of blue/green deployments, we observed a failure pattern where the active service selector (rollouts-pod-template-hash) became unstable or empty during prolonged Progressing states triggered by bad image deployments (CrashLoopBackOff / readiness failures).
This resulted in a complete traffic blackout on the active service, requiring manual intervention to restore stable routing.
Observed Behavior
Trigger Condition
Deploying a bad image (CrashLoopBackOff) as a second or later failed revision, with a prior successful revision already stable and active.
Failure Sequence
Bad image deploys → new ReplicaSet spins up
→ Pods enter CrashLoopBackOff
→ Rollout enters Progressing state
→ Active service selector (rollouts-pod-template-hash) becomes empty
→ NLB has no valid endpoint target
→ Full traffic blackout on active service
→ progressDeadlineAbort does not fire (referenced in #3988)
→ Rollout remains stuck in Progressing indefinitely
→ Manual intervention required:
Observed Events
SwitchService: Switched selector for service 'my-service' from '' to ''
RolloutCompleted: Initial deploy ← bypass of prePromotionAnalysis
SwitchService: Switched selector for service 'my-service' from '' to ''
Key Observation
When the active service selector was empty (''), subsequent deploys were classified as "Initial deploy" by the controller, bypassing:
prePromotionAnalysis
autoPromotionEnabled: false gate
All configured safety gates
This caused the bad image to be promoted directly to stable,active without any analysis or manual approval.
Configuration
yamlstrategy:
blueGreen:
activeService: my-service
previewService: my-service-preview
autoPromotionEnabled: false
scaleDownDelaySeconds: 3600
abortScaleDownDelaySeconds: 30
prePromotionAnalysis:
templates:
- templateName: my-health-check
args:
- name: service-name
value: my-service-preview
spec:
progressDeadlineSeconds: 1200
progressDeadlineAbort: true
Questions
Is the active service selector becoming empty ('') during prolonged Progressing states a known condition? We believe this may be related to issue Rollout stuck in infinite "Progressing" even after progressDeadlineSeconds exceeds #3988 where progressDeadlineAbort does not fire when CrashLoopBackOff restarts reset the progress deadline timer.
Are there any recommended configurations or patterns for preserving stable traffic routing during startup-failure scenarios in blue/green deployments? Specifically around ensuring the active service selector is never left in an empty state during controller recovery.
Is the "Initial deploy" classification triggered intentionally when the active service selector is empty? If so, is there a way to prevent this from bypassing prePromotionAnalysis and autoPromotionEnabled: false when a stable revision already exists?
Are there any planned improvements around:
Selector stability during prolonged Progressing states?
Recovery behavior when progressDeadlineAbort fails to fire?
Related to issues #3988 and #4453?
Additional Context
We applied SSA merge semantics (kustomize.toolkit.fluxcd.io/ssa: merge) to the active and preview services to prevent Flux from overwriting Argo-managed selector fields. This resolved the Flux-side selector conflict but did not address the Argo-internal selector clearing behavior.
We are currently evaluating whether Kubernetes-native rolling deployments with strict capacity-preservation settings (maxUnavailable: 0, minReadySeconds, progressDeadlineSeconds) may provide a simpler and more deterministic operational recovery model for these specific failure modes, while we await upstream resolution.
Any guidance from the maintainers or community on recommended patterns for this scenario would be greatly appreciated.