bug: BlueGreen rollout stuck with stale currentPodHash/stableRS on image-only update (v1.8.3) #4976

We're hitting what looks like the exact same bug, but on v1.8.3 — the version this issue's report treats as the "safe" pre-upgrade baseline. Wanted to flag that this may not be v1.9.0-specific.

Environment

• Argo Rollouts Controller: v1.8.3 (quay.io/argoproj/argo-rollouts:v1.8.3)
• Strategy: BlueGreen, autoPromotionEnabled: true
• Kubernetes: GKE
• Deployed via ArgoCD (auto-sync), image tag bumped via CI (Jenkins) commit to git
• Not a high-scale environment — single-digit number of Rollouts managed by this controller

What happened

Updated the container image tag in the Rollout spec (154 → 155) via a normal GitOps flow (Jenkins commits new tag → ArgoCD auto-syncs). ArgoCD reported Synced/Healthy and the sync succeeded. However:

• The live ReplicaSet's pod template showed the new image (:155)
• The actual running Pod under that same ReplicaSet was still on the old image (:154), and had not been recreated — it had simply had its container OOMKilled and restarted in place
• status.currentPodHash and status.stableRS remained pinned to the old hash (b4c7b7ccb) even though the desired pod template's computed hash was different (95756dc7c)
• No new ReplicaSet was ever created for the new hash

Logs (repeats indefinitely on every reconcile, including after a full controller pod restart):

time="2026-08-13T10:48:47Z" level=info msg="Started syncing rollout" generation=51 namespace=backend resourceVersion=... rollout=booking-qc-service
time="2026-08-13T10:48:47Z" level=info msg="ComputePodTemplateHash hash changed (expected: 95756dc7c, actual: b4c7b7ccb)" namespace=backend rollout=booking-qc-service
time="2026-08-13T10:48:47Z" level=info msg="Reconciling stable ReplicaSet 'booking-qc-service-b4c7b7ccb'" namespace=backend rollout=booking-qc-service
time="2026-08-13T10:48:47Z" level=info msg="No status changes. Skipping patch" generation=51 namespace=backend resourceVersion=... rollout=booking-qc-service
time="2026-08-13T10:48:47Z" level=info msg="Reconciliation completed" generation=51 namespace=backend resourceVersion=... rollout=booking-qc-service time_ms=13.5

status block confirming the stale state:

status:
  blueGreen:
    activeSelector: b4c7b7ccb
    previewSelector: b4c7b7ccb
  currentPodHash: b4c7b7ccb
  observedGeneration: "51"
  phase: Healthy
  stableRS: b4c7b7ccb
  conditions:
  - type: Healthy
    status: "True"
    reason: RolloutHealthy
    message: Rollout is healthy
  - type: Progressing
    status: "True"
    reason: NewReplicaSetAvailable
    message: ReplicaSet "booking-qc-service-b4c7b7ccb" has successfully progressed.

So the controller detects the hash mismatch every single reconcile, logs it, and then immediately decides there's nothing to patch — leaving the Rollout permanently reporting Healthy/Progressing against the old ReplicaSet, while genuinely believing (per its own log line) that a different hash is expected.

What did NOT fix it

• Deleting/restarting the argo-rollouts controller pod (new pod hit the same stuck state within ~4 minutes, immediately on first reconcile of this Rollout)
• Waiting — this has now persisted across multiple image-tag-only deploys

What DID unblock it (workaround)

• Changing an additional field in the pod template alongside the image (e.g. bumping a container resources.requests.cpu value) reliably triggers a proper new ReplicaSet + rollout. An image-only diff does not.

This matches the workaround your report mentions ("triggering a new deployment with the exact same image tag also temporarily forces the sync loop to unblock") — in our case, any incidental extra template diff (not just a re-applied same tag) has the same unblocking effect.

Happy to provide the full Rollout manifest / additional logs if useful for reproduction. This is currently forcing us to bundle unrelated config changes with every image deploy as a workaround, which isn't sustainable for a production GitOps pipeline.