Skip to content

Canary Rollout: Scaling up Pod-Count increases Canary Pod Count after aborted #4973

Description

@Ghost-B0t

Checklist:

  • I've included steps to reproduce the bug.
  • I've included the version of argo rollouts.

Describe the bug

When an Argo Rollout is in aborted state, the aborted replicaset gets scaled back up whenever the stable ReplicaSet's status.availableReplicas falls short of spec.replicas, even though the canary is serving 0% of traffic and the abort is complete.
Any increase to spec.replicas triggers it, because spec.replicas changes instantly while the stable ReplicaSet's availability lags by however long a new pod takes to pass its readiness probe. An HPA scale-up is the common trigger; a manual kubectl edit reproduces it identically. Pod evictions and node loss produce the same shortfall and therefore the same effect.

This occurs when:
Canary strategy is used.
dynamicStableScale: true
Rollout is in aborted state.
Rollout is scaled up (by HPA or manual increase in spec.Replicas).

Observations:

  1. The aborted canary ReplicaSet is scaled up and then back down, once per scale-up event.
  2. If the canary pod stays up long enough to become Available, it opens the availability gate in ensureSVCTargets, and the canary Service selector then flaps between the stable and canary pod hashes several times per second --> reconcileStableAndCanaryService points it at newRS and the abort branch of reconcileTrafficRouting immediately points it back at stableRS, both within the same reconcile.
  3. Traffic to canary remains at 0%(expected behaviour)

Root cause

GetDesiredCanaryWeight in utils/replicaset/canary.go infers the canary's size from the stable ReplicaSet's shortfall:

expectedCanaryReplicas := rolloutSpecReplica - stableRS.Status.AvailableReplicas
canaryReplicas := max(expectedCanaryReplicas, newRS.Status.AvailableReplicas)
// walk steps in reverse, return the first setWeight whose replica-equivalent is below canaryReplicas

This assumes the only reason stable is below spec.replicas is that the canary still holds those replicas. That assumption breaks whenever the shortfall has any other cause. The resulting non-zero weight is converted back to a replica count and the aborted ReplicaSet is scaled to it.

To Reproduce

  1. Create a rollout with 10 replicas enabling dynamicStableScale and canarySteps for multiple setWeights leading upto 100 e.g., 5 -> 10 -> 20 -> 50 -> 100
  2. Trigger a rollout and promote few steps before aborting it whole.
  3. After the pods from aborted replicaset are terminated, edit the rollout spec with 12 replicas.
    With this the canary replicaset will scaleup from 0 -> 1 -> 0
    Also you can find the service label flap logs in argo-rollout logs.

Expected behavior

After the rollout is aborted, i.e. 0% traffic to canary, the aborted canary replicaset should not scale up.

Screenshots

Image

Rollout aborted at 15:15 and updated rollout replica at 15:20 and again at 15:26 observing spikes for aborted canary replicaset at both times.

Version

argo-rollouts: v1.9.0
k8s: 1.33

Logs
mock-service-75db5f99dd is the canary RS which was aborted.

# Logs for a specific rollout:
kubectl logs -n argo-rollouts deployment/argo-rollouts | grep rollout=mock-service | grep mock-service-75db5f99dd
2026-08-11 15:20:22.816 INFO time="2026-08-11T09:50:22Z" level=info msg="Scaled up ReplicaSet mock-service-75db5f99dd (revision 9) from 0 to 6" event_reason=ScalingReplicaSet namespace=test rollout=mock-service

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions