fix(controller): caps canary replicas for an aborted rollout to its Traffic Weight

[Ghost-B0t]

Fixes: #4973

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
• The title of the PR is (a) conventional with a list of types and scopes found here, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
• I've signed my commits with DCO
• My builds are green. Try syncing with master if they are not.
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• I have run all tests locally (including the flaky ones) and they pass on my workstation
• I have used LLM/AI/Agent tools for this PR but I am responsible for all code of this PR
• I understand what the code does and WHY/HOW it works in several scenarios
• I know if my code is just adding new functionality or changing old functionality for existing users
• My organization is added to USERS.md.

Summary

An aborted rollout that uses dynamicStableScale: true prematurely scales up the canary replicaset on any scaleup of rollout.Spec.Replicas(scaled due to HPA/Keda, or manually).

Root cause

GetDesiredCanaryWeight in utils/replicaset/canary.go infers the canary's size from the stable ReplicaSet's shortfall:

expectedCanaryReplicas := rolloutSpecReplica - stableRS.Status.AvailableReplicas
canaryReplicas := max(expectedCanaryReplicas, newRS.Status.AvailableReplicas)
// walk steps in reverse, return the first setWeight whose replica-equivalent is below canaryReplicas

This assumes the only reason stable is below spec.replicas is that the canary still holds those replicas. That assumption breaks whenever the shortfall has any other cause. The resulting non-zero weight is converted back to a replica count and the aborted ReplicaSet is scaled to it.

Worked example, spec.replicas 10 → 12, steps [5, 10, 20, 50, 100], stable at 10 available, canary at 0:

expectedCanaryReplicas = 12 - 10 = 2
canaryReplicas = max(2, 0) = 2
reverse scan: 100 -> 12 , 50 -> 6, 20 -> 3, 10 -> 2, 5 -> 1 < 2 -> weight 5
canary replicas = ceil(5% × 12) = 1
It reverts once stable reports 11 available: the gap becomes 1, no step qualifies, weight returns to 0, canary back to 0.
A large scale-up or a significant availability dip can therefore resurrect most of an aborted ReplicaSet.

Fix

utils/replicaset/canary.go: In CalculateReplicaCountsForTrafficRoutedCanary, when dynamicStableScale is enabled, add a ceiling cap of its assigned traffic weight and add checks to skip the cases where we need to scale the canary above its traffic boundry, such as full-promote, rollback & when abortScaleDownDelaySeconds is set to 0.

Compared to the progressive abort behaviour (#4035): Abort with
dynamicStableScale reverse scans the canary steps finding the canary replica count for the previous step and progressively making it to 0. But it also leads to re-evaluation of these steps for an aborted rollout(canary: 0) resulting in the scaleup of aborted canary. This PR instead caps the canary scale up to its traffic distribution and preserves the progressive scale down of canary pods during an abort.

Testing

• New unit test TestAbortedCanaryNotScaledUpOnStableAvailabilityGap: fails on master with
  the scaleup of canary replicas, passes with the fix. No-delay and zero-delay behavior pinned.
• go test ./rollout/... ./utils/replicaset/... passes unchanged.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
18.5% Duplication on New Code

See analysis details on SonarQube Cloud

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.17%. Comparing base (f2c5c2b) to head (87b95a8).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4974      +/-   ##
==========================================
- Coverage   85.17%   85.17%   -0.01%     
==========================================
  Files         166      166              
  Lines       19453    19457       +4     
==========================================
+ Hits        16570    16572       +2     
- Misses       2030     2031       +1     
- Partials      853      854       +1     

Flag	Coverage Δ
e2e	53.07% <100.00%> (+0.04%)	⬆️
unit-tests	81.65% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Published E2E Test Results

  4 files    4 suites   4h 8m 36s ⏱️
149 tests 134 ✅  7 💤  8 ❌
608 runs  568 ✅ 28 💤 12 ❌

For more details on these failures, see this check.

Results for commit 87b95a8.

[github-actions]

Published Unit Test Results

2 632 tests   2 632 ✅  3m 30s ⏱️
  131 suites      0 💤
    1 files        0 ❌

Results for commit 87b95a8.

[Ghost-B0t]

@zachaller We are facing this issue in our production system due to frequent HPA triggers. As we are using service mesh the impact is getting multiplied for the same due to very frequent endpoints update choking the service mesh control plane.
Since the endpoint update count is (addition new canary replica + flush old stable replica) behind the service, it will impact all the systems running with service mesh as they are very sensitive to the endpoint updates.