Checklist:
Describe the bug
reconcilePingAndPongService only reconciles the ping/pong service currently acting as canary. The service acting as stable is not reconciled against the stable ReplicaSet.
If both ping/pong services are initially created without a rollouts-pod-template-hash selector, the stable service continues matching all pods belonging to the Rollout. This includes the pods from the new canary ReplicaSet, so traffic assigned to the stable service can be served by canary pods.
The problem is easiest to observe on the first rollout after ping/pong is enabled. On later rollouts, the service becoming stable usually already has a hash selector from when it previously acted as canary, which hides the bug.
Possibly related to #2726. That report also creates both ping/pong services without hash selectors, but additionally involves ALB stickiness and 503s.
To Reproduce
- Create ping and pong services with a common selector and no pod template hash:
- Configure a canary Rollout with
pingPong.
- Complete an initial deployment.
- Start a rollout for a new revision while
status.canary.stablePingPong is pong.
- Inspect both Service selectors.
Actual result:
pong (stable) selector = map[foo:bar]
ping (canary) selector = map[foo:bar rollouts-pod-template-hash:5cb4fd98cf]
Because the stable service has no hash selector, it selects pods from both the stable and canary ReplicaSets.
This can also be reproduced with only pingPong configured, without stableService or canaryService.
Expected behavior
Both ping/pong services should be reconciled to their respective ReplicaSets:
pong (stable) selector = map[foo:bar rollouts-pod-template-hash:7fc44855c7]
ping (canary) selector = map[foo:bar rollouts-pod-template-hash:5cb4fd98cf]
The stable service should select only pods from stableRS, and the canary service should select only pods from newRS.
Version
Current master at 4e6a2798688e22868340d9871a3c8d78371f1568.
Additional context
A fix with unit tests is available in #4994.
Checklist:
Describe the bug
reconcilePingAndPongServiceonly reconciles the ping/pong service currently acting as canary. The service acting as stable is not reconciled against the stable ReplicaSet.If both ping/pong services are initially created without a
rollouts-pod-template-hashselector, the stable service continues matching all pods belonging to the Rollout. This includes the pods from the new canary ReplicaSet, so traffic assigned to the stable service can be served by canary pods.The problem is easiest to observe on the first rollout after ping/pong is enabled. On later rollouts, the service becoming stable usually already has a hash selector from when it previously acted as canary, which hides the bug.
Possibly related to #2726. That report also creates both ping/pong services without hash selectors, but additionally involves ALB stickiness and 503s.
To Reproduce
pingPong.status.canary.stablePingPongispong.Actual result:
pong(stable) selector =map[foo:bar]ping(canary) selector =map[foo:bar rollouts-pod-template-hash:5cb4fd98cf]Because the stable service has no hash selector, it selects pods from both the stable and canary ReplicaSets.
This can also be reproduced with only
pingPongconfigured, withoutstableServiceorcanaryService.Expected behavior
Both ping/pong services should be reconciled to their respective ReplicaSets:
pong(stable) selector =map[foo:bar rollouts-pod-template-hash:7fc44855c7]ping(canary) selector =map[foo:bar rollouts-pod-template-hash:5cb4fd98cf]The stable service should select only pods from
stableRS, and the canary service should select only pods fromnewRS.Version
Current master at
4e6a2798688e22868340d9871a3c8d78371f1568.Additional context
A fix with unit tests is available in #4994.