Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Argo rollouts manages virtualService field after a rollout is complete. Causes future changes to VS to not sync #3854

Open
2 tasks done
taer opened this issue Sep 26, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@taer
Copy link

taer commented Sep 26, 2024

Checklist:

  • I've included steps to reproduce the bug.
  • I've included the version of argo rollouts.

Describe the bug
We use rollouts with istio traffic management. Initially, when argo-rollouts changed the weights in the virtual service, argocd would fight it, causing sync flapping. We used to have auto-sync on, so this was a problem. To remedy, we added this to the application

  ignoreDifferences:
  - group: '*'
    kind: '*'
    managedFieldsManagers:
    - rollouts-controller

This solved the sync issue.

Now, when a rollout occurs, Rollouts correctly modifies the VirtualService weights for the canary, and argo doesn't fight it because rollouts set this in the VS prior to changing the weights

    - apiVersion: networking.istio.io/v1alpha3
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:http: {}
      manager: rollouts-controller
      operation: Update
      time: '2024-09-26T20:24:14Z'

All is well for the rollout. However, once one rollout happens, that managedField remains in the virtualService, making Argocd never see any changes to the VS unless we manually remove that managed field. Then it instantly sees it

To Reproduce
reproducible deploy here: https://github.com/taer/argo-rollout-issue/tree/master

use the linked application. Deploy it. No VS changes occur. Change the VS http section. I changed the prefix

    - match:
        - uri:
            prefix: /headers

Prior to a rollout, changing this results in an expected diff. Now change the pod template. I edited

        - name: CHANGEME
          value: aa

to force a rollout. ArgoRollouts starts, deploys the new pod, then adds the ManagedField to the VS and starts to alter the weights for the canary deploy. When complete, the managed fields still exist. Now this means any future changes to the VirtualService in git are ignored.

Expected behavior

I expect the managedField argo rollouts inserted to be removed when the rollout is complete so that future virtual service changes can be noticed by argocd.

Version
Argo-rollouts: 1.7.2
argo-rollouts-helm: 2.37.5
istio: 1.22.3

Logs

Logs are in the github repro repo.
https://github.com/taer/argo-rollout-issue/blob/master/errorLog.log


Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

@taer taer added the bug Something isn't working label Sep 26, 2024
@taer
Copy link
Author

taer commented Sep 26, 2024

Removed old comment about RemoveManagedRoutes()

This was all incorrect. I assumed managedRoutes was the managed fields stuff, but instead was https://argo-rollouts.readthedocs.io/en/stable/features/traffic-management/#traffic-router-support-istio

@taer
Copy link
Author

taer commented Sep 27, 2024

linked logs
errorLog.log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant