Skip to content

Helm chart 3.32.2 upgrade issue on EKS 1.36 #14019

Description

@joshuaganger

I ran into an issue upgrading from 3.30.0 to 3.32.2 on an EKS 1.36 cluster.

Kubernetes version: Server Version: v1.36.3-eks-cb19647

The general sequence of operations:

  1. I upgraded the cluster & node group from 1.34 -> 1.36 in two stages.
  2. I server-side applied new CRDs
  3. I attempted the helm upgrade.

The outcome is a tigera-operator pod that exits after logging "Could not wait for Cache to sync:". None of the other Calico components were upgraded, and remain on image v3.30.0. kubectl get tigerastatus shows all components as available.

My install/upgrade looks like this:

helm repo add projectcalico https://docs.tigera.io/calico/charts
helm repo update
kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/operator-crds.yaml
helm upgrade --install calico projectcalico/tigera-operator --version v3.32.2 -f values.yaml --namespace tigera-operator

The values.yaml just contains the image registry:

installation:
  kubernetesProvider: EKS
  registry: quay.io/

After the helm upgrade succeeds, when tigera-operator is crashing:

kubectl get pods -n calico-system -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | sort -u
quay.io/calico/csi:v3.30.0 quay.io/calico/node-driver-registrar:v3.30.0
quay.io/calico/goldmane:v3.30.0
quay.io/calico/kube-controllers:v3.30.0
quay.io/calico/node:v3.30.0
quay.io/calico/typha:v3.30.0
quay.io/calico/whisker:v3.30.0 quay.io/calico/whisker-backend:v3.30.0

kubectl get tigerastatus
NAME        AVAILABLE   PROGRESSING   DEGRADED   SINCE
apiserver   True        False         False      8m44s
calico      True        False         False      23m
goldmane    True        False         False      23m
ippools     True        False         False      24m
whisker     True        False         False      24m

Prior to the exit, there are a lot of log entries like:
{"level":"info","ts":"2026-09-22T17:41:23Z","logger":"controller-runtime.cache","msg":"Warning: event bookmark expired","err":"pkg/mod/k8s.io/client-go@v0.36.4/tools/cache/reflector.go:343: awaiting required bookmark event for initial events stream, no events received for 1m20.117864412s"}

I was able to work around the crash by setting the following environment variable on the tigera-operator deployment. This caused the upgrade to complete, and all components came up on 3.32.2. This isn't a permanent fix, but could potentially be integrated into the helm values file if necessary:

kubectl -n tigera-operator set env deployment/tigera-operator KUBE_FEATURE_WatchListClient=false

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions