How to distribute Kafka controllers across availability zones

[bollmann]

Hi Strimzi experts,

We're currently on Strimzi 0.45.1, Kafka 3.9.1, Kubernetes 1.32, and recently migrated to KRaft. Unfortunately, we misconfigured our controller KafkaNodePool consisting of 3 brokers. Our mistake was to only use a preferredDuringSchedulingIgnoredDuringExecution setting in our podAntiAffinity section to distribute the controller nodes across zones. Unfortunately this "preferred" podAntiAffinity wasn't respected in most of our landscapes and thus our 3 controller nodes ended up in the same AZ when migrating to KRaft mode.

Our badly configured controller KafkaNodePool looks as follows:

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  labels:
    strimzi.io/cluster: kafka-cluster
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 64Gi
        class: kafka-controller-ebs-gp3-xfs
        deleteClaim: false
  template:
    pod:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    strimzi.io/cluster: kafka-cluster
                    strimzi.io/controller-role: "true"
                topologyKey: topology.kubernetes.io/zone
              weight: 100
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  strimzi.io/cluster: kafka-cluster
                  strimzi.io/controller-role: "true"
              topologyKey: "kubernetes.io/hostname"

Note how each one of our Kafka controllers uses a PVC either on AWS, Azure, or GCP.

To distribute the Kafka controllers from the single AZ, say 1a, in AWS evenly across zones 1a, 1b, and 1c, I thought to use the following additional topologySpreadConstraints definition:

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule 
          labelSelector:
            matchLabels:
              strimzi.io/cluster: kafka-cluster
              strimzi.io/controller-role: "true"
          matchLabelKeys:
            - pod-template-hash

However, when I add such a topologySpreadConstraints definition to the above KafkaNodePool, the strimzi operator tries to roll the Kafka controllers and then gets stuck because the topology's maxSkew parameter is already violated. Ideally, the strimzi operator would not immediately roll the Kafka pods after I add such a topology spread constraint such that I could also annotate a controller pod with annotation

kubectl annotate pod kafka-cluster-controller-N "strimzi.io/delete-pod-and-pvc=true"

That is, ideally, I'd like to first delete a single Kafka controller together with its PVC and only let the strimzi operator reconcile the updated KafkaNodePool resource based on the newly added topologySpreadConstraints when the pod gets recreated. Is something like this possible?

Or should I follow a completely different approach to distribute the Kafka controllers from a single AZ to different AZs ?

My issue is similar to https://github.com/orgs/strimzi/discussions/11997. The solution there is to use the strimzi.io/delete-pod-and-pvc annotation to delete and recreate the controller nodes. However, in my attempts so far I couldn't figure out how to use that annotation together with an updated topology spread constraints definition without the strimzi operator rolling the kafka node pool and the controller pods becoming un-schedulable before I can annotate the pods I want to have recreated.

Any insights about how to overcome my problem are very much appreciated :-)

[scholzj]

What storage do you use in the different clusters? In AWS, if you use something like gp3, the volumes cannot move between the zones. That is of course a complication. But at least in GCP, there are storage types that are available across zones IIRC. So that should make things there easier, I guess?

---

However, when I add such a topologySpreadConstraints definition to the above KafkaNodePool, the strimzi operator tries to roll the Kafka controllers and then gets stuck because the topology's maxSkew parameter is already violated.

Are you saying that when you add the toplogySpreadConstraint to a 3 node cluster in a single zone, it is not handled by Kubernetes because the first Pod cannot be rolled without breaking skew constraint? I never really tried that and thought about it TBH. I guess it makes some sense, although it makes things also pretty wierd.

Can you use requiredDuringSchedulingIgnoredDuringExecution affinity instead? That should care less about other Pods I guess?

So in an environment like AWS with AZ-bound gp3 volumes, I would expect something like this:

• You add the requiredDuringSchedulingIgnoredDuringExecution
• The operator starts rolling the Pods to update the affinity
• The first Pod is deleted, but cannot be scheduled because it cannot run in the original zone due to the affinity and in any of the other zones because of the storage. So it keeps hanging as unschedulable or something and the Operator will keep waiting for the Pod to become ready.
• You go and manually delete the PVC of this Pod and wait for the operator to timeout the waiting for the Pod readiness (5 minutes by default)
• Once the waiting for readiness times out, new reconciliation should start, recreate the PVC, and Pod should bind to it and a new volume should be created in the new zone. -> so you will have 1 new Pod with new PV in the new zone and two old ones still in the old zone.
• Next, the operator will proceed witht he next Pod and try to schedule it into the 3rd zone in the same way. so you delete the PVC again, it gets recreated in the 3rd zone -> second Pod is fixed.
• And at that point, the 3rd Pod will be ok in the original zone and you should be done. + now oyu can in theory switch from the affinity to topologySpreadConstraint

That might work? (Please make sure to try it first in some test environment -> I did not test it, just wrote it based on my knowledge of the code base)

[bollmann]

Hi Jakub,

Many thanks for your detailed answer! I really appreciate it.

What storage do you use in the different clusters? In AWS, if you use something like gp3, the volumes cannot move between the zones. That is of course a complication. But at least in GCP, there are storage types that are available across zones IIRC. So that should make things there easier, I guess?

In AWS we use gp3 volumes (as you have guessed correctly). In GCP we use "Balanced persistent disk" volumes and in Azure we use "Standard SSD LRS" disks.

However, when I add such a topologySpreadConstraints definition to the above KafkaNodePool, the strimzi operator tries to roll the Kafka controllers and then gets stuck because the topology's maxSkew parameter is already violated.

Are you saying that when you add the toplogySpreadConstraint to a 3 node cluster in a single zone, it is not handled by Kubernetes because the first Pod cannot be rolled without breaking skew constraint? I never really tried that and thought about it TBH. I guess it makes some sense, although it makes things also pretty wierd.

Yes, exactly. In this case the first controller cannot be rolled because all three controllers reside in the same availability zone and recreating the pod in that same zone violates the skew constraint. In fact I think adding my mentioned topology spread constraint gives the same problem as a requiredDuringSchedulingIgnoredDuringExecution anti affinity constraint based on the topologyKey: topology.kubernetes.io/zone. In either case when the 3 Kafka controllers are already in the same zone, the first one cannot be rolled by strimzi because of incompatible constraints (the PVC requires the controller to stay in the same availability zone but the topology spread constraint or the pod anti affinity constraint require it to move to a different availability zone). Note that this is just my theory though and I didn't pursue my approach with topology spread constraints further.

When I read your detailed recipe of how to overcome the problem using a requiredDuringSchedulingIgnoredDuringExecution pod anti affinity rule, I went with your approach. And your trick to delete the PVC of the pending controller pod (which at this point in time is in "unschedulable" state due to conflicting scheduling constraints) solves the problem of the conflicting constraints: Once the PVC of the stuck controller is deleted, strimzi's next reconciliation cycle recreates the previously pending controller pod in an availability zone as mandated by the pod anti affinity constraints.

I tested this approach already in our dev and canary landscapes. I'll apply it to our productive clusters next.

Thanks again for unblocking me here!

[scholzj]

Great, glad this helped!