Kafka Pod high memory utilization after GKE nodepool upgrade from v1.30 to v1.32, causing OOM

[dsolanki-aeris]

I upgraded my GKE node pool for Kafka from version 1.30 to 1.32. After the upgrade, the Kafka Pod memory utilization is consistently high, between 92% and 97%. Before the upgrade, the memory utilization would reach 90% and then reduce to 15%. Now, the Kafka pods are experiencing Out of Memory (OOM) errors, leading to outages. Below are the specs of kafka:

strimzi-kafka-operator version: 0.39.0
kafkaVersion: 3.6.1

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka
  namespace: infra
spec:
  kafka:
    config:
      auto.create.topics.enable: false
      auto.leader.rebalance.enable: true
      controlled.shutdown.enable: true
      default.replication.factor: 3
      delete.topic.enable: true
      leader.imbalance.check.interval.seconds: 900
      log.retention.check.interval.ms: 300000
      log.retention.hours: 72
      log.segment.bytes: 1073741824
      min.insync.replicas: 1
      num.io.threads: 24
      num.network.threads: 5
      num.recovery.threads.per.data.dir: 1
      socket.receive.buffer.bytes: 102400
      socket.request.max.bytes: 33554432
      socket.send.buffer.bytes: 102400
      zookeeper.connection.timeout.ms: 18000
    jmxOptions: {}
    jvmOptions:
      -Xms: 9g
      -Xmx: 9g
    listeners:
    - configuration:
        brokers:
        - advertisedHost: pre-kafka-0.aer.com
          advertisedPort: 9092
          annotations:
            cloud.google.com/neg: '{"exposed_ports": {"9092":{}}}'
          broker: 0
        - advertisedHost: pre-kafka-1.aer.com
          advertisedPort: 9092
          annotations:
            cloud.google.com/neg: '{"exposed_ports": {"9092":{}}}'
          broker: 1
        - advertisedHost: pre-kafka-2.aer.com
          advertisedPort: 9092
          annotations:
            cloud.google.com/neg: '{"exposed_ports": {"9092":{}}}'
          broker: 2
      name: plain
      port: 9092
      tls: false
      type: cluster-ip
    - name: gkeinternal
      port: 19092
      tls: false
      type: internal
    - configuration:
        brokers:
        - advertisedHost: pre-int-kafka-0.aer.com
          advertisedPort: 29092
          annotations:
            cloud.google.com/neg: '{"exposed_ports": {"29092":{}}}'
          broker: 0
        - advertisedHost: pre-int-kafka-1.aer.com
          advertisedPort: 29092
          annotations:
            cloud.google.com/neg: '{"exposed_ports": {"29092":{}}}'
          broker: 1
        - advertisedHost: pre-int-kafka-2.aer.com
          advertisedPort: 29092
          annotations:
            cloud.google.com/neg: '{"exposed_ports": {"29092":{}}}'
          broker: 2
      name: projectint
      port: 29092
      tls: false
      type: cluster-ip
    livenessProbe:
      timeoutSeconds: 60
    logging:
      loggers:
        kafka.root.logger.level: WARN
        log4j.logger.kafka: WARN
        log4j.logger.kafka.authorizer.logger: WARN
        log4j.logger.kafka.controller: INFO
        log4j.logger.kafka.log.LogCleaner: WARN
        log4j.logger.org.I0Itec.zkclient.ZkClient: WARN
        log4j.logger.org.apache.kafka: WARN
        log4j.logger.org.apache.zookeeper: WARN
        log4j.logger.state.change.logger: WARN
      type: inline
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: my-cluster-kafka-jmx-exporter-configuration.yaml
          name: my-cluster-kafka-jmx-exporter-configuration
    readinessProbe:
      timeoutSeconds: 60
    replicas: 3
    resources:
      limits:
        cpu: 24
        memory: 102Gi
      requests:
        cpu: 20
        memory: 90Gi
    storage:
      type: jbod
      volumes:
      - class: sc-kafka
        deleteClaim: false
        id: 0
        size: 1Ti
        type: persistent-claim
      - class: sc-kafka
        deleteClaim: false
        id: 1
        size: 1Ti
        type: persistent-claim
      - class: sc-kafka
        deleteClaim: false
        id: 2
        size: 1Ti
        type: persistent-claim
    template:
      bootstrapService:
        metadata:
          annotations:
            prometheus.io/path: /metrics
            prometheus.io/port: "9404"
            prometheus.io/scrape: "true"
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: cloud.google.com/gke-nodepool
                  operator: In
                  values:
                  - node-pool-kafka
        metadata:
          annotations:
            prometheus.io/path: /metrics
            prometheus.io/port: "9404"
            prometheus.io/scrape: "true"
    version: 3.6.1
  kafkaExporter:
    groupRegex: .*
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: cloud.google.com/gke-nodepool
                  operator: In
                  values:
                  - node-pool-kafka
    topicRegex: .*
  zookeeper:
    jvmOptions:
      -Xms: 512m
      -Xmx: 512m
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: my-cluster-zookeeper-jmx-exporter-configuration.yaml
          name: my-cluster-zookeeper-jmx-exporter-configuration
    replicas: 3
    storage:
      class: sc-zookeeper
      deleteClaim: false
      size: 5000Mi
      type: persistent-claim
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: cloud.google.com/gke-nodepool
                  operator: In
                  values:
                  - node-pool-kafka

[scholzj]

I think a good start would be to try to use the same memory request as memory limit and see if that helps or not.

[dsolanki-aeris]

We already tried with same request and limit but no change in behaviour.