Fluent-Bit kubernetes_events input plugin failed buffer write error

[Akila-I]

We are using kubernetes_events input plugin and forwarding events to OpenSearch via Fluent-Bit (version 3.16). I'm getting this error in fluent-bit pod logs continuously and the pod crashes at some point. Can someone explain why this is happening and a way to fix this please. I think its because too many events are generated in the cluster and fluent-bit pod goes out of memory. Is that the case?

error (log level - debug):

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=0

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=1

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=2

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=3

[2025/01/08 08:44:59] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=4

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=5

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=6

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=7

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=8

[2025/01/08 08:45:00] [debug] [input:kubernetes_events:kubernetes_events.0] failed buffer write, retries=9

[2025/01/08 08:45:00] [error] [input:kubernetes_events:kubernetes_events.0] could not enqueue records into the ring buffer

config:

        name kubernetes_events
        kube_url https://kubernetes.default.svc/
        tag k8s_events
        tls.verify Off

[patrick-stephens]

there's a ring buffer yeah and it looks like it is filling up

@pwhelan is the ring buffer size configurable?

The alternative is might be to split into per-namespace K8S event inputs as well @Akila-I, they can all have the same tag for output

[Akila-I]

Hi @patrick-stephens ,

We have a situation where namespaces are dynamically created (and there's a lot of namespaces) which we can't know the namespace names beforehand. We could try to categorise them into few groups of namespaces (e.g system namespaces and application namespaces). But I don't think we can provide a list of namespaces or a namespace pattern (e.g system-ns-*) to configure the kubernetes_events input plugin. It either accepts one namespace name or collect events from all namespaces is what I understood when reading the codebase.

I believe the solution should be to provide a configurable for ring buffer size. @pwhelan appreciate your suggestions as well.

[pwhelan]

The ring buffer is not configurable at this moment, it is hard coded to hold at most 1024 records at a time. It could be configurable in the future.

@edsiper thoughts?

[Akila-I]

The ring buffer is not configurable at this moment, it is hard coded to hold at most 1024 records at a time. It could be configurable in the future.

@edsiper thoughts?

Have we encountered this ring buffer overflow in a load test before? We are collecting kubernetes events from 10,000+ namespaces. I believe 1024 records for the buffer will not be enough for such cases.