help request: Fluentbit - Mixing Clusterfilters with Filter

[Crazyigor1987]

Describe the issue

I intend to run the following settings on our Kubernetes cluster. CloudOps create ClusterConfigs + Filter, which pre-analyse the log data and feed it with important metadata. We have created "ClusterFilter" for this purpose. In the second step, developers should be allowed to create their own filters at namespace level. This is what "Filters" are intended for.

Something like that:

BLOCK "Sources " - ClusterFilter by CloudOp
001-systemd
002-containerd
003-multiline
004-kubernetes

BLOCK "Application" - Filter Namespaced by Developer
filter-web-ns1
filter-web-ns2
...
...

BLOCK "METADATA + INDEX" - ClusterFilter by CloudOp
zzz-cluster-metadata
zzz-opensearch-index-name

The operator does not report any errors here, but only generates the config that it finds ClusterFilter, not that of the developer within his namespace.

This Filter is not created within the "fluent-bit-config" secret:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: Filter
metadata:
  labels:
    fluentbit.fluent.io/component: logging
    fluentbit.fluent.io/enabled: "true"
  name: test-filter
  namespace: ns1
spec:
  match: kube.*
  filters:
  - parser:
      keyName: raw_event
      parser: json

How did you install fluent operator?

Helm

Additional context

No response

[Crazyigor1987]

It was my mistake. You need for the Filter also the config, which refers to the clusteroutput. Something like

apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBitConfig
metadata:
  name: fluentbit-clusterconfig-referrer
  namespace: ns1
spec:
  outputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
      fluentbit.fluent.io/component: logging

After that, the Filter could successfully created.

[cw-Guo]

@Crazyigor1987 Thanks for your feedback. We will add a more detailed document to talk about the namespaced CRDs.