Skip to content

Concat filter : multiline_end_regexp value is not formatted properly in Fluentd config #2071

Open
@loic-brtd

Description

@loic-brtd

Describe the bug:
When using the Concat Filter, I think the regexp value for the field multiline_end_regexp is not properly formatted into the generated Fluentd config :

My Flow configuration:

apiVersion: v1
- apiVersion: logging.banzaicloud.io/v1beta1
  kind: Flow
  metadata:
    name: my-flow
    namespace: xxx
  spec:
    filters:
    - concat:
        key: log
        multiline_end_regexp: /\n$/
        separator: ""
        use_first_timestamp: true
    - grep:
        regexp:
        - key: log
          pattern: /^\{.*\}/
    - parser:
        key_name: log
        parse:
          type: json
        remove_key_name_field: true
        reserve_data: true
    localOutputRefs:
    - my-output
    match:
    - select:
        container_names:
        - xxx

The resulting FluentdConfig (taken from the logs of the pod rancher-logging-root-fluentd-configcheck) has the value "/\\n$/" for the multiline_end_regexp field :

  <label @2b0669f6af9ddff877bae45460fe9d59>
    <filter **>
      @type concat
      @id flow:xxx:my-flow:0
      key "log"
      multiline_end_regexp "/\\n$/"
      use_first_timestamp true
    </filter>
    <filter **>
      @type grep
      @id flow:xxx:my-flow:1
      <regexp>
        key "log"
        pattern /^\{.*\}/
      </regexp>
    </filter>
    <filter **>
      @type parser
      @id flow:xxx:my-flow:2
      key_name "log"
      remove_key_name_field true
      reserve_data true
      <parse>
        @type "json"
      </parse>
    </filter>
    <match **>
      @type elasticsearch_data_stream

Expected behaviour:
I expect to obtain the value /\n$/ in the Fluentd config :

    <filter **>
      @type concat
      @id flow:xxx:my-flow:0
      key "log"
      multiline_end_regexp /\n$/
      use_first_timestamp true
    </filter>

Steps to reproduce the bug:

  • Create a flow with the above Flow configuration
  • Check the generated Fluentd configuration

Additional context:
I'm dealing with JSON logs which have a size greater than 16 Kib an Docker splits them in multiples logs. I was trying to solve this problem by adding this Fluentd configuration but when porting it to my Flow resource, the regexp for the field multiline_end_regexp is not formatted properly in the resulting Fluentd config.

Environment details:

  • Kubernetes version: v1.25.14
  • Cloud-provider/provisioner: RKE1
  • logging-operator version: 3.17.10
  • Install method (e.g. helm or static manifests): Rancher Logging plugin (chart is rancher-logging:102.0.3+up3.17.10)
  • Logs from the misbehaving component (and any other relevant logs): /
  • Resource definition (possibly in YAML format) that caused the issue, without sensitive data: see above

/kind bug

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions