Kafka Output With SSL Fails

[EmrahKK]

Hi,

When configuring a ClusterOutput to kafka with ssl authentication, logging operator produces fluent d config with ssl_ca_cert, ssl_client_cert and ssl_client_cert_key fields which includes certificates add key itself. After fluentd keeps logging error about certificate and key paths.

<label @06cb8f659345bc35696661e33ae28ac2>
  <match **>
    @type kafka2
    @id clusterflow:cattle-logging-system:clusterlogs-flow:clusteroutput:cattle-logging-system:clusterlogs-output
    brokers mymperf-cluster-kafka-bootstrap.message-bus:9093
    default_topic clusterlogs
    sasl_over_ssl false
    ssl_ca_cert -----BEGIN CERTIFICATE-----
    *
    *
    *
    -----END CERTIFICATE-----

    ssl_client_cert -----BEGIN CERTIFICATE-----
    *
    *
    *
    -----END CERTIFICATE-----

    ssl_client_cert_key -----BEGIN PRIVATE KEY-----
    *
    *
    *
    -----END PRIVATE KEY-----

    <format>
      @type json
    </format>
  </match>
</label>

Environment details:

• Kubernetes version : v1.18.8
• logging-operator version (e.g. 2.1.1): logging-operator:3.14.2
• Install method (e.g. helm or static manifests): helm
• Logs from the misbehaving component (and any other relevant logs):

<Fluent::Config::Section {"@label":"@06cb8f659345bc35696661e33ae28ac2","tag":"","matches":["<Fluent::Config::Section {\"labels\":{},\"namespaces\":[],\"hosts\":[],\"container_names\":[\"echo-server\"],\"negate\":false}>"]}>
2021-09-18 19:39:41 +0000 [error]: fluent/log.rb:371:error: unexpected error error_class=Errno::ENOENT error="No such file or directory @ rb_sysopen - -----BEGIN CERTIFICATE-----"
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/kafka_plugin_util.rb:39:in `read'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/kafka_plugin_util.rb:39:in `block in read_ssl_file'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/kafka_plugin_util.rb:39:in `map'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/kafka_plugin_util.rb:39:in `read_ssl_file'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/out_kafka2.rb:109:in `refresh_client'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-kafka-0.16.0/lib/fluent/plugin/out_kafka2.rb:183:in `start'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:200:in `block in start'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:179:in `block (2 levels) in lifecycle'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/agent.rb:121:in `block (2 levels) in lifecycle'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/agent.rb:120:in `each'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/agent.rb:120:in `block in lifecycle'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/agent.rb:113:in `each'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/agent.rb:113:in `lifecycle'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:178:in `block in lifecycle'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:175:in `each'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:175:in `lifecycle'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/root_agent.rb:199:in `start'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/engine.rb:248:in `start'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/engine.rb:147:in `run'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/supervisor.rb:607:in `block in run_worker'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/supervisor.rb:845:in `main_process'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/supervisor.rb:598:in `run_worker'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/command/fluentd.rb:361:in `<top (required)>'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/bin/fluentd:8:in `<top (required)>'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/bin/fluentd:23:in `load'
  2021-09-18 19:39:41 +0000 [error]: fluent/supervisor.rb:844:main_process: /usr/bin/fluentd:23:in `<main>'
<Fluent::Config::Section {"@label":"@06cb8f659345bc35696661e33ae28ac2","tag":"","matches":["<Fluent::Config::Section {\"labels\":{},\"namespaces\":[],\"hosts\":[],\"container_names\":[\"echo-server\"],\"negate\":false}>"]}>

• Resource definition (possibly in YAML format) that caused the issue, without sensitive data:

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: clusterlogs-output
spec:
  kafka:
    brokers: mymperf-cluster-kafka-bootstrap.message-bus:9093
    default_topic: clusterlogs
    format:
      type: json
    ssl_ca_cert:
      valueFrom:
        secretKeyRef:
          key: ca.pem
          name: clusterlogs-ca-cert
    ssl_client_cert:
      valueFrom:
        secretKeyRef:
          key: user-cert.pem
          name: clusterlogs-client-cert
    ssl_client_cert_key:
      valueFrom:
        secretKeyRef:
          key: user-key.pem
          name: clusterlogs-client-key

[jam-re]

Same bug here

[belgaied2]

Some analysis on the issue:

The ca_file attribute in the ClusterOutput object for multiple backends has the same type as a secret Object in the Kubernetes API, which can have either a mountFrom field or a valueFrom field.

However, Fluentd or more exactly fluent-plugin-kafka accept the ssl_ca_cert parameter as a certificate path, but Logging operator generates the value of that parameter as path only if mountFrom is used. If the user uses valueFrom the generated configuration is wrong!

That means, this is indeed a bug in Usability. The user should not have the possibility to use valueFrom because it always causes misconfiguration for fluentd.

BTW, this does not only happen with Kafka, but also with Elasticsearch when using ca_file or client_cert or client_key parameters.

Two possibilities I can image:

• Change the object type of parameters such as ca_file or such to NOT be a *secret.Secret but a custom object only accepting mountFrom
• OR Change the way the operator works to generate a path even if user has chosen valueFrom

[tarokkk]

The above comment is the answer as well. For certificates requiring a path as configuration use the mountFrom option. This is a trade-off between overcomplicating the configuration types / let the user decide what the plugin needs. There are plugins where both methods work like google JSON keys etc...