[Q&A] config from fluent-package 5 errors on 6 with an alleged duplicate label

[dsmith4-godaddy]

What is the problem?

I'm testing upgrading a few systems from fluent-package 5.0.7 to fluent-package 6.0.0. I haven't made any configuration changes (yet), just upgraded the package and reinstalled the plugins for the new version.

The config works fine on fluent-package 5, but on 6, fluentd refuses to start up, saying Section <label @ARES_JSON_TCP> appears twice. The label name only appears twice in the configs -- once in a section to add the label to incoming events, then in a section to match and process those events.

I've tried some simple config edits to remove the label and match only on tags, but I then get similar problems with other label/match pairs that aren't shown in the below config. (I omitted a couple of my included config files, to try to simplify this test case.) I don't see any sort of include repeats/loops (and I'd hope that fluentd would know not to do that anyway).

Are there any changes in how fluentd 1.19.0 parses configs that might have made this configuration invalid, or is this a possible regression?

Describe the configuration of Fluentd

When fluentd is started via systemd it reads in /etc/fluent/fluentd.conf:

# Ansible managed
<system>
  root_dir /var/log/fluent
</system>

<label @FLUENT_LOG>
  <match fluent.**>
    @type stdout
  </match>
</label>

@include conf.d/*.conf

The include above pulls in a few *.conf files from conf.d. The first, alphabetically, is /etc/fluent/conf.d/gateway2.conf, which listens for log events on a port on localhost, adds a label to those events, then copies events with that label both to a local file, and to another label. (Note that this is the only file that contains the text that is seemingly preventing fluentd from starting.)

# Ansible managed
<source>
  @type tcp
  @id in_gateway2_tcp
  @label @ARES_JSON_TCP
  tag gateway_access.gateway2_json
  bind 127.0.0.1
  port 6666
  format json
</source>

<label @ARES_JSON_TCP>
  <match gateway_access.gateway2_json>
    @type copy
    <store>
      @type file
      @id out_gateway2_json
      path /mnt/log/gateway2/json/access
      format json
      compress gzip
      <buffer>
        timekey_use_utc true
        path /mnt/log/gateway2/json/access
      </buffer>
    </store>
    <store>
      @type relabel
      @label @ARES_TO_ELASTIC
    </store>
  </match>
</label>

The second, /etc/fluent/conf.d/gw2_to_elastic.conf, takes the label we duplicated events to above, and sends them to an outside log store:

# Ansible managed
<label @ARES_TO_ELASTIC>

  <filter gateway_access.gateway2_json>
    @type grep
    <exclude>
      key mode
      pattern /miss/
    </exclude>
  </filter>

  <filter gateway_access.gateway2_json>
    @type record_modifier
    <record>
      ## Here, I've removed a few dozen event renames/transforms/etc
    </record>

    whitelist_keys  event_timestamp,...
    ## Again, I removed a few dozen event keys from the whitelist_keys
  </filter>

  <match gateway_access.gateway2_json>
    @type kinesis_streams_aggregated
    region us-west-2
    stream_name gw2-access-logs
    <assume_role_credentials>
      ## Removing the actual AWS credentials for obvious reasons
    </assume_role_credentials>
    <buffer>
      @type file
      path /mnt/log/fluent/gw2_elastic/
      total_limit_size 1GB
      overflow_action drop_oldest_chunk
    </buffer>
  </match>
</label>

There are two other files alphabetically after the above, which I've omitted here to simplify the test case. One pulls in events from the system journal and sends them to a SIEM; the other tails a couple of log files, and forwards that to another destination. Those files do exhibit a similar issue to the ones above, in that one of them adds a label to events, then uses a section to match that label, and if the above are modified to remove the @ARES_JSON_TCP label, fluentd still won't start because it says a label in those files appears twice.

Describe the logs of Fluentd

fluent-package 6 startup logs:

2025-09-16 14:22:35 +0000 [info]: init supervisor logger path="/mnt/log/fluent/fluent.log" rotate_age=nil rotate_size=nil
2025-09-16 14:22:35 +0000 [info]: parsing config file is succeeded path="/etc/fluent/fluentd.conf"
2025-09-16 14:22:35 +0000 [info]: loading additional configuration file path="/etc/fluent/conf.d/gateway2.conf"
2025-09-16 14:22:35 +0000 [info]: loading additional configuration file path="/etc/fluent/conf.d/gw2_elastic.conf"
2025-09-16 14:22:35 +0000 [info]: loading additional configuration file path="/etc/fluent/conf.d/includes/gateway-forwarder.conf"
2025-09-16 14:22:35 +0000 [info]: loading additional configuration file path="/etc/fluent/conf.d/systemd_secure.conf"
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-dict-map' version '0.1.1'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '6.0.0'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-fluent-package-update-notifier' version '0.2.3'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-kafka' version '0.19.5'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.5.0'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-metrics-cmetrics' version '0.1.2'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-obsolete-plugins' version '0.2.2'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-opentelemetry' version '0.3.0'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.2.1'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.2.1'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.2.1'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-systemd' version '1.1.1'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-td' version '1.2.0'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2025-09-16 14:22:35 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.6.0'
2025-09-16 14:22:35 +0000 [info]: gem 'fluentd' version '1.19.0'
2025-09-16 14:22:35 +0000 [error]: config error file="/etc/fluent/fluentd.conf" error_class=Fluent::ConfigError error="Section <label @ARES_JSON_TCP> appears twice"

Environment

- Fluentd version: 1.19.0
- Fluent Package version: 6.0.0
- Operating system: Ubuntu 24.04.1 LTS
- Kernel version: 6.8.0-1032-aws #34-Ubuntu SMP Fri Jun 27 19:10:36 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

[daipom]

@dsmith4-godaddy
Sorry for the trouble.
Please try the following setting.

<system>
  config_include_dir ""
</system>

• ref: config_include_dir

fluent-package v6.0.0 reads config files under config_include_dir.
The default value is /etc/fluent/conf.d.
If a user manually creates and uses that directory, those config files will be included duplicatedly.

[dsmith4-godaddy]

That seems to have worked. (It exposed a different possible issue with fluentd 1.19, but I'll open a different discussion for that if needed.) Thank you for the help!