Unable to send syslog messages to graylog server

[nitrogene]

Hello,

I use wsl2 to launch some containers via docker-compose: graylog, mongo, elastic search, fluentd, a client embedding an instance of fluentbit and a container running fluentd.

In the docker compose file, the fluentd container is configured as:

    # fluentd: https://hub.docker.com/r/fluent/fluentd
    fluentd:
      build: ./fluentd
      container_name: some-fluentd
      hostname: some-fluentd
      volumes:
        - ./fluentd/fluent.conf:/fluentd/etc/fluent.conf:ro
        - fluentd-log-storage:/fluentd/log:rw
      networks:
        - graylog
      restart: unless-stopped
      ports:
        # syslog
        - 514:514/udp
        # fluentd
        - 24224:24224
        - 24224:24224/udp

While the graylog container is configured as:

# Graylog: https://hub.docker.com/r/graylog/graylog/
    graylog:
      image: graylog/graylog:4.2.7
      container_name: some-graylog
      hostname: some-graylog
      volumes:
        - graylog-storage:/usr/share/graylog/data:rw
      environment:
        ...
      entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
      networks:
        - graylog
      restart: unless-stopped
      depends_on:
        - fluentd
        - mongo
        - elasticsearch
      ports:
        # Graylog web interface and REST API
        - 9000:9000
        # Syslog TCP
        - 1514:1514
        # Syslog UDP
        - 1514:1514/udp
        # GELF TCP
        - 12201:12201
        # GELF UDP
        - 12201:12201/udp
      logging:
        driver: fluentd
        options:
          fluentd-address: localhost:24224
          tag: "docker.{{.Name}}"

In my fluentd.conf file I have the following sources:

# watch for messages on port 24224
<source>
  @type  forward
  port  24224
</source>

# watch for syslog messages on port 514
<source>
  @type syslog
  port 514
  tag syslog
</source>

And the corresponfin match section:

# match tag=syslog.** and write to file
<match syslog.**>
  @type copy
  <store>
    @type file
    <buffer time>
      timekey 3600
      timekey_wait 600
    </buffer>
    path         /fluentd/log/syslog.%Y-%m-%d-%H.log
    symlink_path /fluentd/log/syslog.log
    append       true
    time_slice_format %Y%m%d
    time_format       %Y%m%dT%H%M%S%z
    time_key time 
  </store>
  <store>
    @type gelf
    host some-graylog
    port 12201
    protocol udp
    <buffer>
      flush_at_shutdown true
      flush_mode immediate
      flush_thread_count 8
      flush_thread_interval 1
      flush_thread_burst_interval 1
      retry_forever true
      retry_type exponential_backoff
    </buffer>
  </store>
</match>

In a wsl2 shell, I simulate a syslog message via:

logger -n localhost --rfc3164 This is an awesome log line

By logging into the fluentd container, I can see the above line in the corresponding log file. But nothing goes to graylog. In graylog, I have added the right input (I am receiving other logs such as the docker logs via docker fluentd logging driver).

I'm kind of lost there...

Any help would be greatly appreciated.

Jean-Pierre

[fujimotos]

Is your graylog server reachable from Fluentd? You need to check it by
logging into Fluentd container and send a UDP datagram to some-graylog:12201
manually.

The probability is that Fluentd is sending UDP messages, but no one catch
them.

[nitrogene]

I can confirm that the graylog server is reachable from fluentd as I have other messages (coming from the source listening on port 24224) the that are forwarded to the graylog server.

Could my problem be related to message format ? I tried to define the syslog source as:

# watch for syslog messages on port 514
<source>
  @type syslog
  port 514
  tag syslog
  <parse>
    message_format rfc5424
  </parse>
</source>

Then if I send a syslog message using:

logger -n localhost --rfc5424 This is an awesome log line, yes?

I can see an error in graylog:

2022-03-08T14:52:18+00:00       docker.some-graylog     {"container_name":"/some-graylog","source":"stdout","log":"2022-03-08 14:52:18,014 ERROR: org.graylog.storage.elasticsearch7.MessagesAdapterES7 - Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:","container_id":"1599219abde8013bee03d106c3dfbe44c76660ae4c8cc0c4e718f06bad121fa0","time":1646751138.0}
2022-03-08T14:52:18+00:00       docker.some-graylog     {"container_id":"1599219abde8013bee03d106c3dfbe44c76660ae4c8cc0c4e718f06bad121fa0","container_name":"/some-graylog","source":"stdout","log":"[0]: index [graylog_0], type [_doc], id [5926a541-9eef-11ec-9d27-0242ac140006], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse field [pid] of type [long] in document with id '5926a541-9eef-11ec-9d27-0242ac140006'. Preview of field's value: '-']]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=For input string: \"-\"]];]","time":1646751138.0}

It seems that it has some issues while parsing the date, and the id ?

To summarize:

• using rfc5424 format causes an error in graylog
• using rfc3164 format causes no error, but I can not see the message

[nitrogene]

Hello,
I have found the reason. All my containers (including fluend and graylog) are using UTC Timezone, while the WSL2 shell I am using is Europe/Paris (UTC+1). So when I am executing the following command:

logger -n localhost --rfc5424 This is an awesome log line, yes?

It will generate a syslog message with a timestamp in the future (for graylog). Thus, the message will only appear after 1h.