Aws::Xml::Parser::ParsingError error="Missing end tag for 'BR'

[imran9m]

Hello,

We are using following versions of FluentD and FluentD-S3 Plugin in Kubernetes Environment.

2021-07-06 16:11:09 +0000 [info]: gem 'fluent-plugin-s3' version '1.5.1'
2021-07-06 16:11:09 +0000 [info]: gem 'fluentd' version '1.8.1'

We are observing lot of following warning messages in FluentD logs and our application is going through unexpected performance issues. We are wondering what is the cause for this?.

2021-07-06 16:23:46 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-07-06 16:23:47 +0000 chunk="5c676d89062282078b02f574dc1c9bee" error_class=Aws::Xml::Parser::ParsingError error="Missing end tag for 'BR' (got 'FONT')\nLine: \nPosition: \nLast 80 unconsumed characters:\n\r <blockquote>\r <TABLE border=0 cellPadding=1 width=\"80%\">\r <TR><TD>\r <FONT face="
2021-07-06 16:23:46.330838951 +0000 fluent.warn: {"retry_time":0,"next_retry_seconds":"2021-07-06 16:23:47 +0000","chunk":"5c676d89062282078b02f574dc1c9bee","error":"#<Aws::Xml::Parser::ParsingError: Missing end tag for 'BR' (got 'FONT')\nLine: \nPosition: \nLast 80 unconsumed characters:\n\r <blockquote>\r <TABLE border=0 cellPadding=1 width=\"80%\">\r <TR><TD>\r <FONT face=>","message":"failed to flush the buffer. retry_time=0 next_retry_seconds=2021-07-06 16:23:47 +0000 chunk=\"5c676d89062282078b02f574dc1c9bee\" error_class=Aws::Xml::Parser::ParsingError error=\"Missing end tag for 'BR' (got 'FONT')\\nLine: \\nPosition: \\nLast 80 unconsumed characters:\\n\\r <blockquote>\\r <TABLE border=0 cellPadding=1 width=\\\"80%\\\">\\r <TR><TD>\\r <FONT face=\""}
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/xml/parser/stack.rb:47:in `error'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/xml/parser/engines/rexml.rb:32:in `rescue in parse'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/xml/parser/engines/rexml.rb:26:in `parse'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/xml/parser.rb:36:in `parse'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/rest/response/body.rb:45:in `parse'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/rest/response/body.rb:27:in `apply'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/rest/response/parser.rb:34:in `extract_body'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/rest/response/parser.rb:14:in `apply'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/rest/handler.rb:11:in `block in call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/seahorse/client/response.rb:46:in `block in on'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/seahorse/client/http/response.rb:146:in `block in on_success'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/seahorse/client/http/response.rb:173:in `block in listener'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/seahorse/client/http/response.rb:137:in `on_done'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/seahorse/client/http/response.rb:144:in `on_success'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/seahorse/client/response.rb:45:in `on'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/rest/handler.rb:11:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/plugins/endpoint_discovery.rb:80:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/seahorse/client/plugins/endpoint.rb:47:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-s3-1.91.0/lib/aws-sdk-s3/plugins/arn.rb:72:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/seahorse/client/plugins/raise_response_errors.rb:16:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-s3-1.91.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-s3-1.91.0/lib/aws-sdk-s3/plugins/dualstack.rb:36:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-s3-1.91.0/lib/aws-sdk-s3/plugins/accelerate.rb:50:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/seahorse/client/plugins/request_callback.rb:71:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.113.0/lib/seahorse/client/request.rb:72:in `send_request'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-s3-1.91.0/lib/aws-sdk-s3/client.rb:11267:in `put_object'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/aws-sdk-s3-1.91.0/lib/aws-sdk-s3/object.rb:1330:in `put'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluent-plugin-s3-1.5.1/lib/fluent/plugin/out_s3.rb:365:in `write'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.8.1/lib/fluent/plugin/output.rb:1133:in `try_flush'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.8.1/lib/fluent/plugin/output.rb:1439:in `flush_thread_run'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.8.1/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2021-07-06 16:23:46 +0000 [warn]: #0 /usr/local/lib/ruby/gems/2.6.0/gems/fluentd-1.8.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Below is the Fluentd configuration.

<label @FORWARD>
  <match app-tag**>
    @type s3

    s3_bucket "xxxx"
    aws_key_id "xxxxx"
    aws_sec_key "xxxxx"
    s3_region "xxxxx"
    path "prefix/${tag}/%Y/%m/%d/%H"
    s3_object_key_format "%{path}/%{time_slice}_#{Socket.gethostname}_%{index}_%{hex_random}.%{file_extension}"
    include_time_key true
    time_format "%Y-%m-%dT%H:%M:%S.%L"
    ssl_verify_peer false #since FluentD hosted in Private Trusted Network

    <buffer tag,time>
      @type file
      path /fluentd/buffer/s3/app-tag
      timekey_wait 5s
      timekey 5s
      chunk_limit_size 64m
      flush_at_shutdown true
      flush_thread_count 5
      total_limit_size 896m
      overflow_action block
    </buffer>
    <format>
      @type json
    </format>
    time_slice_format %Y%m%d%H%M%S
  </match>
</label>

Please advise