out_file: By symlink_path, some symlinks are not created with bulk inputs

[tty061]

Describe the bug

We are using the symlink_path setting with placeholders of the out_file plugin. However, symlinks are not created as expected for all buffer chunks (the latest chunk for each chunk key). In practice, when data containing multiple metadata is input at once (e.g., records with different myid values that chunk keys includes), only one symlink is created, and no others are created thereafter.

To Reproduce

I used a docker image fluentd:v1.19.0-1.0 in the experiment.

1. Run fluentd with configs below.
2. Input multiple data at once: echo -e '{"myid":"aa","value":"111"}\n{"myid":"bb","value":"222"}' >> /tmp/test.log
3. data.bb.new.log (the symlink for bb) is only created. /fluentd/log/buffer/ has two buffer files and two buffer meta files.
4. Input data with myid=aa: echo -e '{"myid":"aa","value":"333"}' >> /tmp/test.log
5. data.aa.new.log (the symlink for aa) is not created.

Configure file:

<source>
  @type  tail
  path /tmp/test.log
  tag test
  <parse>
    @type json
  </parse>
</source>

<match test>
  @type file
  @id   output1
  path         /fluentd/log/data.${myid}.*.log
  symlink_path /fluentd/log/data.${myid}.new.log
  append       true
  <buffer myid,time>
    @type file
    timekey 1d
    timekey_wait 1m
    path /fluentd/log/buffer
  </buffer>
</match>

Expected behavior

A symlink is created for the latest chunk for each chunk key.

Your Environment

- Fluentd version: 1.19.0
- Package version: Docker image `fluentd:v1.19.0-1.0`
- Operating system:
- Kernel version:

(in actual operation: td-agent 4.3.2 fluentd 1.14.6 (c0f48a0080550eff6aa6fa19d269e480684e7a45))

Your Configuration

Specified in “To Reproduce” section.

Your Error Log

N/A

Additional context

I tried to trace the relevant code.

SymlinkBufferMixin#metadata is implemented to hold a single @latest_metadata. It seems that this is held to use in the next generate_chunk call. I suspect that metadata might be called with different parameters before generate_chunk is called.

fluentd/lib/fluent/plugin/out_file.rb

Lines 75 to 84 in ed15edc

	def metadata(timekey: nil, tag: nil, variables: nil)
	metadata = super
	@latest_metadata ||= new_metadata(timekey: 0)
	if metadata.timekey && (metadata.timekey >= @latest_metadata.timekey)
	@latest_metadata = metadata
	end
	metadata
	end

[daipom]

Sorry for my late response. Thanks for your report.
It seems to be an old bug of Fluentd.

symlink_path supports placeholders in the following PR:

• out_file: Support placeholders in symlink_path parameter #2254

but it should consider @latest_metadata as well.

[daipom]

The following works, but the bulk inputs referred to in this issue don't work.

<source>
  @type sample
  tag test
  sample {"myid":"aa","value":"111"}
</source>

<source>
  @type sample
  tag test
  sample {"myid":"bb","value":"222"}
</source>

[daipom]

fluentd/lib/fluent/plugin/out_file.rb

Lines 94 to 100 in 35f390c

	def generate_chunk(metadata)
	chunk = super
	# "symlink" feature is to link from symlink_path to the latest file chunk. Records with latest
	# timekey will be appended into that file chunk. On the other side, resumed file chunks might NOT
	# have timekey, especially in the cases that resumed file chunks are generated by Fluentd v0.12.
	# These chunks will be enqueued immediately, and will be flushed soon.
	if chunk.metadata == @latest_metadata