SIGSEGV when using forward input plugin with low Mem_Buf_Limit #11113
Description
Bug Report
Describe the bug
When using the forward input plugin, we are able to consistently trigger a segfault when using a low Mem_Buf_Limit value. From our testing it appears to happen when Mem_Buf_Limit is less than the forward input plugin's Buffer_Max_Size value (default 6mb). EDIT: after confirming with valgrind, the bug doesnt appear to be related to exact buffer sizes, only the frequency and duration of the input pauses.
To Reproduce
seg-fault 1:
[2025/11/05 17:48:47.746440550] [ info] [input] forward.3 resume (mem buf overlimit - buf size 2053099B now below limit 4000000B)
[2025/11/05 17:48:47.748714140] [ warn] [input] forward.3 paused (mem buf overlimit - event of size 5579B exceeded limit 4000000 to 4000195B)
[2025/11/05 17:48:47.748742248] [ info] [input] pausing forward.3
[2025/11/05 17:48:47] [engine] caught signal (SIGSEGV)
#0 0xaaaad1ba66c7 in fw_conn_event() at plugins/in_forward/fw_conn.c:75
#1 0xaaaad1aeff03 in flb_engine_start() at src/flb_engine.c:1136
#2 0xaaaad1acab63 in flb_lib_worker() at src/flb_lib.c:835
#3 0xffff9af7202f in start_thread() at reate.c:442
#4 0xffff9afdbf1b in thread_start() at sysv/linux/aarch64/clone.S:79
#5 0xffffffffffffffff in ???() at ???:0
seg-fault 2:
[2025/11/05 20:07:54.252168021] [ info] [input] forward.0 resume (mem buf overlimit - buf size 1955268B now below limit 4000000B)
[2025/11/05 20:07:54.649327921] [ warn] [input] forward.0 paused (mem buf overlimit - event of size 7323B exceeded limit 4000000 to 4005721B)
[2025/11/05 20:07:54.649336291] [ info] [input] pausing forward.0
[2025/11/05 20:07:54] [engine] caught signal (SIGSEGV)
#0 0x7f8810d91e55 in ???() at ???:0
#1 0x98aca9 in receiver_recv() at plugins/in_forward/fw_prot.c:1086
#2 0x98ad5b in receiver_to_unpacker() at plugins/in_forward/fw_prot.c:1102
#3 0x98b795 in fw_prot_process() at plugins/in_forward/fw_prot.c:1291
#4 0x9831d4 in fw_conn_event() at plugins/in_forward/fw_conn.c:126
#5 0x54981a in flb_engine_start() at src/flb_engine.c:1136
#6 0x4cbe48 in flb_lib_worker() at src/flb_lib.c:904
#7 0x7f8810c8b2e9 in ???() at ???:0
#8 0x7f8810d104ff in ???() at ???:0
#9 0xffffffffffffffff in ???() at ???:0
- Steps to reproduce the problem:
- Fluent-bit config file:
[SERVICE]
Flush 1
[INPUT]
Name forward
Mem_Buf_Limit 4MB
unix_path /var/run/fluent.sock
[OUTPUT]
Name flowcounter
Match *
Unit second
[OUTPUT]
Name file
Match *
Path /home/ec2-user/fluent
- Build and run latest fluent-bit binary with debug/backtrace flags enabled:
cmake -DFLB_RELEASE=Off -DFLB_MTRACE=On -DFLB_DEV=On -DFLB_BACKTRACE=On -DFLB_DEBUG=On -DFLB_TRACE=On -DFLB_VALGRIND=On ..
make -j $(nproc)
bin/fluent-bit -c fluent.conf
- Run high load log generator script on the socket. I am using a docker container to do this like this
docker run -d --log-driver=fluentd --log-opt mode=non-blocking --log-opt fluentd-address=unix:///var/run/fluent.sock -e LOG_RATE_PER_SECOND=1000 -e LOG_SIZE_KB=7 public.ecr.aws/cssparr/loggen:latest
- Observe the segfault log message that occurs above.
Expected behavior
no segfault
Your Environment
see config and image/version used in steps to repro above