[Q&A] Upgrading fluent-package 5.0.7 to 6.0.0, hangs on startup with rebuilding weight array lost_weight=0
#5101
Replies: 4 comments 6 replies
-
|
Hmm, I can't reproduce your issue... If you can introduce https://github.com/fluent-plugins-nursery/fluent-stackprof, If we could see that dump file, we might be able to figure something out... |
Beta Was this translation helpful? Give feedback.
-
|
I'm trying to create this file, but without success. I ran to the top of my /etc/fluent/fluentd.conf, even before the section that includes the problem section. When I start fluentd (I'm presently only doing so via systemctl, to avoid the possible duplicate-fluentd instances issue in @daipom 's note below), the behavior is exactly as before -- the systemctl CLI hangs for 3-4 minutes, then fails; in the background, systemd kill-9's that instance and tries again. Running the stackprof command, to attempt to connect to fluentd and get the info you're looking for, fails: Presumably fluentd hasn't gotten far enough in the startup process to begin sending that profiling data. The startup logs look identical to before, the last line is the "adding forwarding server" one (the "rebuilding weight array" line isn't shown when starting via systemctl, since debug logging isn't enabled in that case). I'll try to do some straces and see if that yields anything helpful, and also will try to spin up a test box where I can do an even more narrowed-down test case. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your report! Does this mean the logs only show up to here? The fact that logs from other Input plugins, like the following, are not appearing suggests that it is highly likely the process is hanging somewhere during the initialization of
Having minimal reproduction steps would be really helpful. |
Beta Was this translation helpful? Give feedback.
-
Looks like multiple Fluentd instances are running simultaneously. |
Beta Was this translation helpful? Give feedback.
-
|
We need more detailed steps to reproduce the issue. |
Beta Was this translation helpful? Give feedback.
-
|
That may have been an artifact of running fluentd from the CLI, rather than from systemd as I normally do. (systemd may have been trying to restart it in the background.) To confirm, I explicitly killed anything fluentd-related, then tried a startup from systemctl. In this case, the terminal in which I ran (and so on, after 3-4 minutes the startup attempt fails, systemd kill-9s it, and another startup is attempted.) |
Beta Was this translation helpful? Give feedback.
-
|
I see. |
Beta Was this translation helpful? Give feedback.
-
|
We need the steps to reproduce. For example, does this issue occur only with the following <match gateway_access.*>
@type forward
@id out_util
send_timeout 10s
recover_wait 10s
hard_timeout 10s
transport tls
tls_cert_path /etc/fluent/certs/fluentd.crt
tls_verify_hostname false
keepalive true
<buffer>
@type file
path /mnt/log/fluent/gateway
flush_mode interval
flush_interval 5s
retry_forever true
retry_type periodic
retry_wait 10s
chunk_limit_size 256MB
queued_chunks_limit_size 10
overflow_action throw_exception
</buffer>
<server>
name util
host 10.x.x.x
port 24224
weight 60
</server>
<security>
self_hostname my-hostname-here
shared_key (omitted)
</security>
</match>Also, does deleting some options improve the situation, such as |
Beta Was this translation helpful? Give feedback.
-
|
I'm still trying to narrow this down - I don't think the actual problem is in the above any more. The main config includes several other files, including the one I snipped above. One of those others takes some incoming events (JSON, over TCP on an internal port), runs them through a When I remove that file, instead of hanging at this spot, fluentd starts up normally. So the real issue is somewhere over there, in all likelihood. I'm still testing, but didn't want anyone else to think I'd just dropped this entirely. |
Beta Was this translation helpful? Give feedback.
-
|
Upgrading My apologies for wasting everyone's time. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
What is the problem?
I'm trying to bump a test server from fluent-package 5.0.7 to 6.0.0. The only change I've made to the config as part of the version jump, is the include change I discovered was needed in another Q&A. That change shouldn't directly impact this, though.
Starting up fluentd via systemctl times out after a couple minutes. Attempting to start fluentd manually, the startup hangs at
rebuilding weight array lost_weight=0, the fluentd process consumes 100% CPU, and fluentd never processes any log events. In either case, akill -9is needed to stop the running fluentd process.Describe the configuration of Fluentd
This actually is a follow-up to #5100 . This is one of the files I'd omitted from the configs there :) This part of the config is fairly standalone, though.
The reason I'm doing a
copy, with only one action below it, is because this config is generated by Ansible, and some of the hosts we manage do have other<store>clauses included. (I wouldn't expect a copy, with only one destination, to be inherently problematic. Again, this works with fluent-package 5.x.)I did try removing the "weight" value from the server clause, this didn't help. Actual server IP and security credentials removed from below, of course.
Describe the logs of Fluentd
From starting fluentd manually on the CLI with
sudo /opt/fluent/bin/fluentd --verbose:(and then it just sits there eating up CPU, and ultimately requires a
kill -9to stopFrom the above, here's a
psof that startup attempt, after about an hour:So, after about an hour, the fluentd process has taken about an hour's worth of CPU. Similar logs and behavior seen when trying to start fluentd via systemctl.
Environment
Beta Was this translation helpful? Give feedback.
All reactions