Testing with emqtt-bench: Connections stop at 3953, then time out. What's the reason? #286
Description
Client operation
./emqtt_bench conn -c 10000 -i 1 -h serverIP -u test -P test@test
Start with 2 workers, addrs pool size: 1 and req interval: 2 ms
1s connect_succ total=636 rate=634.73/sec
2s connect_succ total=1296 rate=660.00/sec
3s connect_succ total=1803 rate=507.00/sec
4s connect_succ total=2377 rate=573.43/sec
5s connect_succ total=2723 rate=346.35/sec
6s connect_succ total=3073 rate=350.00/sec
7s connect_succ total=3456 rate=383.00/sec
8s connect_succ total=3657 rate=201.00/sec
9s connect_succ total=3709 rate=52.00/sec
10s connect_succ total=3756 rate=47.00/sec
11s connect_succ total=3788 rate=32.00/sec
12s connect_succ total=3813 rate=25.00/sec
13s connect_succ total=3849 rate=36.00/sec
14s connect_succ total=3861 rate=12.00/sec
15s connect_succ total=3949 rate=88.00/sec
16s connect_succ total=3953 rate=4.00/sec
client(841): connect error - timeout
client(841): EXIT for {shutdown,timeout}
Server side: The Ubuntu 22.04 server has implemented the system tuning recommendations provided by EMQX. Checking the logs, when the client experiences connection timeouts, the server logs show:
[debug] clientid: code_bench_conn_3539568098_7782, msg: cancel_stats_timer, peername: ClientIP:49225, username: test
[debug] tag: SOCKET, clientid: code_bench_conn_2472424169_8032, msg: emqx_connection_terminated, peername: ClientIP:65194, username: test, reason: {shutdown,tcp_closed}
[debug] tag: SOCKET, clientid: code_bench_conn_1503790609_6356, msg: emqx_connection_terminated, peername: ClientIP:59565, username: test, reason: {shutdown,tcp_closed}
EMQX server details:
emqx.service - emqx daemon
Loaded: loaded (/lib/systemd/system/emqx.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2025-03-24 10:45:03 CST; 12min ago
Main PID: 2058069 (beam.smp)
Tasks: 345 (limit: 15000)
Memory: 447.5M
CPU: 3min 42.201s
CGroup: /system.slice/emqx.service
├─2058069 emqx -Bd -spp true -A 4 -IOt 4 -SDio 8 -C multi_time_warp -c true -pc unicode -e 262144 -zdbbl 8192 -Q 2097152 -P 4194304 -- -roo>
├─2058369 erl_child_setup 1048576
├─2058755 /usr/lib/emqx/lib/os_mon-2.9.1/priv/bin/memsup
├─2058756 /usr/lib/emqx/lib/os_mon-2.9.1/priv/bin/cpu_sup
├─2058767 /usr/lib/emqx/erts-14.2.5.2/bin/inet_gethost 4
└─2058768 /usr/lib/emqx/erts-14.2.5.2/bin/inet_gethost 4
Mar 24 10:45:04 codeds bash[2058069]: WARNING: Default (insecure) Erlang cookie is in use.
Mar 24 10:45:04 codeds bash[2058069]: WARNING: Configure node.cookie in /etc/emqx/emqx.conf or override from environment variable EMQX_NODE__COOKIE
Mar 24 10:45:04 codeds bash[2058069]: WARNING: NOTE: Use the same cookie for all nodes in the cluster.
Mar 24 10:45:05 codeds emqx[2058362]: EXEC: /usr/lib/emqx/erts-14.2.5.2/bin/erlexec -enable-feature maybe_expr -noinput -noshell +Bd -boot /usr/lib/emqx>
Mar 24 10:45:07 codeds bash[2058069]: Listener tcp:default on 0.0.0.0:1883 started.
Mar 24 10:45:07 codeds bash[2058069]: Listener ssl:default is NOT started due to: disabled.
Mar 24 10:45:07 codeds bash[2058069]: Listener ws:default is NOT started due to: disabled.
Mar 24 10:45:07 codeds bash[2058069]: Listener wss:default is NOT started due to: disabled.
Mar 24 10:45:08 codeds bash[2058069]: Listener http:dashboard on :18083 started.
Mar 24 10:45:08 codeds bash[2058069]: EMQX 5.8.5 is running now!
Dear experts, what could be the reason for this?
This issue was automatically translated from Chinese.