Omni node is sending out lots of request

[ggwpez]

(preface: I am not a network person)

I was running the omni node on a server and got this from the admin:

[..] there is a process on the server sending connection attempts out at rates of thousands a minute to private Lan addresses not on the server network, so un-routable and filling the firewall logs - going to ports 30333 and 30334.

Looks like it is polkadot-omni-n and it is snowballing to 100 attempts a second. Changed firewall rule to send REJECT Network Unreachable reply rather than just silently drop the packets so maybe the process will learn although I know little about what it is attempting to do - Oliver might be able to enlighten us but could be a bug that might want squishing?

I am running omni node release binaries like anybody else would version polkadot-omni-node 1.19.2-2caeef482a4. It does not seem to sync the Asset hub? Not finding peers?
But I am running with the correct spec this time, command is here: https://github.com/paritytech/ahm-dryrun/blob/5af2f31dad9ff97d074c03d0a140f8dfb7c88add/justfiles/node.justfile#L55-L67

I also checked that it seems to come from the omni node, at least for port 30333.

logs:

Details

2025-10-10 13:48:43 [Relaychain] maintain txs=(6, 0) a=2 i=2 views=[(28136771, 2, 0), (28136772, 0, 0)] event=Finalized { hash: 0xabddd22fe6a6e721e94e5a15670094c6122075dd1d32a3029370ef642b2f0e43, tree_route: [] } duration=346.796µs
2025-10-10 13:48:44 [Parachain] ⏩ Block history, #0 (0 peers), best: #9889911 (0xd723…2e21), finalized #9889911 (0xd723…2e21), ⬇ 0 ⬆ 0    
2025-10-10 13:48:44 [Relaychain] 💤 Idle (13 peers), best: #28136772 (0xb921…33c2), finalized #28136769 (0xabdd…0e43), ⬇ 204.8kiB/s ⬆ 280.3kiB/s    
2025-10-10 13:48:47 [Relaychain] maintain txs=(6, 0) a=2 i=1 views=[(28136771, 4, 0), (28136772, 2, 0)] event=Finalized { hash: 0xe3874436dc4d6e9e082f91674e4510dd81617245291a46444a20be14bf12a182, tree_route: [] } duration=216.108µs
2025-10-10 13:48:48 [Relaychain] 🏆 Imported #28136773 (0xb921…33c2 → 0x556f…02ef)    
2025-10-10 13:48:48 [Parachain] Update at relay chain block #28136773 (0x556f…02ef) - backed: #9916219 (0xffbc…30ed), included: #9916218 (0xe0fa…80fd)    
2025-10-10 13:48:48 [Relaychain] maintain txs=(6, 0) a=2 i=2 views=[(28136771, 4, 0), (28136773, 0, 0)] event=NewBestBlock { hash: 0x556fbe2f6ca65ef3dad98d7743dc783958f350a7d0696e396561b00d599102ef, tree_route: None } duration=2.808331ms
2025-10-10 13:48:48 [Relaychain] 🆕 Imported #28136773 (0xb921…33c2 → 0xdc81…33b0)    
2025-10-10 13:48:48 [Parachain] Update at relay chain block #28136773 (0xdc81…33b0) - backed: #9916219 (0xffbc…30ed), included: #9916218 (0xe0fa…80fd)    
2025-10-10 13:48:48 [Relaychain] 🆕 Imported #28136773 (0xb921…33c2 → 0x63a1…1dba)    
2025-10-10 13:48:48 [Parachain] Update at relay chain block #28136773 (0x63a1…1dba) - backed: #9916219 (0xffbc…30ed), included: #9916218 (0xe0fa…80fd)    
2025-10-10 13:48:49 [Parachain] ⏩ Block history, #0 (0 peers), best: #9889911 (0xd723…2e21), finalized #9889911 (0xd723…2e21), ⬇ 0 ⬆ 0    
2025-10-10 13:48:49 [Relaychain] 💤 Idle (13 peers), best: #28136773 (0x556f…02ef), finalized #28136770 (0xe387…a182), ⬇ 476.4kiB/s ⬆ 616.9kiB/s    
2025-10-10 13:48:54 [Relaychain] 🏆 Imported #28136774 (0x556f…02ef → 0x9bc6…700a)    
2025-10-10 13:48:54 [Relaychain] maintain txs=(6, 0) a=2 i=3 views=[(28136771, 4, 0), (28136774, 0, 0)] event=NewBestBlock { hash: 0x9bc6c3f153072bd4ad98fa83e9887298b8b22c289a31326c160da0967f68700a, tree_route: None } duration=571.547µs
2025-10-10 13:48:54 [Parachain] Update at relay chain block #28136774 (0x9bc6…700a) - backed: #9916220 (0x1398…564e), included: #9916219 (0xffbc…30ed)    
2025-10-10 13:48:54 [Parachain] ⏩ Block history, #0 (0 peers), best: #9889911 (0xd723…2e21), finalized #9889911 (0xd723…2e21), ⬇ 0 ⬆ 0    
2025-10-10 13:48:54 [Relaychain] 💤 Idle (13 peers), best: #28136774 (0x9bc6…700a), finalized #28136770 (0xe387…a182), ⬇ 274.4kiB/s ⬆ 300.0kiB/s    
2025-10-10 13:48:59 [Relaychain] maintain txs=(4, 0) a=1 i=2 views=[(28136774, 0, 0)] event=Finalized { hash: 0x90ea912eb454a54a8f291660d52f5e5e615107d3d8d9189e9a2b7cb86f08f621, tree_route: [] } duration=414.95µs
2025-10-10 13:48:59 [Relaychain] 🔍 Discovered new external address for our node: /ip4/78.137.198.97/tcp/30334/ws/p2p/12D3KooWQ7V4UTzbr7845uahYNiq5xsh1ionLzvAAtjrwgLmzU8D    
2025-10-10 13:48:59 [Parachain] ⏩ Block history, #0 (0 peers), best: #9889911 (0xd723…2e21), finalized #9889911 (0xd723…2e21), ⬇ 3.2kiB/s ⬆ 0.2kiB/s    
2025-10-10 13:48:59 [Relaychain] 💤 Idle (13 peers), best: #28136774 (0x9bc6…700a), finalized #28136771 (0x90ea…f621), ⬇ 626.6kiB/s ⬆ 536.4kiB/s    
2025-10-10 13:49:00 [Relaychain] 🏆 Imported #28136775 (0x9bc6…700a → 0x703e…02b6)    
2025-10-10 13:49:00 [Relaychain] maintain txs=(4, 0) a=1 i=3 views=[(28136775, 0, 0)] event=NewBestBlock { hash: 0x703e9222dc86c5069e5765b0bb93c0a6fb1c22504e78023ee4d2a541cd5b02b6, tree_route: None } duration=494.37µs
2025-10-10 13:49:00 [Parachain] Update at relay chain block #28136775 (0x703e…02b6) - backed: #9916221 (0xc733…ca72), included: #9916220 (0x1398…564e)    
2025-10-10 13:49:03 [Relaychain] maintain txs=(0, 0) a=1 i=2 views=[(28136775, 0, 0)] event=Finalized { hash: 0x556fbe2f6ca65ef3dad98d7743dc783958f350a7d0696e396561b00d599102ef, tree_route: [0xb9219764774b8c6bd4935a8c4eaa07cfe498a2dbae8ce5079cceda1dcce733c2] } duration=265.803µs
2025-10-10 13:49:04 [Parachain] ⏩ Block history, #0 (0 peers), best: #9889911 (0xd723…2e21), finalized #9889911 (0xd723…2e21), ⬇ 0 ⬆ 0    
2025-10-10 13:49:04 [Relaychain] 💤 Idle (13 peers), best: #28136775 (0x703e…02b6), finalized #28136773 (0x556f…02ef), ⬇ 3.5MiB/s ⬆ 378.4kiB/s    
2025-10-10 13:49:06 [Relaychain] 🏆 Imported #28136776 (0x703e…02b6 → 0x2eef…e923)    
2025-10-10 13:49:06 [Relaychain] maintain txs=(0, 0) a=1 i=3 views=[(28136776, 0, 0)] event=NewBestBlock { hash: 0x2eefabaff2704fc7fd7e5d7519cdf9c0544ff8de5e938c26d20a19f8ad46e923, tree_route: None } duration=390.883µs
2025-10-10 13:49:06 [Parachain] Update at relay chain block #28136776 (0x2eef…e923) - backed: #9916222 (0xaaf7…e462), included: #9916221 (0xc733…ca72)    
2025-10-10 13:49:09 [Parachain] ⏩ Block history, #0 (0 peers), best: #9889911 (0xd723…2e21), finalized #9889911 (0xd723…2e21), ⬇ 0 ⬆ 0    
2025-10-10 13:49:09 [Relaychain] 💤 Idle (13 peers), best: #28136776 (0x2eef…e923), finalized #28136773 (0x556f…02ef), ⬇ 1.8MiB/s ⬆ 352.4kiB/s    
2025-10-10 13:49:12 [Relaychain] 🏆 Imported #28136777 (0x2eef…e923 → 0x03f5…2073)    
2025-10-10 13:49:12 [Parachain] Update at relay chain block #28136777 (0x03f5…2073) - backed: #9916223 (0xcfe5…f2a9), included: #9916222 (0xaaf7…e462)    
2025-10-10 13:49:12 [Relaychain] maintain txs=(0, 0) a=1 i=4 views=[(28136777, 0, 0)] event=NewBestBlock { hash: 0x03f5dc3057ff5973c52c0afbca850b44324391778d5dcabc79e6b23461152073, tree_route: None } duration=419.997µs
2025-10-10 13:49:12 [Relaychain] ♻️   Reorg on #28136777,0x03f5…2073 to #28136777,0x2a3e…019c, common ancestor #28136776,0x2eef…e923    
2025-10-10 13:49:12 [Relaychain] 🏆 Imported #28136777 (0x2eef…e923 → 0x2a3e…019c)    
2025-10-10 13:49:12 [Parachain] Update at relay chain block #28136777 (0x2a3e…019c) - backed: #9916223 (0xcfe5…f2a9), included: #9916222 (0xaaf7…e462)    
2025-10-10 13:49:12 [Relaychain] maintain txs=(0, 0) a=2 i=4 views=[(28136777, 0, 0), (28136777, 0, 0)] event=NewBestBlock { hash: 0x2a3eccc09ace035a4401ae09b2a80f0e85be8e3a7e453a6e4e6f18370bcc019c, tree_route: Some(TreeRoute { route: [HashAndNumber { number: 28136777, hash: 0x03f5dc3057ff5973c52c0afbca850b44324391778d5dcabc79e6b23461152073 }, HashAndNumber { number: 28136776, hash: 0x2eefabaff2704fc7fd7e5d7519cdf9c0544ff8de5e938c26d20a19f8ad46e923 }], pivot: 1 }) } duration=464.725µs
2025-10-10 13:49:14 [Parachain] ⏩ Block history, #0 (0 peers), best: #9889911 (0xd723…2e21), finalized #9889911 (0xd723…2e21), ⬇ 6.6kiB/s ⬆ 0.5kiB/s    
2025-10-10 13:49:14 [Relaychain] 💤 Idle (13 peers), best: #28136777 (0x2a3e…019c), finalized #28136773 (0x556f…02ef), ⬇ 1.9MiB/s ⬆ 477.8kiB/s    
2025-10-10 13:49:15 [Relaychain] maintain txs=(0, 0) a=2 i=3 views=[(28136777, 0, 0), (28136777, 0, 0)] event=Finalized { hash: 0x9bc6c3f153072bd4ad98fa83e9887298b8b22c289a31326c160da0967f68700a, tree_route: [] } duration=274.407µs
2025-10-10 13:49:16 [Relaychain] Successfully persisted AddrCache on disk    
2025-10-10 13:49:18 [Relaychain] 🏆 Imported #28136778 (0x2a3e…019c → 0x0016…1d9b)    
2025-10-10 13:49:18 [Parachain] Update at relay chain block #28136778 (0x0016…1d9b) - backed: #9916224 (0x532f…5ffb), included: #9916223 (0xcfe5…f2a9)    
2025-10-10 13:49:18 [Relaychain] maintain txs=(0, 0) a=2 i=4 views=[(28136777, 0, 0), (28136778, 0, 0)] event=NewBestBlock { hash: 0x0016f79f4b7f602397d5b1e6f99601a8d3e01e3d7c58fd9fd7d7cb80cb981d9b, tree_route: None } duration=400.718µs
2025-10-10 13:49:19 [Parachain] ⏩ Block history, #0 (0 peers), best: #9889911 (0xd723…2e21), finalized #9889911 (0xd723…2e21), ⬇ 3.0kiB/s ⬆ 0.2kiB/s    
2025-10-10 13:49:19 [Relaychain] 💤 Idle (13 peers), best: #28136778 (0x0016…1d9b), finalized #28136774 (0x9bc6…700a), ⬇ 1.5MiB/s ⬆ 405.2kiB/s    
2025-10-10 13:49:24 [Relaychain] maintain txs=(0, 0) a=2 i=3 views=[(28136777, 0, 0), (28136778, 0, 0)] event=Finalized { hash: 0x703e9222dc86c5069e5765b0bb93c0a6fb1c22504e78023ee4d2a541cd5b02b6, tree_route: [] } duration=425.315µs
2025-10-10 13:49:24 [Relaychain] 🏆 Imported #28136779 (0x0016…1d9b → 0xed0d…5514)    
2025-10-10 13:49:24 [Parachain] Update at relay chain block #28136779 (0xed0d…5514) - backed: #9916225 (0x9437…8b1c), included: #9916224 (0x532f…5ffb)    
2025-10-10 13:49:24 [Relaychain] maintain txs=(0, 0) a=2 i=4 views=[(28136777, 0, 0), (28136779, 0, 0)] event=NewBestBlock { hash: 0xed0ddbb4229eb8099156a1e409a38cdf0c2a9a088e7930212c4cd592545e5514, tree_route: None } duration=389.531µs
2025-10-10 13:49:24 [Parachain] ⏩ Block history, #0 (0 peers), best: #9889911 (0xd723…2e21), finalized #9889911 (0xd723…2e21), ⬇ 0 ⬆ 0    
2025-10-10 13:49:24 [Relaychain] 💤 Idle (13 peers), best: #28136779 (0xed0d…5514), finalized #28136775 (0x703e…02b6), ⬇ 670.5kiB/s ⬆ 611.7kiB/s    
2025-10-10 13:49:28 [Relaychain] maintain txs=(0, 0) a=2 i=3 views=[(28136777, 0, 0), (28136779, 0, 0)] event=Finalized { hash: 0x2eefabaff2704fc7fd7e5d7519cdf9c0544ff8de5e938c26d20a19f8ad46e923, tree_route: [] } duration=262.007µs
2025-10-10 13:49:29 [Parachain] ⏩ Block history, #0 (0 peers), best: #9889911 (0xd723…2e21), finalized #9889911 (0xd723…2e21), ⬇ 0 ⬆ 0    
2025-10-10 13:49:29 [Relaychain] 💤 Idle (13 peers), best: #28136779 (0xed0d…5514), finalized #28136776 (0x2eef…e923), ⬇ 233.1kiB/s ⬆ 357.2kiB/s    
2025-10-10 13:49:31 [Relaychain] 🏆 Imported #28136780 (0xed0d…5514 → 0xa43b…9b50)    
2025-10-10 13:49:31 [Parachain] Update at relay chain block #28136780 (0xa43b…9b50) - backed: #9916226 (0x050e…f085), included: #9916225 (0x9437…8b1c)    
2025-10-10 13:49:31 [Relaychain] maintain txs=(0, 0) a=2 i=4 views=[(28136777, 0, 0), (28136780, 0, 0)] event=NewBestBlock { hash: 0xa43bd50e6d6b8f9e03fae2f29a0cbd7caf564165b918933c29878c1dea679b50, tree_route: None } duration=463.393µs
2025-10-10 13:49:32 [Relaychain] maintain txs=(0, 0) a=1 i=3 views=[(28136780, 0, 0)] event=Finalized { hash: 0x2a3eccc09ace035a4401ae09b2a80f0e85be8e3a7e453a6e4e6f18370bcc019c, tree_route: [] } duration=509.894µs
2025-10-10 13:49:34 [Parachain] ⏩ Block history, #0 (0 peers), best: #9889911 (0xd723…2e21), finalized #9889911 (0xd723…2e21), ⬇ 13.5kiB/s ⬆ 0.9kiB/s    
2025-10-10 13:49:34 [Relaychain] 💤 Idle (13 peers), best: #28136780 (0xa43b…9b50), finalized #28136777 (0x2a3e…019c), ⬇ 583.5kiB/s ⬆ 565.1kiB/s    
2025-10-10 13:49:36 [Relaychain] 🏆 Imported #28136781 (0xa43b…9b50 → 0x11f7…6bea)    
2025-10-10 13:49:36 [Parachain] Update at relay chain block #28136781 (0x11f7…6bea) - backed: #9916227 (0xbcc2…7ea8), included: #9916226 (0x050e…f085)    
2025-10-10 13:49:36 [Relaychain] maintain txs=(0, 0) a=1 i=4 views=[(28136781, 0, 0)] event=NewBestBlock { hash: 0x11f7c7b2d1d07f2c17d2a8c0f341a596e56ca1710d01f43346a3d96ff6e06bea, tree_route: None } duration=478.106µs
2025-10-10 13:49:39 [Parachain] ⏩ Block history, #0 (0 peers), best: #9889911 (0xd723…2e21), finalized #9889911 (0xd723…2e21), ⬇ 0.2kiB/s ⬆ 0.2kiB/s    
2025-10-10 13:49:39 [Relaychain] 💤 Idle (13 peers), best: #28136781 (0x11f7…6bea), finalized #28136777 (0x2a3e…019c), ⬇ 442.7kiB/s ⬆ 284.5kiB/s    
^C2025-10-10 13:49:39 [Parachain] Essential task `txpool-background` failed. Shutting down service.    
2025-10-10 13:49:39 [Relaychain] Essential task `txpool-background` failed. Shutting down service.    
error: Interrupted by SIGINT

[alexggh]

CC: @paritytech/networking

[bkchr]

Please run with -lsub-libp2p=trace for some minutes and provide the log.

[lexnv]

This litep2p PR would make sure we are no longer dialing private addresses:

• manager: Expose option to forbid dialing private IP addresses litep2p#412
  This will likely be included in a release once we finish the SCC work.

For example, some peers are wrongfully reporting back the loopback addresses in the identify messages:

// This might be considered an attempt at port-scanning by some cloud providers:

2025-10-10T14:25:24.603900Z  INFO subp2p_explorer_core::libp2p: Polling libp2p swarm events and commands
2025-10-10T14:25:24.603963Z  INFO subp2p_explorer_core::libp2p: Polled command: Some(AddKnownAddress { peer_id: PeerId("12D3KooWDwS1hHLtuG9ojpYt9kYRr46geYyxyAbLMUS1pEKkxdqn"), address: /ip4/127.0.0.1/tcp/30331 })

The backend attempts to dial peers to establish connections; this is a side-effect of having no peers connected on the parachain node. Providing back a REJECT Network Unreachable would decrease the internal score of the dialed address. This will eventually mark the address as stale, only if there are other addresses.

I was able to connect to the following bootnode from the chainspec using https://github.com/lexnv/subp2p-explorer/tree/main/bench-cli:

RUST_LOG=info cargo run -- discovery --genesis 68d56f15f85d3136970ec16946040bc1752654e906147f7e43e9d539d7c3de2f --bootnodes /dns/statemint-bootnode.turboflakes.io/tcp/30315/p2p/12D3KooWL8CyLww3m3pRySQGGYGNJhWDMqko3j5xi67ckP7hDUvo --backend-type litep2p

Please run with -lsub-libp2p=trace for some minutes and provide the log.

Indeed we need a bit more logs 🙏

[ggwpez]

I did not make the mental connection earlier, but both my routers restarted multiple times when I ran it locally. I think it was the node.

10 minute logs with trace on RC and Para node lz4 zipped: https://drive.google.com/file/d/1P-3k7NWQ7uEFSuqibtjpSKkbhuWJMRcm

I can try with your branch @lexnv nvm, i thought it was an SDK branch

I will try to turn off litep2p then, since i need the node synced for AHM Polkadot testing.