nf-tower plugin blocks JVM exit after workflow completion — missing HTTP read timeout

[adamrtalbot]

Bug report

Expected behavior and actual behavior

Expected: After a workflow completes successfully, the Nextflow JVM should exit cleanly, allowing the Azure Batch head task to terminate and release the node.

Actual: The JVM hangs indefinitely after workflow completion. The TowerClient.onFlowComplete() method blocks the main thread on an HTTP PUT call (sendHttpMessage(urlTraceComplete, ...)) to the Tower API that never receives a response. The JVM has been stuck for 35+ hours with 0% CPU. The workflow shows as COMPLETE on Seqera Platform but the Azure Batch task remains in active/running state forever.

Steps to reproduce the problem

Run any pipeline via Seqera Platform on Azure Batch. The issue is timing-dependent — it occurs when the Tower API connection becomes stale during the shutdown HTTP call. Reproduction requires a network condition where the TCP connection is established but the response is never delivered.

Program output

Last lines of Nextflow log — log goes silent after TimelineObserver, the next observer (TowerClient) never produces output:

Mar-03 00:33:40.068 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed
Mar-03 00:33:40.068 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
<EOF — no further output, no "Session destroyed", no System.exit()>

Thread state from the stuck node (via /proc, 35+ hours after completion):

TID=4474   name=java               state=S (sleeping)    ← main thread blocked
TID=4821   name=tower-logs-chec    state=S (sleeping)    ← next observer never called
TID=4645   name=HttpClient-1-Se    state=S (sleeping)    ← Java HttpClient threads alive
(Tower-thread ABSENT — sender exited cleanly, sender.join() is not the issue)

Network state from the stuck node:

ESTAB 0 0  10.0.0.4:59344 → 13.41.18.99:443  (pid=4474)   ← Tower API

No TCP keepalive timers, no retransmission timers — stale connection sitting idle for 35+ hours. Tower API health check responds 200 in 0.25s — the issue is specific to this stale connection.

Environment

• Nextflow version: 25.10.4 (build 11173)
• Java version: Amazon Corretto 21.0.10+7-LTS
• Operating system: Linux 6.8.0-1044-azure (Azure Batch)
• nf-tower plugin: 1.17.5
• nf-azure plugin: 1.20.2

Additional context

Root cause analysis:

TowerClient.onFlowComplete() calls sendHttpMessage(urlTraceComplete, req, 'PUT') to report workflow completion to the Tower API. The underlying HxClient is built with connectTimeout(60s) but no read timeout — Java HttpClient's default is infinite. When the Tower API accepts the TCP connection but never sends a response (stale/dead connection), the HTTP read blocks the main thread forever, System.exit() is never reached, and the JVM hangs indefinitely.

The shutdown observer chain is sequential with no timeout (notifyEvent() catches exceptions but cannot handle infinite blocking), so one stuck observer prevents all subsequent cleanup.

Suggested fixes:

1. Add a read/response timeout to HxClient (e.g. .timeout(Duration.ofSeconds(300)))
2. Add a bounded timeout to the shutdown HTTP call in onFlowComplete() so a stale connection can't block JVM exit
3. Consider enabling TCP keepalive to detect dead connections at the transport level
4. Close HxClient on shutdown (currently never closed)

[adamrtalbot]

Claude suggestion why this is more of an issue on Azure:

Azure's Silent Connection Drop

Azure Load Balancer / NAT Gateway has a default idle timeout of 4 minutes. When a TCP connection is idle beyond this, Azure silently removes the NAT mapping — no RST packet, no FIN, nothing. Both sides think the connection is still alive, but packets from the client go into a black hole.
Here's why this hits the Tower completion call specifically:

Timeline during TowerClient.onFlowComplete():
─────────────────────────────────────────────
1. Tower-thread sends last HTTP call to Tower API     ← connection is active
2. Tower-thread exits                                  ← connection goes idle
3. reports.publishRuntimeReports()                     ← time passes (Azure blob I/O)
4. sender.join() returns                               ← time passes
5. reports.flowComplete()                              ← time passes (Timer + Agent + Azure blob copy)
6. sendHttpMessage(urlTraceComplete, PUT)              ← reuses stale connection 💀
                                                          Azure already dropped the NAT mapping
                                                          packets go nowhere, read blocks forever

Between step 1 (last sender HTTP activity) and step 6 (completion PUT), there's a gap where the TCP connection sits idle while Azure blob operations run. If that gap exceeds 4 minutes, Azure's NAT silently kills the connection. Java HttpClient reuses the dead connection from its pool, sends the request into the void, and waits for a response that will never come.

Why Not on AWS/GCP?

Factor	Azure	AWS	GCP
NAT idle timeout	4 min (default)	350 sec (~6 min)	10 min
Silent drop (no RST)	Yes	Sometimes sends RST	Sends RST
Connection path to Tower API	Azure → AWS (cross-cloud)	AWS → AWS (same cloud)	GCP → AWS (cross-cloud)

On AWS, the Tower API is on the same cloud, so there's no cross-cloud NAT. Connections stay within AWS's network fabric, and idle timeout is longer. On GCP, the idle timeout is 10 minutes and RST packets are sent on drop, so Java's HttpClient would get an IOException (caught and handled) rather than a silent hang.

Azure is the perfect storm: short idle timeout + silent drop + cross-cloud NAT.

This Also Explains Why It's Intermittent

It only triggers when the gap between the sender thread's last HTTP call and the completion PUT exceeds 4 minutes. If reports.flowComplete() is fast (small/no reports, quick blob ops), the connection stays warm and the PUT succeeds. If blob operations are slow that day, the gap grows past 4 minutes and the connection dies silently.

[bentsherman]

Possibly related: #6840

[bentsherman]

The above PR is included since 26.02.0-edge, so it might be worth trying that version and seeing if the error persists