[bug]: [wtclient] catch-up after a prolonged tower outage starves a running node — forwarding stops, gRPC hangs, one Postgres txn left open

[Wired4ncer]

Pre-Submission Checklist

• I have searched the existing issues and believe this is a new bug.
• I am not asking a question about how to use lnd, but reporting a bug (otherwise open a discussion).

LND Version

v0.21.1-beta (both the client node and the watchtower server node)

LND Configuration

Client:

wtclient.active=true
wtclient.sweep-fee-rate=100
db.backend=postgres
db.postgres.maxconnections=25
db.postgres.timeout=0
db.postgres.channeldb-with-global-lock=false

wtclient.db was migrated into Postgres in Dec 2025 (use-native-sql=true), so the
"delete wtclient.db and restart" workaround from older threads is not available.

Tower (separate host):

watchtower.active=true
watchtower.listen=0.0.0.0:9911

Backend Version

bitcoind (client node is fully synced throughout; the backend is not implicated)

OS/Distribution

Ubuntu, x86_64, 8 vCPU / 16 GB, lnd under systemd

Bug Details & Steps to Reproduce

Two phases. Phase 1 is the known failure; phase 2 is what I think is new.

Phase 1 — tower rejects everything for a long period.
The tower rejected every state update from this client with CodePermanentFailure and
StateUpdateCodeSeqNumOutOfOrder. Client-side seqnums were frozen; WTCL=debug showed
Unable to get new tower candidate ... reason: exhausted all tower candidates on every client
type. Duration is unknown — it predates log retention on both hosts. lncli wtclient towers,
stats and remove all hung during this period. This part matches #6397 / #4420 / #6785.

Restarting lnd on the tower (not the client) cleared it. Within ~20 minutes each client
type negotiated a fresh session and uploads resumed with code=0 acks and in-order seqnums.

Phase 2 — the catch-up then took the node down.
The client began a sustained catch-up and held it for ~3 hours:

• ~100 state updates/min accepted, flat, no taper
• one new session acquired per ~10 min (consistent with the 1024-updates-per-session cap in [watchtower] [feature]: mark exhausted sessions as inactive #7045)
• sessions(acquired=…) went 202 → 219, exhausted 205 → 222 over that window
• meanwhile the node's own live backup demand was ~7 tasks/hour
  (tasks(received=255) total across 36 h of uptime, against accepted=226526)

After ~3 hours the node had effectively stopped working as a routing node, while
systemctl still reported lnd active:

Signal	Value
HTLCs forwarded, last 10 min	0
Peers	61, down from 110
pong response failure warnings	38 in 14 min
lncli getinfo	89.7 s
gRPC clients (a Django/gunicorn dashboard)	killed on their 120 s timeout, respawning
Postgres	one lnd connection idle in transaction for 30m50s
all other lnd Postgres connections	idle — no query active anywhere
CPU / iowait	~74% idle, 0–1% iowait

The last three rows are the interesting part: lnd was not blocked waiting on the database.
Nothing was executing and no backend was waiting on a lock — lnd had stopped issuing work,
while leaving one transaction open. With db.postgres.timeout=0 nothing ever times out, so the
condition does not self-clear.

Resolution: set wtclient.active=false, restart lnd. Immediately after:

Signal	Before	After
lncli getinfo	89.7 s	1.9 s
Forward/settle activity, 10 min	0	33
Peers	61	77 and climbing
Longest idle in transaction	30m50s	none
Load average	4.54	1.00

Expected Behavior

A watchtower backlog is not urgent work — the node is online and self-watching. It should be
paced or backgrounded such that it cannot starve the htlcswitch, the peer subsystem and the RPC
server. At minimum, an unbounded catch-up should not be able to leave a Postgres transaction
open indefinitely while the client makes no progress elsewhere.

Debug Information

Sanitised log shapes (client):

[INF] WTCL: (anchor) Client stats: tasks(received=255 accepted=226526 ineligible=0) sessions(acquired=219 exhausted=222)

Tower side, with WTWR=debug (120 s sample, IDs redacted):

[DBG] WTWR: Received MsgStateUpdate(seqnum=177 last_applied=176 is_complete=0 hint=<redacted>) from <session-key>
[DBG] WTWR: State update 177 accepted for <session-key>
[DBG] WTWR: Sending MsgStateUpdateReply(code=0 last_applied=177) to <session-key>

112 unique hints in that window, and a second window ~13 min later shared zero hints with
the first — so the client was sending genuinely new states, not replaying.

Also, and this may be the more actionable half: lncli wtclient towers --include_sessions
never returned — killed after 18m16s having produced no output. An earlier call was killed
at 25 s. getinfo was measured at 89.7 s while that call was outstanding, and killing the
client process did not restore responsiveness. This looks like #6660 (that reporter saw
10.9 GB RES and a 6m57s runtime, with the same trigger: data queued up while the tower was
down). It makes the one RPC you need for diagnosis unusable exactly when you need it.

Related

• [watchtower] [feature]: mark exhausted sessions as inactive #7045 — documents the 1024-updates-per-session design and the candidate loop. Note its fix
  retires a session only once all its updates are acked; with a tower rejecting everything,
  no session can ever be retired, so the fix cannot engage.
• [bug]: slow startup on watchtower setup #7121 — closest prior match (Postgres, exhausted: 2116 vs acquired: 3, lnd+postgres
  spinning), also resolved by wtclient.active=false.
• [bug]: [wtclient] Tower that died make lnd start for hours #10646 — open, v0.20.1; dead tower, wtclient towers taking minutes, tower not removable.
• lncli wtclient tower uses massive amounts of memory #6660 — lncli wtclient tower memory/latency blowup.
• [bug]: WTCL: received error code CodePermanentFailure in StateUpdateReply for seqnum=xxx #7949 — the phase-1 error; resolved upstream by wtclient session terminate (wtclient: add DeactivateTower and TerminateSession commands #8239).

Caveat on my own data: my exhausted-session count (222) is an order of magnitude below #7121's
2,116, so I am not certain the candidate loop alone explains the stall — the hung RPC may be
carrying more of it. Happy to gather more if there is something specific worth capturing before
I re-enable the client.

[Wired4ncer]

Correcting my own caveat above, and adding the number I should have led with.

I wrote that my exhausted-session count (222) was an order of magnitude below #7121's 2,116 and
that I therefore wasn't sure the candidate loop explained the stall. That comparison was wrong —
222 is the in-memory counter since the last restart, not the persisted dataset. The persisted
figure, measured on this node in July 2026:

towerclientdb_kv = 2.4 GB / ~12.0 M rows

Second-largest table in the database after channeldb_kv. #7121's reporter — the closest prior
match — had 496 MB. So this is roughly 5× the largest previously reported case, and the fit to
#7045 / #7121 is much stronger than I first said, not weaker.

This node had already hit the same code path, with a pprof stack. Three weeks before the
incident I reported here, it had an ~8-minute silent cold start. Two goroutine dumps 60 s apart
both caught lnd's main goroutine in:

newServer()
  -> wtclient.NewManager
    -> newClient
      -> getTowerAndSessionCandidates
        -> wtdb.(*ClientDB).ListClientSessions / listTowerSessions
          -> getClientSession(Body)

a single kvdb View transaction issuing serial Postgres round trips (pgx ExecPrepared, goroutine
in IO wait). The CPU profile was almost entirely I/O wait — round-trip latency, not compute.
Millions of tiny sequential queries against a 12 M-row table is an ~8 minute startup.

Why the existing cleanup never engaged. #7069 (DeleteSession) merged in March 2023 and #7045
landed the same month, so a healthy node should be retiring exhausted sessions. Neither can fire
here, because both have the same precondition: a session only becomes deletable once all of its
updates are acked by the tower. My tower rejected everything for months, so no session ever
became deletable, and the table grew without bound. The cleanup is unavailable in exactly the
failure mode that produces the growth it is meant to prevent — that feedback loop seems like the
core of it, and it would equally explain #10646, where the tower is permanently unreachable.

One more calibration point: getinfo on this node has a chronic ~2.5 s baseline (it is a busy
routing node with Postgres). So the 89.7 s I reported is ~35× its own normal, not 89 s against an
otherwise instant RPC.

A question, if a maintainer has a view. Given that remove is refused while updates are
unacked, deactivate still leaves sessions loaded at startup (#10646), and session terminate is
per-session — is there a supported way to reset a watchtower client whose sessions can never
become deletable? Concretely: is truncating towerclientdb_kv (lnd stopped, dump taken) safe, or
does lnd require the kv root buckets to pre-exist rather than recreating them on init? I have
wtclient.active=false for now and no urgency, so I would rather ask than experiment on this node —
and I am happy to gather anything specific before I re-enable the client, since some of this
evidence only exists while it is running.

[Wired4ncer]

Re-enabled the client today, deliberately, to see whether the catch-up on its own reproduces
phase 2. It did not — and the one thing I changed is which commands I ran, not the load.

lnd restarted with wtclient.active=true and the catch-up resumed immediately: ~100–125
accepted/min, flat, no taper, one session acquired per ~10 min, against live demand of
~4 tasks/hour. Same shape and same ratio as the run in the report above.

Nearly five hours in, none of the phase-2 signals appeared:

Signal	Phase 2 (reported above)	This run, +4.7 h
lncli getinfo	89.7 s	1.20–1.26 s (chronic baseline here is ~2.5 s)
Peers	61, down from 110	stable, no drop
Forwarding	0 in 10 min	normal for this node
[ERR] WTCL	rejection loop	none in 60 min
Postgres	one conn idle in transaction 30m50s	no long-lived transaction
pong response failure	38 in 14 min	none

That is well past the ~3 hours of sustained catch-up that preceded the stall the first time.

The deliberate difference: I did not call lncli wtclient towers --include_sessions at any
point during this run. In the original incident two of those calls (one killed at 25 s, one
that ran 18m16s) sat inside the stall window, and I could not separate them from the sustained
write load — both live in the same subsystem, which is why the fix removed both at once.

This run separates them, and it points at the RPC rather than the catch-up traffic. That makes
#6660 the more likely mechanism for phase 2 than the candidate loop, at least on this node.

Stating the limits plainly:

• One observation, not proof. I have not tried to reproduce the stall, and I am not going to
  deliberately hang that RPC again on this node to complete the experiment.
• The dataset is unchanged — towerclientdb_kv is still ~2.4 GB / ~12 M rows, so if the
  candidate loop contributes, the input for it is still fully present.
• Client-side counters only. I did not capture tower-side acks this time; the tower's own
  config suppresses success logging at WTWR=warn, which is worth knowing if you ask anyone
  else for both-ends evidence.

If it is useful to others landing here from a prolonged tower outage: let the catch-up run, but
avoid lncli wtclient towers while it is running. On this node a single call is enough to
leave a blocked goroutine, and killing the client does not release it.

Still open from my previous comment: is truncating towerclientdb_kv with lnd stopped a
supported reset for a client whose sessions can never become deletable, or do the kv root
buckets need to pre-exist? That is the difference between a clean reset and a restore.

One small note for anyone building monitoring on these counters: accepted is in-memory and
zeroes on restart, so a checker that alerts when it stops climbing will read a restart as a
permanent stall unless it treats a decrease as a re-baseline. That cost me 18 hours of false
alarms on my own tooling, not lnd's fault, but easy to get wrong.