[bug]: LND crashed on postgres "out of shared memory" error

[IanPasteur]

Background

I run LND with PostgreSQL backend on a virtual server on server hardware. There is 4 GB RAM dedicated to LND and PostgreSQL only (bitcoind runs on another virtual server). I have about 8 channels opened and there is not much routing nor load at all.

Since update to 18.5-beta day before yesterday (not sure if it is related), LND crashed twice with this error:

[CRT] SQLB: Caught unhandled error: ERROR: out of shared memory (SQLSTATE 53200)
[INF] SQLB: Sending request for shutdown
[ERR] DISC: Unable to determine if node **** is advertised: unknown postgres error: ERROR: out of shared memory (SQLSTATE 53200)

This "out of memory" error is strange as there should be enough memory available (there is only 25 % of 4 GB memory used in average). Postgres memory configuration follows:

shared_buffers = 512MB
max_connections = 64  # RTL does not work well if set to lower value
work_mem = 8MB

maintenance_work_mem = 64MB  # should not be relevant for this issue
effective_cache_size = 2GB  # only an estimation for query planner

The whole database size is currently 270 MB.

There could be some issue with the db locks, as I found these errors in the postgres log:

ERROR:  out of shared memory
HINT:  You might need to increase "max_pred_locks_per_transaction".
STATEMENT:  SELECT id FROM channeldb_kv WHERE parent_id IS NULL AND key=$1 AND value IS NULL

I also found another error in postgres log and I am not sure if it is related:

ERROR:  could not serialize access due to read/write dependencies among transactions
DETAIL:  Reason code: Canceled on identification as a pivot, during write.
HINT:  The transaction might succeed if retried.
STATEMENT:  INSERT INTO channeldb_kv (key, value, parent_id) VALUES($1, $2, $3) ON CONFLICT (key, parent_id) WHERE parent_id IS NOT NULL DO UPDATE SET value=$2 WHERE channeldb_kv.value IS NOT NULL

Similar errors were mentioned in #8049 and it should be fixed already, so I am confused.

Your environment

• lnd Version: 0.18.5-beta
• Linux 6.1.0-31-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.128-1 (2025-02-07) x86_64 GNU/Linux
• Bitcoin Core version v28.1.0
• Dedicated virtual server for LND and PostgreSQL with 4 GB RAM and 4 vCPU. Server hardware (Xeon CPU, ECC RAM, SSD drives with RAID, ...)

Steps to reproduce

I will provide more details if I would be able to discover some steps to reproduce.

Expected behaviour

LND with postgres does not consume nor demand so much RAM with such low load.

Actual behaviour

Postgres returns "out of shared memory" and LND crashes.

[Roasbeef]

Is it able to run consistently with this extra section in the config?

max_locks_per_transaction = 128
max_pred_locks_per_transaction = 1024

The shared memory error here is actually complaining about there not being enough locks in a given transaction.

[DerEwige]

Is it able to run consistently with this extra section in the config?

max_locks_per_transaction = 128
max_pred_locks_per_transaction = 1024

The shared memory error here is actually complaining about there not being enough locks in a given transaction.

afaik it is the other way around

it is out of shared memory, because lnd uses 50 connections to the DB per default

lnd/sample-lnd.conf

Line 1573 in 27440e8

; db.postgres.maxconnections=50

so especially during bootstraping of the graph there are a lot of parallel transactions running and

max_locks_per_transaction = 128
max_pred_locks_per_transaction = 1024

will only increase the memory usage

postgress's best practice is for one client not to use more than 2x[number of cores] connections
so if your postgress sql has 4 cores, lnd should not use more than 8 connections

futhermore lnd uses
Serializable isolation

Which is the savest, but causes the most conflicting transactions, so a minimum amount of parallel transactions should be used

[IanPasteur]

Thank you both for the reply 🙌

I will try to reduce db.postgres.maxconnections to 8, then?

Is there some documentation about recommended lnd and postgresql parameters? I could't find any on github or in the docs

[djkazic]

@IanPasteur the error is not because of the connection count. It is because of the locks configuration, please raise both and restart postgres to see if the issue is resolved. The maxconnections value of 50 is just an upper limit and is typically not hit with KVDB access patterns.

Even if you reduce the connections count you will still encounter that shared memory error eventually with a burst of DB activity.

The serialize errors are expected and benign -- they are an artifact of the KVDB schema we use to store data in postgres and will be phased out once SQL schema migration paths are available.

[IanPasteur]

Thanks for clearing this out.

I encountered higher connection count while I am browsing in RTL web interface. So I downscale db.postgres.maxconnections to 16 for lnd.

PostgreSQL is configured currently this way:

max_connections = 24
shared_buffers = 512MB
work_mem = 8MB
max_locks_per_transaction = 128
max_pred_locks_per_transaction = 1024

I will let you know how it works in few days.

PS: Are these tuning parameters recommended somewhere? Should I open a PR to LND docs?

[saubyk]

Are these tuning parameters recommended somewhere?

here's a gist authored by @djkazic
https://gist.github.com/djkazic/526fa3e032aea9578997f88b45b91fb9

[saubyk]

moving this thread to a discussion instead. Can open an issue, if there is something to fix

[DerEwige]

here's a gist authored by @djkazic https://gist.github.com/djkazic/526fa3e032aea9578997f88b45b91fb9

It is important to note that those tests were conducted on a PostgreSQL setup with 6 cores, 12 threads, and 32GB of RAM, with 6GB of shared memory allocated in PostgreSQL settings.
These settings are not applicable for a machine with 4 cores and 4GB of RAM.

One cannot just throw around these numbers for someone using a Raspberry Pi as their PostgreSQL server.

Breaking It Down

Each connection an application makes to PostgreSQL can hold one active transaction at a time.
Some of these transactions can use multiple parallel workers, depending on:

• Number of active connections
• max_parallel_workers_per_gather (workers per query)
• max_parallel_workers (total parallel workers available)
• Other cost-based planner decisions

However, having more parallel workers than the system can actually utilize is pointless. It won’t improve performance but can increase lock contention, which may lead to longer transaction durations and higher conflict rates. This results in more transaction rollbacks, requiring application retries, ultimately decreasing overall performance.

Optimal Connection Limits

There is no single "best" number of connections for every workload, but for a high-conflict workload, running significantly more connections than CPU cores can handle is counterproductive.

A widely accepted guideline (not a strict rule) is to keep active connections around 2× the number of cores/threads.
Beyond that point, context switching overhead and transaction conflicts typically outweigh any performance benefits.

Additionally, most performance tests and online benchmarks assume the default isolation level Read Committed, which is optimized for parallel workloads.

The Impact of Serializable Isolation

LND uses Serializable isolation, which is stricter than the default Read Committed level.
Serializable ensures transactions execute as if they happened in a serial order, preventing anomalies but increasing rollback rates when conflicts occur.

In high-contention workloads, Serializable transactions frequently abort and require retries, leading to wasted work. This makes Serializable a poor choice for workloads with heavy parallel execution and frequent write conflicts.

For a workload like this, tuning max_locks_per_transaction and max_pred_locks_per_transaction can help reduce lock exhaustion, but this does not prevent transaction conflicts—it only prevents PostgreSQL from running out of locks.

Conclusion

In my opinion, increasing max_locks_per_transaction and max_pred_locks_per_transaction only addresses the symptoms of the issue and is only effective if you have sufficient memory to support it. The more sustainable solution is to decrease the number of active connections to a reasonable amount. By doing so, you reduce the number of parallel transactions, which directly addresses the root cause of the problem—excessive lock contention—without negatively impacting performance. This approach helps maintain a balance between system resources and workload demands, especially in environments with limited hardware.

[DerEwige]

here is another example of a big node having problems because LND fires to many queries during startup and they have to kill some by hand on the DB server to prevent LND from failing during startup:

#9532

Database Query Overload at Startup:
Each startup triggers an excessive number of PostgreSQL queries, leading to a bottleneck. Many queries get stuck, and if not manually terminated, they eventually cause the node to crash.

[djkazic]

This is because LND waits for connections to become available rather than just blindly creating new connections per query:

lnd/kvdb/sqlbase/db_conn_set.go

Lines 51 to 57 in 5fe900d

	// Limit maximum number of open connections. This is useful to prevent
	// the server from running out of connections and returning an error.
	// With this client-side limit in place, lnd will wait for a connection
	// to become available.
	if d.maxConnections != 0 {
	db.SetMaxOpenConns(d.maxConnections)
	}

I am aware of that. that number used here is read from this variable here right:

lnd/sample-lnd.conf

Line 1573 in 27440e8

; db.postgres.maxconnections=50

With these default settings that means that LND will queue up to 50 transactions in parallel if for whatever reason the DB cannot process them faster than LND sends them right?

The amount of parallelism for LND queries to the DB is rather limited, but note that I've run on a rpi with 50 connections before without incident. The retries that you are seeing are normal and expected due to the level of transaction serializability used in LND.

[djkazic]

To be clear, I mean that even with 50 connections it is unlikely to observe 50 concurrent queries to the DB at a time.

[DerEwige]

To be clear, I mean that even with 50 connections it is unlikely to observe 50 concurrent queries to the DB at a time.

Unless your DB is running low on memory.
Oh now it starts to swapping.
Ah and an auto vacuum just started.
Now all transactions take 10 times longer than usually.

Now the usually low parallel count of 2-4 running transactions went up to 10.
Ah that makes the system use even more memory.
Ah now it is even slower.

Now transactions take even more time.
And now we are at 20 parallel transactions.

And so the death spiral started.
If the auto vacuum finishes before the system runs out of memory it usually survives.
If the DB hits 50 parallel transactions, then more and more of those transactions will be very long running high memory using transactions and the DB will slowly grind to a halt and crash.

Simply if you do not have enough memory to handle X transactions with max memory usage per transaction, then you should not allow X transactions in parallel.

If you have not enough CPU resources to handle X transactions in parallel, you should not allow X transactions in parallel.

[djkazic]

Hmm. I'm running my node off of just 4GB as well, it shouldn't be an issue. I actually run with 0 swap on my machine.

Autovacuuming is independent of the connections that LND makes; it should be tuned for the system (namely max_worker_processes, autovacuum_max_workers, autovacuum_work_mem, and autovacuum_vacuum_cost_delay).

My point here is that the max connections do not map to the number of queries made in parallel. But if you're experiencing such crashes you can decrease it.

[dunkfan]

I'm encountering this problem once every ~24 hours. 0.19.3 showing messages like this with lnd shutting down:
DISC: failed to check if scid 515889:2617:1 is closed: unknown postgres error: ERROR: out of shared memory (SQLSTATE 53200)
[CRT] SQLB: Caught unhandled error: ERROR: out of shared memory (SQLSTATE 53200)

I've upped the shared_buffers postgres setting to 2048MB
Is there anything else I should be doing?

[djkazic]

Please see: https://gist.github.com/djkazic/526fa3e032aea9578997f88b45b91fb9

The shared memory error is likely not about the shared_buffers config but rather the locks limits. These may take some tuning.