Skip to content

onionmessage: bound per-peer rate limiter registry #10981

Open
erickcestari wants to merge 2 commits into
lightningnetwork:masterfrom
erickcestari:fix-oom-onion-relay-all
Open

onionmessage: bound per-peer rate limiter registry #10981
erickcestari wants to merge 2 commits into
lightningnetwork:masterfrom
erickcestari:fix-oom-onion-relay-all

Conversation

@erickcestari

@erickcestari erickcestari commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

The PeerRateLimiter kept a token bucket per peer and never removed any
entry. With the channel-presence gate disabled via --onion-msg-relay-all
any connecting pubkey that sends an onion message allocates a bucket,
and buckets are retained across disconnects, so a peer reconnecting
under a stream of fresh identities could grow the map without bound and
exhaust memory.

Bound the registry by evicting buckets that have refilled to their full
burst once it grows past a high-water mark. A full bucket holds the same
tokens as a freshly-created one, so recreating it lazily on the peer's
next message preserves the limiter's behaviour, while a bucket that
still owes tokens is retained and the per-peer rate stays a ceiling
across reconnects.

Since only a full bucket is evictable, AllowN debits a newly created
bucket before publishing it. Otherwise the sweep that its own insertion
triggers reclaims it while still full and discards the debit, handing
the peer a fresh burst on every message.

@erickcestari
erickcestari force-pushed the fix-oom-onion-relay-all branch from 89763e3 to 91f0df9 Compare July 20, 2026 13:16
@github-actions github-actions Bot added severity-medium Focused review required and removed severity-medium Focused review required labels Jul 20, 2026
@github-actions

Copy link
Copy Markdown

🟡 PR Severity: MEDIUM

Classification | 3 files | 184 additions / 22 deletions

🟡 Medium (1 file)
  • onionmessage/ratelimit.go - Not a categorized critical/high package; falls under "other Go files not categorized above"
🟢 Low (2 files)
  • docs/release-notes/release-notes-0.22.0.md - Release notes
  • onionmessage/ratelimit_test.go - Test-only change

Analysis

This PR modifies rate-limiting logic in onionmessage/ratelimit.go, which is not one of the explicitly enumerated CRITICAL or HIGH severity packages (e.g. it is not lnwire/*, peer/*, htlcswitch/*, etc.). Onion message rate limiting is peripheral logic rather than core wallet, channel, HTLC, or peer-transport code, so it defaults to the "other Go files not categorized above" MEDIUM bucket.

No severity bump applies: excluding the test file, only 2 files are in scope (ratelimit.go and the release notes) with ~119 non-test lines changed in ratelimit.go, well under both the >20-file and >500-line bump thresholds. No critical packages are touched.

No override label is present.


To override, add a severity-override-{critical,high,medium,low} label.

@saubyk saubyk added this to lnd v0.22 Jul 20, 2026
@github-project-automation github-project-automation Bot moved this to Backlog in lnd v0.22 Jul 20, 2026
@saubyk saubyk added this to the v0.22.0 milestone Jul 20, 2026
@saubyk saubyk moved this from Backlog to In progress in lnd v0.22 Jul 20, 2026
Comment thread onionmessage/ratelimit.go
// A CompareAndSwap guard ensures at most one sweep runs at a time; callers
// that cross the high-water mark while a sweep is in flight skip it and let
// the running sweep do the work.
func (p *PeerRateLimiter) evictFullBuckets() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great fix! I was looking through evictFullBuckets and I think there might be a small TOCTOU race around the double-check that could cause numPeers to leak.

If a sweep happens at the same time a worker processes a message for the same peer, I think the sequence would be:

  • The sweeper removes the full bucket with LoadAndDelete.
  • A worker calls AllowN, doesn't find the peer, creates a new bucket, and increments numPeers.
  • The sweeper then hits the double-check, decides the bucket shouldn't have been deleted, and restores the old bucket, overwriting the worker's newly created one. Since it "undoes" the deletion, it also skips decrementing numPeers.

In that case, the map size stays the same, but numPeers ends up +1. If this happens repeatedly, numPeers could eventually drift past maxPeers, causing every new peer to trigger an O(N) sweep.

Would it make sense to just delete unconditionally here and avoid the double-check altogether? That seems like it would keep the map and numPeers in sync, but I may be missing something.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I've updated to delete unconditionally (and added a race condition test for regression) which might reset a bucket, however it is bounded to only 1 message, and it only happens when evicting, so it's bounded.

@erickcestari
erickcestari force-pushed the fix-oom-onion-relay-all branch from 91f0df9 to 60858a1 Compare July 22, 2026 13:54

@Roasbeef Roasbeef left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this fix!

Spotted a bug in the logic that would eliminate rate limiting all together once maxPeers was reached, see the comment inline.

Comment thread onionmessage/ratelimit.go
lim, _ = p.peers.LoadOrStore(peer, newLim)

var loaded bool
lim, loaded = p.peers.LoadOrStore(peer, newLim)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if we just added a new bucket/peer, then the token bucket is full. The call to evictFullBuckets below will then evict that bucket. The next message sent by the peer will then do the same thing, meaning they can use a full burst each time. We should do the sweep only at the end here to ensure the new message removes tokens from the bucket before we do the eviction attempt.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To talk thru a trivia case here, assume maxPeers is 1, then a new peer connects.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the code to avoid that. So now it first debit the bucket and then sweep it. I've also added more tests.

The PeerRateLimiter kept a token bucket per peer and never removed any
entry. With the channel-presence gate disabled via --onion-msg-relay-all
any connecting pubkey that sends an onion message allocates a bucket,
and buckets are retained across disconnects, so a peer reconnecting
under a stream of fresh identities could grow the map without bound and
exhaust memory.

Bound the registry by evicting buckets that have refilled to their full
burst once it grows past a high-water mark. A full bucket holds the same
tokens as a freshly-created one, so recreating it lazily on the peer's
next message preserves the limiter's behaviour, while a bucket that
still owes tokens is retained and the per-peer rate stays a ceiling
across reconnects.

Since only a full bucket is evictable, AllowN debits a newly created
bucket before publishing it. Otherwise the sweep that its own insertion
triggers reclaims it while still full and discards the debit, handing
the peer a fresh burst on every message.
@erickcestari
erickcestari force-pushed the fix-oom-onion-relay-all branch from 60858a1 to 650ff7a Compare July 25, 2026 17:55
@erickcestari
erickcestari requested a review from Roasbeef July 25, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

severity-medium Focused review required

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

4 participants