Fix double-forward, prefer legacy forward maps #4289

valentinewallace · 2025-12-16T22:46:46Z

Addresses a chunk of the feedback from #4227 (review) (tracked in #4280). Splitting it out for ease of review.

Fix a bug that would cause double-forwarding of inbound HTLCs when using reconstructed forward maps
Prefer legacy forward maps in production while randomly using reconstructed maps in tests for coverage (see commit message)
a few other nits from the aforementioned review

ldk-reviews-bot · 2025-12-16T22:46:49Z

👋 Thanks for assigning @joostjager as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

valentinewallace · 2025-12-16T22:47:05Z

Going to take another look at this tomorrow before un-drafting it

codecov · 2025-12-17T00:06:18Z

Codecov Report

❌ Patch coverage is 74.47917% with 49 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.59%. Comparing base (c9f022b) to head (5a4912c).
⚠️ Report is 16 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/ln/channelmanager.rs	74.73%	43 Missing and 5 partials ⚠️
lightning/src/ln/channel.rs	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4289   +/-   ##
=======================================
  Coverage   86.58%   86.59%           
=======================================
  Files         158      158           
  Lines      102287   102368   +81     
  Branches   102287   102368   +81     
=======================================
+ Hits        88568    88644   +76     
- Misses      11304    11311    +7     
+ Partials     2415     2413    -2

Flag	Coverage Δ
fuzzing	`35.95% <15.42%> (-0.90%)`	⬇️
tests	`85.87% <74.47%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

lightning/src/ln/channelmanager.rs

lightning/src/ln/reload_tests.rs

TheBlueMatt · 2026-01-07T20:06:01Z

lightning/src/ln/channelmanager.rs

 			(17, in_flight_monitor_updates, option),
 			(19, peer_storage_dir, optional_vec),
 			(21, WithoutLength(&self.flow.writeable_async_receive_offer_cache()), required),
+			(23, reconstruct_manager_from_monitors, required),


Oops sorry for the delay. Writing a bool here to determine whether to look at data we're always writing seems quite weird? I'm not sure what the right answer is, but ideally we run tests with both the new and old code. In the past (with block connection) we've taken a somewhat hacky approach of just flipping a coin and using a random value to decide. In general its worked and we haven't seen many cases of flaky tests making their way upstream, but its a bit more annoying for devs. Still, absent a better option (I'm not a huge fan of running the entire test suite twice every time, even running it an extra time in CI kinda sucks...) that seems reasonable.

Agreed, I went with the random option you mention.

We are now still running the entire test suite twice, but not on the same CI run. I think better options are:

Deliberately pick a few tests that sufficient cover the logic, and only run those twice.

Do a more extensive test matrix nightly.

I feel more comfortable with the entire test suite running with the new vs old code rather than selected tests, but adding a whole extra CI run to an already slow CI does suck IMO. So I like this current tradeoff.

How about another environment variable? lol

I think that there will be much more of this for the chan mgr refactor. My draft PRs also assume a safe_channels feature flag and an associated CI job. My preference would be to use the same mechanism here, and accept that while the project is underway, we'll have an additional CI job (a single job for all PRs in the project) that only runs on a single platform. It's not that significant.

valentinewallace · 2026-01-07T20:44:35Z

Addressed feedback, main diff is here. Also pushed some whitespace fixes after.

joostjager · 2026-01-08T08:04:19Z

lightning/src/ln/channelmanager.rs

+		// to ensure the legacy codepaths also have test coverage.
+		#[cfg(not(test))]
+		let reconstruct_manager_from_monitors = false;
+		#[cfg(test)]


What happened to the idea of using the safe_channels flag here, so that we can gate this and all other changes in the chan mgr refactor project, and make it worth doing a separate CI run for it?

Using conditional compilation for the legacy code (not safe_channels) might improve readability. I noticed that I did had to pay some attention to the control flow interventions with for example continue statements when reconstructing. Also makes it easier to delete that code eventually.

valentinewallace · 2026-01-08T17:54:22Z

lightning/src/ln/channelmanager.rs

+		#[cfg(not(test))]
+		let reconstruct_manager_from_monitors = false;
+		#[cfg(test)]
+		let reconstruct_manager_from_monitors = {


@TheBlueMatt pointed out that the way this is currently structured, a future version of LDK that does not write the pending_intercepted_htlcs/forward_htlcs maps will not be able to downgrade to this version of the code, because it only runs the reconstruction logic in tests.

So instead of running reconstruction logic in tests only, we should consider running it if the manager's written version is >= X, where X is a future version where we can assume that the new data is always present and the old data stopped being written.

I'm not sure what version that would be, and I also think we can hold off on this a little until we look into reconstructing more maps. Otherwise we might have some additional complexity (i.e. one var for reconstruct_fwd_maps_from_monitors if version > X, one for reconstruct_claimable_map_from_monitors if version > Y, etc). But worth thinking about and incorporating into upcoming PRs.

How can we add a conditional like that if we don't know yet which version the new data is always present?

valentinewallace · 2026-01-08T19:02:52Z

Discussed offline, going to add an environmental variable to set which manager reconstruction paths to use. Rebased on main to get the changes from #4296

valentinewallace · 2026-01-08T19:32:47Z

Added the environment variable: diff

joostjager

As mentioned before, I reluctantly accept the test input randomization and the reload boolean.

The only thing I'd like to be sure of that I really understand is the upgrade/downgrade plan that was landed on. Ideally the new data would be written and used in the next release. Postponing that another release is a big decision and it seems there is not that much gained from it except for removal of some code that already exists in main.

lightning/src/ln/channelmanager.rs

joostjager · 2026-01-09T12:39:50Z

lightning/src/ln/channelmanager.rs

+		// persist that state, relying on it being up-to-date on restart. Newer versions are moving
+		// towards reducing this reliance on regular persistence of the `ChannelManager`, and instead
+		// reconstruct HTLC/payment state based on `Channel{Monitor}` data if
+		// `reconstruct_manager_from_monitors` is set below. Currently it is only set in tests, randomly


Bringing everything together, would the upgrade/downgrade situation look like this?

Version Read Write Upgrade to version (max) Downgrade to version (min)

0.2 legacy legacy 0.5 -

0.3 legacy legacy+new 0.6 0.2

0.4 legacy+new legacy+new 0.6 0.2

0.5 legacy+new new 0.6 0.4

0.6 new new - 0.4

Also wondering - in relation to the discussion in yesterday's sync meet - how the reconstruct_manager_from_monitors makes for a faster path?

would the upgrade/downgrade situation look like this?

My understanding is that our goal is that 0.3 would support reading new as well, so that 0.5 can downgrade to 0.3 rather than 0.4.

Also wondering - in relation to the discussion in yesterday's sync meet - how the reconstruct_manager_from_monitors makes for a faster path?

So that we get the above. I don't see a reason to want to only allow 0.5 to downgrade to 0.4 rather than 0.3. The code currently doesn't do that but presumably in a followup we could do that?

@valentinewallace and I discussed this more. It seems that having a dedicated flag for signaling that the old maps are not written is better than guessing at a future version number where this is the case.

This currently assumes we'll skip one version before merging the final changes (new read and new write only), but the flexibility remains to wait more versions and enlarge the downgrade window.

Version Read Write Upgrade to version (max) Downgrade to version (min) Set flag

0.2 (current) legacy legacy 0.4 any

0.3 legacy OR new with flag legacy+new any any

0.4 legacy OR new with flag legacy+new any any

0.5 new new any 0.3 X

It seems that having a dedicated flag for signaling that the old maps are not written is better than guessing at a future version number where this is the case.

You mean instead of using SERIALIZATION_VERSION/MIN_SERIALIZATION_VERSION constants you want to use a TLV? I guess that's fine, but it seems much simpler to use the version numbers so that we can also drop some of the legacy crap that is written as non-TLVs that we'll never be writing anymore. Don't really see a reason to avoid that.

The upgrade path looks right to me, though.

I still think it is not a great idea to assume things about a specific future version number.

joostjager · 2026-01-09T12:41:06Z

lightning/src/ln/channelmanager.rs

+		#[cfg(not(test))]
+		let reconstruct_manager_from_monitors = false;
+		#[cfg(test)]
+		let reconstruct_manager_from_monitors = {


How can we add a conditional like that if we don't know yet which version the new data is always present?

lightning/src/ln/channelmanager.rs

joostjager · 2026-01-09T13:41:34Z

You may want to update the PR title and description with details

ldk-reviews-bot · 2026-01-10T00:01:03Z

🔔 1st Reminder

Hey @TheBlueMatt! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2026-01-12T00:01:41Z

🔔 2nd Reminder

Hey @TheBlueMatt! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

Necessary for the next commit and makes it easier to read.

We recently began reconstructing ChannelManager::decode_update_add_htlcs on startup, using data present in the Channels. However, we failed to prune HTLCs from this rebuilt map if a given HTLC was already forwarded to the outbound edge (we pruned correctly if the outbound edge was a closed channel, but not otherwise). Here we fix this bug that would have caused us to double-forward inbound HTLC forwards.

No need to iterate through all entries in the map, we can instead pull out the specific entry that we want.

We are working on removing the requirement of regularly persisting the ChannelManager, and as a result began reconstructing the manager's forwards maps from Channel data on startup in a recent PR, see cb398f6 and parent commits. At the time, we implemented ChannelManager::read to prefer to use the newly reconstructed maps, partly to ensure we have test coverage of the new maps' usage. This resulted in a lot of code that would deduplicate HTLCs that were present in the old maps to avoid redundant HTLC handling/duplicate forwards, adding extra complexity. Instead, always use the old maps in prod, but randomly use the newly reconstructed maps in testing, to exercise the new codepaths (see reconstruct_manager_from_monitors in ChannelManager::read).

joostjager

Assuming the upgrade story will be refined in the follow up, all my comments have been addressed.

This was referenced Dec 16, 2025

Reconstruct ChannelManager forwarded HTLCs maps from Channels #4227

Merged

Stale ChannelManager causes channel force-closures #4286

Open

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch from 89f5d07 to c6bb096 Compare December 17, 2025 20:25

valentinewallace commented Dec 17, 2025

View reviewed changes

lightning/src/ln/channelmanager.rs Outdated Show resolved Hide resolved

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch from c6bb096 to 425747e Compare December 17, 2025 21:13

joostjager reviewed Dec 18, 2025

View reviewed changes

lightning/src/ln/channelmanager.rs Show resolved Hide resolved

lightning/src/ln/channelmanager.rs Outdated Show resolved Hide resolved

valentinewallace mentioned this pull request Jan 6, 2026

Follow-ups to #4227 (Part 2) #4303

Open

1 task

valentinewallace added this to the 0.3 milestone Jan 6, 2026

TheBlueMatt reviewed Jan 7, 2026

View reviewed changes

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch 3 times, most recently from 7b858c1 to 71062f2 Compare January 7, 2026 20:42

valentinewallace marked this pull request as ready for review January 7, 2026 20:44

ldk-reviews-bot requested a review from wpaulino January 7, 2026 20:45

valentinewallace requested review from TheBlueMatt and joostjager and removed request for wpaulino January 7, 2026 20:49

joostjager reviewed Jan 8, 2026

View reviewed changes

valentinewallace commented Jan 8, 2026

View reviewed changes

valentinewallace self-assigned this Jan 8, 2026

valentinewallace added this to Weekly Goals Jan 8, 2026

valentinewallace moved this to Goal: Merge in Weekly Goals Jan 8, 2026

valentinewallace added the weekly goal Someone wants to land this this week label Jan 8, 2026

Remove unnecessary update_add clone on Channel ser

7c97f3c

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch from 71062f2 to f1ea7bb Compare January 8, 2026 19:01

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch from f1ea7bb to f245139 Compare January 8, 2026 19:31

valentinewallace requested a review from joostjager January 8, 2026 19:32

joostjager reviewed Jan 9, 2026

View reviewed changes

valentinewallace changed the title ~~Follow-ups to #4227 (Part 1)~~ Fix double-forward, prefer legacy forward maps Jan 12, 2026

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch from f245139 to 17b562c Compare January 12, 2026 21:02

valentinewallace added 5 commits January 12, 2026 16:13

Move is_chan_closed check into loop

2a1273b

Necessary for the next commit and makes it easier to read.

Optimize dedup_decode_update_add_htlcs

03882bd

No need to iterate through all entries in the map, we can instead pull out the specific entry that we want.

Update outdated comment due to var renames

5a4912c

valentinewallace force-pushed the 2025-12-reconstruct-fwds-followup branch from 17b562c to 5a4912c Compare January 12, 2026 21:15

valentinewallace requested a review from joostjager January 13, 2026 14:51

joostjager approved these changes Jan 13, 2026

View reviewed changes

TheBlueMatt approved these changes Jan 13, 2026

View reviewed changes

TheBlueMatt merged commit c5d7b13 into lightningdevkit:main Jan 13, 2026
19 of 20 checks passed

github-project-automation bot moved this from Goal: Merge to Done in Weekly Goals Jan 13, 2026

Version	Read	Write	Upgrade to version (max)	Downgrade to version (min)
0.2	legacy	legacy	0.5	-
0.3	legacy	legacy+new	0.6	0.2
0.4	legacy+new	legacy+new	0.6	0.2
0.5	legacy+new	new	0.6	0.4
0.6	new	new	-	0.4

Version	Read	Write	Upgrade to version (max)	Downgrade to version (min)	Set flag
0.2 (current)	legacy	legacy	0.4	any
0.3	legacy OR new with flag	legacy+new	any	any
0.4	legacy OR new with flag	legacy+new	any	any
0.5	new	new	any	0.3	X

Fix double-forward, prefer legacy forward maps #4289

Fix double-forward, prefer legacy forward maps #4289

Conversation

valentinewallace commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ldk-reviews-bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

valentinewallace commented Dec 16, 2025

Uh oh!

codecov bot commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

valentinewallace Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joostjager Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

valentinewallace commented Jan 7, 2026

Uh oh!

joostjager Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

valentinewallace commented Jan 8, 2026

Uh oh!

valentinewallace commented Jan 8, 2026

Uh oh!

joostjager left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TheBlueMatt Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joostjager Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joostjager commented Jan 9, 2026

Uh oh!

ldk-reviews-bot commented Jan 10, 2026

Uh oh!

ldk-reviews-bot commented Jan 12, 2026

Uh oh!

joostjager left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

valentinewallace commented Dec 16, 2025 •

edited

Loading

ldk-reviews-bot commented Dec 16, 2025 •

edited

Loading

codecov bot commented Dec 17, 2025 •

edited

Loading

valentinewallace Jan 8, 2026 •

edited

Loading

joostjager Jan 8, 2026 •

edited

Loading

joostjager Jan 8, 2026 •

edited

Loading

TheBlueMatt Jan 11, 2026 •

edited

Loading

joostjager Jan 12, 2026 •

edited

Loading