collectors: collect payment and attempt counts #115

calvinrzachman · 2025-02-26T21:16:26Z

Add support for real time tracking of counters for payments and attempts. These can be used to compute and create visualizations for things like average payment throughput and average # of attempts per payment.

rate(lnd_total_payments[5m])
rate(lnd_total_htlc_attempts[30m]) / rate(lnd_total_payments[30m])

The information is obtained in real time from lnd using the TrackPayments RPC.

After making payments between two directly connected nodes:

alice-64db66cb64-g4sh2:/# curl -s http://localhost:8990/metrics | grep lnd_total
# HELP lnd_total_htlc_attempts Total number of HTLC attempts across all payments
# TYPE lnd_total_htlc_attempts counter
lnd_total_htlc_attempts 5
# HELP lnd_total_payments Total number of payments initiated
# TYPE lnd_total_payments counter
lnd_total_payments 5

calvinrzachman · 2025-02-28T20:56:39Z

I may be able to consolidate the # attempts with the existing HTLC collector. Checking...

collectors/payments_collector.go

Roasbeef · 2025-03-05T23:55:27Z

collectors/payments_collector.go

+
+	paymentLogger.Info("Starting payments monitor...")
+
+	stream, err := p.client.TrackPayments(ctx,


We should also verify that this is using: https://lightning.engineering/api-docs/api/lnd/router/track-payment-v2/

I think it's not: https://github.com/lightningnetwork/lnd/blob/4ccf4fc24c750d098cf24566ef4bbc0311c7d476/lnrpc/routerrpc/router_grpc.pb.go#L39

That said, it might not be comparable as TrackPaymentV2 is for a single payment whereas TrackPayments is for payment events in general

I think we're good to go. Check this out: https://lightning.engineering/api-docs/api/lnd/router/track-payments/. Originally I was looking at using the lndclient library like the rest of the code base, but it doesn't yet support TrackPayments so decided to use raw routerrpc client.

Roasbeef · 2025-03-05T23:57:08Z

collectors/payments_collector.go

+// complete list of all htlc attempts made for this payment.
+func processPaymentUpdate(payment *lnrpc.Payment) {
+	totalPayments.Inc()
+	attemptCount := len(payment.Htlcs)


Hmm, I don't think this can quite be used as a proxy for attempts. For that we'd need to watch a payment over time, and increment this counter with each attempt.

We may also want to introspect into the payment state itself: https://lightning.engineering/api-docs/api/lnd/router/track-payment-v2/#lnrpcpaymentpaymentstatus

Double checked, and I think I'm actually wrong about this. We get a new element here for each new attempt, as it isn't just the set of final HTLCs that were settled.

Prior to learning about this TrackPayments RPC, I had created a draft PR to add these metrics as real time counters in lnd directly. Implemented there, the ChannelRouter can increment counters as each attempt is registered. My hope here was that supplying the NoInflightUpdates directive to the TrackPayments call will give us only the final payment update (settle or fail) from which we can make an accurate determination of how many total attempts were made.

Roasbeef · 2025-03-05T23:57:45Z

collectors/payments_collector.go

+// NOTE: It is expected that this receive the *final* payment update with the
+// complete list of all htlc attempts made for this payment.
+func processPaymentUpdate(payment *lnrpc.Payment) {
+	totalPayments.Inc()


We should look at the payment state and only increment this if it's IN_FLIGHT. That way we won't count a new attempt as a new payment.

I think we might be safe here so long as my comment about NoInflightUpdates delivering only the terminal payment update (settle/fail) is correct. That way, we'll only process the payment a single time and capture all of the attempts during its life-cycle.

I could add an explicit check that the payment status is terminal to be extra sure though.

bitromortac

Cool, I think this looks ok with TrackPayments. We won't get a resolution of how quickly the individual attempts are being made over time, as we will only receive the info once settled/failed (for that we'd need a combination of TrackPayments and TrackPaymentsV2?). Will also test once changes were addressed

bitromortac · 2025-03-07T11:02:29Z

collectors/payments_collector.go

+type paymentsMonitor struct {
+	client routerrpc.RouterClient
+
+	ctx context.Context


This is a bit of an anti-pattern to have it as a member, maybe the context could be created at start time with LndServices as a field. In general there are many occurrences of ctx.Background in this repo, which needs an overhaul as well (not this PR).

Updated. Let me know if it looks good.

Looks good, a nice pattern would be to use a fn.ContextGuard, but since that's not used in this repo for other structs, it's better to keep things consistent.

Sounds good. I'll check the type out and keep it in mind going forward. Thanks for the tip.

collectors/payments_collector.go

lightninglabs-deploy · 2025-04-04T13:17:13Z

@calvinrzachman, remember to re-request review from reviewers when ready

calvinrzachman · 2025-04-07T23:45:28Z

After updating to CounterVec + status label the exported metrics look like:

# HELP lnd_total_htlc_attempts Total number of HTLC attempts across all payments
# TYPE lnd_total_htlc_attempts counter
lnd_total_htlc_attempts 5

# HELP lnd_total_payments Total number of payments initiated, labeled by final status
# TYPE lnd_total_payments counter
lnd_total_payments{status="failed"} 1
lnd_total_payments{status="succeeded"} 5

bitromortac

LGTM, tACK 📊

The github workflow actions need updates to v3 or v4 for CI to pass.

collect raw payment and attempt count information from lnd using the TrackPayments RPC.

calvinrzachman requested a review from djkazic February 26, 2025 21:16

calvinrzachman force-pushed the payment-collector branch from ed0fd64 to ebd6b23 Compare February 26, 2025 21:48

djkazic approved these changes Feb 27, 2025

View reviewed changes

calvinrzachman force-pushed the payment-collector branch from ebd6b23 to 0cf9fd4 Compare February 28, 2025 20:03

calvinrzachman requested a review from Roasbeef February 28, 2025 20:31

Roasbeef requested changes Mar 5, 2025

View reviewed changes

calvinrzachman force-pushed the payment-collector branch 2 times, most recently from 5a38420 to 0b15285 Compare March 6, 2025 01:10

bitromortac reviewed Mar 7, 2025

View reviewed changes

calvinrzachman force-pushed the payment-collector branch from 0b15285 to 82fbd65 Compare April 7, 2025 16:54

calvinrzachman requested a review from bitromortac April 7, 2025 16:57

calvinrzachman force-pushed the payment-collector branch from 82fbd65 to da52f5f Compare April 7, 2025 23:45

bitromortac approved these changes Apr 8, 2025

View reviewed changes

calvinrzachman added 2 commits April 8, 2025 10:05

collectors: collect payment and attempt counts

c1159c6

collect raw payment and attempt count information from lnd using the TrackPayments RPC.

gh-actions: bump action versions

29b674f

calvinrzachman force-pushed the payment-collector branch from b758442 to 29b674f Compare April 8, 2025 15:17

saubyk assigned calvinrzachman Apr 9, 2025

Roasbeef merged commit fd56748 into lightninglabs:master Apr 10, 2025
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

collectors: collect payment and attempt counts #115

collectors: collect payment and attempt counts #115

calvinrzachman commented Feb 26, 2025 •

edited

Loading

calvinrzachman commented Feb 28, 2025

Roasbeef Mar 5, 2025

djkazic Mar 6, 2025

djkazic Mar 6, 2025

calvinrzachman Mar 6, 2025

Roasbeef Mar 5, 2025

Roasbeef Mar 5, 2025

calvinrzachman Mar 6, 2025

Roasbeef Mar 5, 2025

calvinrzachman Mar 6, 2025

bitromortac left a comment

bitromortac Mar 7, 2025

calvinrzachman Apr 7, 2025

bitromortac Apr 8, 2025

calvinrzachman Apr 8, 2025

lightninglabs-deploy commented Apr 4, 2025

calvinrzachman commented Apr 7, 2025 •

edited

Loading

bitromortac left a comment


		paymentLogger.Info("Starting payments monitor...")

		stream, err := p.client.TrackPayments(ctx,

collectors: collect payment and attempt counts #115

collectors: collect payment and attempt counts #115

Conversation

calvinrzachman commented Feb 26, 2025 • edited Loading

calvinrzachman commented Feb 28, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitromortac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lightninglabs-deploy commented Apr 4, 2025

calvinrzachman commented Apr 7, 2025 • edited Loading

bitromortac left a comment

Choose a reason for hiding this comment

calvinrzachman commented Feb 26, 2025 •

edited

Loading

calvinrzachman commented Apr 7, 2025 •

edited

Loading