[g175] graph/db: v2 columns and v2 node CRUD #10339

ellemouton · 2025-11-03T08:07:11Z

This PR:

adds the new graph columns that will be needed for v2 gossip.
adds a constructor for creating a v2 models.Node.
updates some node related crud methods so that they explicitly pass in the gossip version they are part of. Then, a few unit tests are updated so that the CRUD for nodes is tested with both v1 and v2 nodes.

gemini-code-assist · 2025-11-03T08:08:19Z

Summary of Changes

Hello @ellemouton, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request lays crucial groundwork for the upcoming V2 gossip protocol by introducing version-aware database operations and extending the node data model. It refactors the graph store interface to support explicit gossip versions, adds new fields to the node model for V2 announcements, and updates all relevant CRUD methods and tests to handle these changes. This ensures that the system can properly store and retrieve node information for both V1 and V2 gossip, facilitating a smooth transition and coexistence of different protocol versions.

Highlights

Introduction of Versioned Graph Reader: A new VersionedReader has been introduced for the graph database, allowing operations to explicitly specify the gossip version (V1 or V2). This enables a clear distinction and handling of different gossip protocol versions within the database layer.
Generalized Graph Store Interface: The graphdb.V1Store interface has been refactored and renamed to graphdb.Store, making it a more general interface that can accommodate multiple gossip versions. Many methods within this interface now accept a lnwire.GossipVersion parameter.
V2 Node Model Enhancements: The models.Node struct has been extended with new fields, LastBlockHeight and ExtraSignedFields, to support the additional data required for V2 gossip node announcements. A new constructor, NewV2Node, has also been added.
Updated Node CRUD Operations: Node-related Create, Read, Update, and Delete (CRUD) methods have been updated to explicitly pass the gossip version, ensuring that operations are performed against the correct version of node data.
Database Schema Migration for V2 Gossip: New columns have been added to the SQL database schema (graph_nodes, graph_channels, graph_channel_policies) to store V2 gossip-specific information, such as block_height, signature, funding_pk_script, merkle_root_hash, and disable_flags.
Comprehensive Test Coverage for V2 Nodes: Extensive unit and integration tests have been updated and expanded to validate the new V2 node model and its corresponding CRUD operations, ensuring correct behavior for both V1 and V2 gossip data.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces support for v2 gossip messages in the graph database. It adds new columns to the database schema, refactors the database store interface to be version-aware, and introduces a VersionedReader to abstract gossip versioning for consumers. The changes are extensive and well-structured. I've found a couple of issues: a potential overflow bug due to an incorrect type cast for block height, and a bug in a database migration script where a CHECK constraint is applied to the wrong column. I've also included some suggestions to improve documentation according to the project's style guide.

sqldb/sqlc/migrations/000009_graph_v2.up.sql

graph/db/graph.go

graph/db/sql_store.go

The config file format changed. The tool golangci-lint migrate was used to migrate the old config. However old comments and also the structure of the disabled linters was preserved. Moreover the new v2 version introduced new linters, we disable 3 of them because they are very noise and we do not really want to check for them: funcorder, noinlineerr, embeddedstructfieldcheck.

The custom file is only needed in the tools directory.

And ensure that both versions 1 and 2 implement it.

Define a GossipVersion enum along with a GossipMessage interface to be satisfied by all gossip related messages. This will be useful later on when we want to make decisions based on the protocol version that a message is part of.

Since the gossip protocols are completely disjoint, we need to treat messages on the two protocols completely separately and should not let rejections on one protocol affect how we treat messages on the other.

We now make sure we only set the custom channel data in the lnrpc.Route only if it contains relevant data.

We now return an error when blinded and non blinded attempts are combined. This was theoretically possible to register a legacy attempt in combination with a blinded payment. This would have been prevented by other checks in the code because legacy payments are not split into shards.

This message type is a message that carries an onion-encrypted payload used for BOLT12 messages.

This commit creates the necessary endpoints for onion messages. Specifically, it adds the following: - `SendOnionMessage` endpoint to send onion messages. - `SubscribeOnionMessages` endpoint to subscribe to incoming onion messages. It uses the `msgmux` package to handle the onion messages.

The only way to unblock SendCustomMessage is if the peer activates, disconnects or the server shuts down. This means that if the context is cancelled, we will still wait until one of those other events happen. With this commit we thread the context through to SendCustomMessage, so that if the context is cancelled, we can return early. This improves the cancellation semantics.

This tests was a temporary helper to let devs test the graph SQL migration before it was plugged in to LND. But that migration has now shipped and so we can remove this.

Copy over all the code that the graph SQL migration needs to a separate folder. This will let us advance the main graph SQL CRUD code without worrying about changing the sql migration code. It will also let us change the SQL queries without changing the migration. In this commit, only the migration logic is "frozen" but in an upcoming commit, the sqlc queries & models will be frozen too.

…Freeze graph/db: freeze the SQL migration code

Add a new migration that updates the graph tables (nodes, channels and policies) in preparation for the new columns required for V2 announcements. This migration has to be added to the set of "live" migrations instead of "dev only" since it edits the columns of existing tables and so changes the existing sql models. We are going to prep the SQLStore code to handle the V2 types in the coming commits, so we need this migration to be in place. In this commit we also remove the TestSchemaMigrationIdempotency test since this test fails with the new "ALTER TABLE" migrations which dont have "IF NOT EXISTS" options like tables and indexes do. Migrations should be idempotent anyways due to the migration tracker file and/or the sqlc migration tracker.

The underling store will store gossip messages across gossip versions and we will instead expose version parameters on many of the methods. So this interface really just abstracts the underlying store/schema type.

We add a new NewV2Node constructor which takes a new NodeV2Fields as a parameter. This NodeV2Fields struct defines the fields that can be set in a models.Node if the version is V2.

Here we update the UpdateNode query so that it can be used to insert the new blockheight field for a v2 node.

Instead of embedding Store in ChannelGraph so that any methods of the Store interface not implemented by the ChannelGraph are redirected to the underlying Store, we update date things in this commit to instead make the "redirection" explicit. This is in preparation for changes we will make soon where some underlying store methods will take an explicit "version" parameter but then we will keep the ChannelGraph methods as is so that existing call-sites dont all need to be updated. We will then add "Versioned" ChannelGraph wrapper which decides the version use. Initially, most call-sites will just create a wrapped V1 ChannelGraph so that the logic remains as it is today.

Update the SQLStore node writer and readers to handle V2 ndoes. Currently no logic will actually add such nodes. The following commits will update what is needed so that CRUD for v2 nodes can be tested.

HasNode currently is very v1 specific since it returns a time.Time timestamp which is specific to V1 node announcements. However, it is mostly only used for the "exists" return value. So here we split it up into HasNode which just checks existence and HasV1Node which retains the same behavaiour as before.

Update some of the node related graph CRUD methods to take a version rather than hardcoding them in the SQLStore layer. Move the version up one layer instead. This will make it easier to make it configurable later on.

Add an isSQLDB variable that will let us quickly check in tests if the backing DB is KVStore or SQLStore.

For now, all instances will use V1 only so as not to change behaviour yet.

In this commit, we update TestNodeInsertionAndDeletion so that it is run twice: once with V1 nodes and another with V2 nodes. This way, we are testing the basic CRUD for V2 nodes. In later commits we can add tests that tests the interaction between V1 and V2 nodes.

Convert a few more CRUD methods to handle v2 nodes. Then update some more tests to be run against both v1 and v2 nodes.

lightninglabs-deploy · 2025-11-10T09:44:18Z

@ellemouton, remember to re-request review from reviewers when ready

ellemouton self-assigned this Nov 3, 2025

gemini-code-assist bot reviewed Nov 3, 2025

View reviewed changes

sqldb/sqlc/migrations/000009_graph_v2.up.sql Outdated Show resolved Hide resolved

graph/db/graph.go Show resolved Hide resolved

graph/db/graph.go Show resolved Hide resolved

graph/db/sql_store.go Show resolved Hide resolved

ellemouton force-pushed the elle-graphSQLMigFreeze branch from 273d8b8 to ae446b8 Compare November 3, 2025 08:53

ellemouton force-pushed the g175Prep3 branch 3 times, most recently from 7988c44 to 9306c58 Compare November 3, 2025 11:19

ellemouton force-pushed the elle-graphSQLMigFreeze branch from ae446b8 to 70894d3 Compare November 3, 2025 11:35

ellemouton force-pushed the g175Prep3 branch 2 times, most recently from c8cd4da to edd9890 Compare November 6, 2025 15:51

saubyk added this to v0.21 Nov 6, 2025

saubyk moved this to In progress in v0.21 Nov 6, 2025

ellemouton force-pushed the g175Prep3 branch from edd9890 to c8cd4da Compare November 6, 2025 16:30

ziggie1984 and others added 17 commits November 7, 2025 17:15

tools: use new golang 1.24 tool directive

3265d86

linter: remove unsupported linters from the codebase

7b0b9b6

CI: also use go tool for gosimports

76d0250

CI: remove unnecessary custom-gcl.yml file

8f0a1dd

The custom file is only needed in the tools directory.

makefile: add linter config check

b977ba8

lnwire: add NodeAnnouncement interface

80b98fd

And ensure that both versions 1 and 2 implement it.

lnwire: define GossipVersion and GossipMessage

750ac7c

Define a GossipVersion enum along with a GossipMessage interface to be satisfied by all gossip related messages. This will be useful later on when we want to make decisions based on the protocol version that a message is part of.

lnwire: let gossip messages implement GossipMessage

5fc1e37

graph/db: use lnwire.GossipVersion instead of ProtocolVersion

e6e5f29

discovery: let reject cache use gossip version in key

cbe5a5f

Since the gossip protocols are completely disjoint, we need to treat messages on the two protocols completely separately and should not let rejections on one protocol affect how we treat messages on the other.

mulit: don't set customData on the lnrpc route level

a0aad14

We now make sure we only set the custom channel data in the lnrpc.Route only if it contains relevant data.

lnwire: add onion message type

b17a49f

This message type is a message that carries an onion-encrypted payload used for BOLT12 messages.

lnrpc: SendOnionMessage rpc endpoint

08d9685

ellemouton added 21 commits November 7, 2025 17:15

graph/db: remove outdated comment

6fcdbbb

graph/db: remove TestPopulateViaMigration

1c3685f

This tests was a temporary helper to let devs test the graph SQL migration before it was plugged in to LND. But that migration has now shipped and so we can remove this.

go.mod: replace local sqldb

dc134c0

graph/db: freeze sql migration queries

058e188

docs: update release notes

2c9b47f

Merge pull request lightningnetwork#10338 from ellemouton/graphSQLMig…

706b22a

…Freeze graph/db: freeze the SQL migration code

devrpc: fix comment

00690cc

graph/db: rename V1Store to Store

d54c54d

The underling store will store gossip messages across gossip versions and we will instead expose version parameters on many of the methods. So this interface really just abstracts the underlying store/schema type.

graph/db: add V2 Node constructor

f733f6c

We add a new NewV2Node constructor which takes a new NodeV2Fields as a parameter. This NodeV2Fields struct defines the fields that can be set in a models.Node if the version is V2.

sqldb: update node query for v2

ab2cd65

Here we update the UpdateNode query so that it can be used to insert the new blockheight field for a v2 node.

graph/db: allow v2 nodes

2115261

Update the SQLStore node writer and readers to handle V2 ndoes. Currently no logic will actually add such nodes. The following commits will update what is needed so that CRUD for v2 nodes can be tested.

graph/db: move gossip versions up one layer

1410779

Update some of the node related graph CRUD methods to take a version rather than hardcoding them in the SQLStore layer. Move the version up one layer instead. This will make it easier to make it configurable later on.

graph/db: add helper variable for tests

4bc6f92

Add an isSQLDB variable that will let us quickly check in tests if the backing DB is KVStore or SQLStore.

graph/db: introduce VersionedReader

f114ba0

For now, all instances will use V1 only so as not to change behaviour yet.

graph/db: let node helpers take version

bd8f180

graph/db: convert other node CRUD methods

46ce8ab

Convert a few more CRUD methods to handle v2 nodes. Then update some more tests to be run against both v1 and v2 nodes.

ellemouton force-pushed the g175Prep3 branch from c8cd4da to 89f9bc0 Compare November 10, 2025 05:53

ellemouton changed the base branch from elle-graphSQLMigFreeze to 0-21-0-staging November 10, 2025 05:53

docs: update release notes

989e778

ellemouton force-pushed the g175Prep3 branch from 89f9bc0 to 989e778 Compare November 10, 2025 09:15

ellemouton requested review from bhandras and bitromortac November 10, 2025 12:03

yyforyongyu force-pushed the 0-21-0-staging branch from 706b22a to 95b84a8 Compare November 12, 2025 14:54

yyforyongyu changed the base branch from 0-21-0-staging to master November 13, 2025 08:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[g175] graph/db: v2 columns and v2 node CRUD #10339

[g175] graph/db: v2 columns and v2 node CRUD #10339

Uh oh!

ellemouton commented Nov 3, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot commented Nov 3, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lightninglabs-deploy commented Nov 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[g175] graph/db: v2 columns and v2 node CRUD #10339

Are you sure you want to change the base?

[g175] graph/db: v2 columns and v2 node CRUD #10339

Uh oh!

Conversation

ellemouton commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot commented Nov 3, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lightninglabs-deploy commented Nov 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ellemouton commented Nov 3, 2025 •

edited

Loading