[graph mig 1]: graph/db: migrate graph nodes from kvdb to SQL #10036

ellemouton · 2025-07-04T12:24:56Z

This PR adds the framework for the migration function and main migration test of the graph store.
It then adds the first part of the migration which migrates the nodes & source node.

Please see #10025 for the final result we are aiming for here.
Part of #9795

gemini-code-assist

Summary of Changes

Hello @ellemouton, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request initiates the significant effort to migrate the graph store from its current KV database backend to a new SQL database. It establishes the core migration framework and successfully migrates all existing graph nodes and the designated source node, ensuring data integrity through rigorous validation and comparison. This is the first in a series of changes aimed at transitioning the graph data storage.

Highlights

Graph Database Migration Framework: Introduces the foundational MigrateGraphToSQL function and supporting helpers (checkGraphExists, compare) to facilitate the transition of the graph store from a KV database to a SQL database. This function is currently only called via tests and will be integrated into the main lnd binary once the full migration is complete.
Node and Source Node Migration: Implements the first phase of the graph migration, specifically handling the transfer of all existing graph nodes and the designated source node from the KV backend to the new SQL schema. This includes robust validation of node data, such as skipping nodes with invalid TLV streams, and post-migration verification to ensure data integrity.
Comprehensive Migration Testing: Adds a dedicated and extensive test suite (sql_migration_test.go) to thoroughly validate the migration logic. This includes tests for various node and source node scenarios, edge cases like malformed data, and a utility for testing against real-world channel.db or channel.sqlite files (though this specific test is skipped by default).
Test Infrastructure Refinement: Refactors the database test setup by introducing a newBatchQuerier helper in both test_postgres.go and test_sqlite.go. A new test_sql.go file centralizes the NewTestDB function, streamlining the creation of SQL-backed test environments and improving modularity.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces the framework for migrating the graph store from a KV backend to a SQL backend. It includes the migration logic for nodes and the source node, along with comprehensive tests. The code is well-structured and the migration process is robust, with verification steps to ensure data integrity.

graph/db/sql_migration_test.go

docs/release-notes/release-notes-0.20.0.md

graph/db/sql_migration.go

Fix a typo in the build directive.

Factor out the transaction executor construction so that we can have access to the raw BatchedSQLQueries type from within tests.

ellemouton · 2025-07-07T07:29:22Z

graph/db/sql_migration.go

+
+		// TODO(elle): At this point, we should check the loaded node
+		// to see if we should extract any DNS addresses from its
+		// opaque type addresses. This is expected to be done in:
+		// https://github.com/lightningnetwork/lnd/pull/9455.
+		// This TODO is being tracked in
+		//  https://github.com/lightningnetwork/lnd/issues/9795 as this
+		// must be addressed before making this code path active in
+		// production.


fyi @mohamedawnallah: this is where the migration will happen for DNS adds currently stored as opaque addrs - which is why we dont need that migration to happen in #9455.

guggero

Awesome test framework! LGTM 🎉

graph/db/sql_migration_test.go

graph/db/sql_migration.go

graph/db/sql_migration_test.go

guggero · 2025-07-07T08:29:26Z

graph/db/sql_migration_test.go

+			name:  "nodes",
+			write: writeUpdate,
+			//nolint:ll
+			objects: []any{


Very nice test framework!

I wonder if it would make sense to use the rapid library to generate some random nodes here as well?
Not sure how much setup code that would require though, so just a thought and definitely non-blocking.

Or perhaps we could use something like this to produce a certain amount of random objects in each test?
https://github.com/lightninglabs/taproot-assets/blob/7ce15016366c38df1670ce6e1928ffa4b30fe9b7/internal/test/copy.go#L15

Same comment, perhaps worth considering (see example in invoices migration tests).

cool, will do - gonna address this in a follow up as a separate test that covers nodes/channels/channel policies just to keep this test deterministic. Added tracking for this in the epic

bhandras

LGTM 🎉

graph/db/sql_migration.go

bhandras · 2025-07-07T10:03:37Z

graph/db/sql_migration_test.go

+			name:  "nodes",
+			write: writeUpdate,
+			//nolint:ll
+			objects: []any{


Same comment, perhaps worth considering (see example in invoices migration tests).

graph/db/sql_migration_test.go

In this commit, the basic framework for the graph SQL migration is added. This sets us up for the commits to follow which will add migration logic for each table in the graph commit by commit.

This commit was inspired by the invoices package TestMigrationWithChannelDB test helper. This test is purely for running locally to test the graph SQL migration. It allows a dev to run the migration against a local `channel.sqlite` or even `channel.db` file.

ellemouton

ultra speed review! thanks yall!

graph/db/sql_migration.go

ellemouton · 2025-07-07T15:22:00Z

graph/db/sql_migration_test.go

+			name:  "nodes",
+			write: writeUpdate,
+			//nolint:ll
+			objects: []any{


cool, will do - gonna address this in a follow up as a separate test that covers nodes/channels/channel policies just to keep this test deterministic. Added tracking for this in the epic

So that we can use it for comparisons in other migrations too.

ellemouton · 2025-07-08T05:11:42Z

cc @guggero for override merge 🙏

ellemouton · 2025-07-08T08:29:38Z

oops - looks like i didnt push the changes properly - will update soon!

This commit expands MigrateGraphToSQL so that it migrates all the graph nodes from kvdb to SQL. The TestMigrateGraphToSQL test is updated to cover the basic LightningNode cases. A new test, `TestSQLMigrationEdgeCases`, is also added and a case is added to tests the edge case where a node exists in our kvdb store that has invalid TLV bytes.

ellemouton self-assigned this Jul 4, 2025

ellemouton added graph migration sql labels Jul 4, 2025

gemini-code-assist bot reviewed Jul 4, 2025

View reviewed changes

graph/db/sql_migration_test.go Show resolved Hide resolved

docs/release-notes/release-notes-0.20.0.md Outdated Show resolved Hide resolved

graph/db/sql_migration.go Show resolved Hide resolved

graph/db/sql_migration.go Show resolved Hide resolved

ellemouton mentioned this pull request Jul 4, 2025

[graph mig 2]: graph/db: migrate graph channels and policies from kvdb to SQL #10037

Merged

ellemouton requested review from bhandras and guggero July 7, 2025 06:31

ellemouton added this to the v0.20.0 milestone Jul 7, 2025

ellemouton added 2 commits July 7, 2025 08:36

graph/db: fix build flag directive

c9a775e

Fix a typo in the build directive.

graph/db: refactor SQL DB creation files

5d7abcd

Factor out the transaction executor construction so that we can have access to the raw BatchedSQLQueries type from within tests.

ellemouton force-pushed the graphMig1-nodes branch from ab4a75e to 0c03a49 Compare July 7, 2025 07:09

ellemouton commented Jul 7, 2025

View reviewed changes

guggero approved these changes Jul 7, 2025

View reviewed changes

bhandras approved these changes Jul 7, 2025

View reviewed changes

ellemouton added 2 commits July 7, 2025 13:34

graph/db: add SQL migration function and test framework

e3572e7

In this commit, the basic framework for the graph SQL migration is added. This sets us up for the commits to follow which will add migration logic for each table in the graph commit by commit.

ellemouton force-pushed the graphMig1-nodes branch from 0c03a49 to f4c4605 Compare July 7, 2025 15:19

ellemouton commented Jul 7, 2025

View reviewed changes

ellemouton mentioned this pull request Jul 7, 2025

[epic]: graph db: SQL implementation & migration #9795

Open

83 tasks

sqldb+invoices: create a re-usable comparion helper function

084563c

So that we can use it for comparisons in other migrations too.

ellemouton force-pushed the graphMig1-nodes branch from f4c4605 to f13874a Compare July 7, 2025 16:42

ellemouton force-pushed the graphMig1-nodes branch from f13874a to d62aa8f Compare July 8, 2025 09:26

ellemouton mentioned this pull request Jul 8, 2025

[graph mig 2]: graph/db: migrate graph channels and policies from kvdb to SQL #10050

Merged

ellemouton added 3 commits July 8, 2025 11:37

graph/db: migrate source node to SQL store

9657707

docs: add release note

cb959bd

ellemouton force-pushed the graphMig1-nodes branch from d62aa8f to cb959bd Compare July 8, 2025 09:37

ellemouton merged commit 11f6648 into lightningnetwork:master Jul 8, 2025
37 of 40 checks passed

ellemouton deleted the graphMig1-nodes branch July 8, 2025 12:15

[graph mig 1]: graph/db: migrate graph nodes from kvdb to SQL #10036

[graph mig 1]: graph/db: migrate graph nodes from kvdb to SQL #10036

Uh oh!

Conversation

ellemouton commented Jul 4, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ellemouton Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

guggero left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guggero Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

guggero Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

bhandras Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

ellemouton Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

bhandras left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bhandras Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ellemouton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ellemouton Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

ellemouton commented Jul 8, 2025

Uh oh!

ellemouton commented Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!