[epic]: graph db: SQL implementation & migration

[ellemouton]

This is the tracking issue for the graph db SQL-ization project. It will be the go-to reference to see how the project is progressing.

I'll start with a check-list for easy reference & then will add any details & context lower down behind some drop-down tags.

High Level Project Check List

• Prepare the code (exported methods and tests) such that everything is DB agnostic (ie, hide db specific types and behaviour) in preparation for having things ready to be run against a different DB backend.
• Introduce an abstract graph DB interface & let the ChannelGraph use this interface instead of a raw pointer.
• Introduce SQL schema, queries & CRUD
• Implement the kvdb -> SQL migration for the above stores.
• Plug the new migration into the main code base & add itest jobs to run against them (behind a feature flag s.t they are not yet available in prod build).
• Once there is enough confidence in the code & migration, add the SQL impl & migrations to the main prod build

Lower Level Check Lists

1. Prep graph DB code to be ready for new db backend

• Clean-up the graph db tests so that they are ready to be run against a SQL DB impl of the graph store.
  This includes removing any kvdb types and replacing them with more abstract types.
  - graph/db: clean up test code in preparation for DB abtraction #9694
  - graph+lnwire: start validating that extra lnwire msg bytes are valid TLV #9787
  - multi: various test preparations for different graph store impl #9800
  - graph/db: final test preparation before SQL impl #9823
• Ensure that any methods/flows currently not covered by unit tests are properly covered.
  - graph/db: add test coverage for all V1Store methods  #9796
• Prepare the exported graph db methods such that they dont reveal any bbolt specific types
  - graph/db: remove various kvdb parameters from exported methods #9695
  - batch: dont expose kvdb.RwTx in batch.SchedulerOptions #9790
• graph/db: init SQLStore caches and batch schedulers #9853

2. Abstract graph DB interface

• graph/db: introduce a V1Store interface #9791

3. Introduce SQL schema, queries & CRUD

We'll define the schemas, queries and implement the crud required for each of the following data strucutres:

• nodes: sqldb+graph/db: add node related tables and implement some node CRUD #9866
• channels: sqldb+graph/db: add channel tables and implement some channel CRUD #9869
• channel updates
• zombies
• source nodes sqldb+graph/db: add node related tables and implement some node CRUD #9866
• closed SCIDs
• prune log

Note that this will be done in an incremental fashion s.t we are implementing the SQL impl of the new abstract graph store interface method by method. Here are all the methods that will be converted:

• AddLightningNode
• AddrsForNode
• ForEachSourceNodeChannel
• ForEachNodeChannel
• ForEachNodeCached
• ForEachNode
• ForEachNodeCacheable
• LookupAlias
• DeleteLightningNode
• NodeUpdatesInHorizon
• FetchLightningNode
• HasLightningNode
• IsPublicNode
• GraphSession
• ForEachChannel
• DisabledChannelIDs
• AddChannelEdge
• HasChannelEdge
• DeleteChannelEdges
• AddEdgeProof
• ChannelID
• HighestChanID
• ChanUpdatesInHorizon
• FilterKnownChanIDs
• FilterChannelRange
• FetchChanInfos
• FetchChannelEdgesByOutpoint
• FetchChannelEdgesByID
• ChannelView
• MarkEdgeZombie
• MarkEdgeLive
• IsZombieEdge
• NumZombies
• PutClosedScid
• IsClosedScid
• UpdateEdgePolicy
• SourceNode
• SetSourceNode
• PruneTip
• PruneGraphNodes
• PruneGraph
• DisconnectBlockAtHeight

Once all the above is complete we can:

• run the full uint test suite against the SQL graph impls
• run the full itest suite against the SQL graph impls

4. Migration

- write the migration code for the graph store along with a test for it - plug in to code behind a dev feature flag

Planned Schemas:

Nodes

NOTE:

• the block_height column here is just to show how v2 data will fit into the picture. Our initial V1 impl will not include this column

(interact with the schema here)

Channels

NOTES:

1. the merkle_root_hash and schnorr_sig columns are only for demonstrating how future gossip V2 data will fit in to the picture. They wont be included in the initial schema.
2. The nodes table here is not complete - it is just to show how channels link to nodes. See "Nodes" section above for details on the nodes table.

(interact here)

Channel Policies

1. the block_height and disabled_flags columns are just to show how v2 data will fit in. (might split disabled flags into own table? need way to easily query "get all disabled channels in v2")
2. The nodes and channels tables here are not complete - it is just to show how channel_policies link to channels and to nodes. See "Nodes" & "Channels" sections above for details on the nodes and `channels tables.

Other Graph Tables

Extra checklist not to forget (comments during review):

• re-usable NewReadTx/NewWriteTx in sqldb packages: sqldb: re-usable TxOptions and NoOpReset #9873
• re-usable sqldb.NoReset: sqldb: re-usable TxOptions and NoOpReset #9873
• ensure that node_addr normalisation is ok as is - check performance & where full node read is used.