feat: tiered data storage

.github/workflows/rust.yml

@@ -80,11 +80,11 @@ jobs:
       - name: Test on Rust ${{ matrix.toolchain }}
         if: "matrix.platform != 'windows-latest'"
         run: |
-          RUSTFLAGS="--cfg no_download" cargo test
+          RUSTFLAGS="--cfg no_download --cfg cycle_tests" cargo test
       - name: Test with UniFFI support on Rust ${{ matrix.toolchain }}
         if: "matrix.platform != 'windows-latest' && matrix.build-uniffi"
         run: |
-          RUSTFLAGS="--cfg no_download" cargo test --features uniffi
+          RUSTFLAGS="--cfg no_download --cfg cycle_tests" cargo test --features uniffi
 
   doc:
     name: Documentation

.github/workflows/semver.yml

@@ -7,9 +7,5 @@ jobs:
     steps:
       - name: Checkout source code
         uses: actions/checkout@v4
-      - name: Install Rust stable toolchain
-        run: |
-          curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile=minimal --default-toolchain stable
-          rustup override set stable
       - name: Check SemVer
         uses: obi1kenobi/cargo-semver-checks-action@v2

.github/workflows/vss-integration.yml

@@ -45,4 +45,4 @@ jobs:
           cd ldk-node
           export TEST_VSS_BASE_URL="http://localhost:8080/vss"
           RUSTFLAGS="--cfg vss_test" cargo test io::vss_store
-          RUSTFLAGS="--cfg vss_test" cargo test --test integration_tests_vss
+          RUSTFLAGS="--cfg vss_test --cfg cycle_tests" cargo test --test integration_tests_vss

CLAUDE.md

@@ -0,0 +1,67 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Development Rules
+
+- Always ensure tests pass before committing.
+- Run `cargo fmt --all` after every code change
+- Never add new dependencies unless explicitly requested
+- Please always disclose the use of any AI tools in commit messages and PR descriptions
+- When adding new `.rs` files, please ensure to always add the licensing header as found in all other files.
+
+## Architecture Overview
+
+LDK-Node is a self-custodial Lightning Network node library built on top of **LDK** (Lightning Development Kit) for Lightning functionality and **BDK** (Bitcoin Development Kit) for on-chain wallet operations. It provides a simple, ready-to-go interface for building Lightning applications with language bindings for Swift, Kotlin, and Python via UniFFI.
+
+### Core Components
+
+| Component | Location | Responsibility |
+|-----------|----------|----------------|
+| `Node` | `src/lib.rs` | Central abstraction containing all subsystems; entry point for API |
+| `Builder` | `src/builder.rs` | Fluent configuration interface for constructing `Node` instances |
+| `Wallet` | `src/wallet/` | BDK-based on-chain wallet with SQLite persistence |
+| `ChainSource` | `src/chain/` | Chain data abstraction (Esplora, Electrum, Bitcoin Core) |
+| `EventHandler` | `src/event.rs` | Translates LDK events to user-facing `Node` events |
+| `PaymentStore` | `src/payment/store.rs` | Persistent payment tracking with status and metadata |
+
+### Module Organization
+
+| Module | Purpose |
+|--------|---------|
+| `payment/` | Payment processing (BOLT11, BOLT12, on-chain, spontaneous, unified) |
+| `wallet/` | On-chain wallet abstraction, serialization, persistence |
+| `chain/` | Chain data sources, wallet syncing, transaction broadcasting |
+| `io/` | Persistence layer (`SQLiteStore`, `VssStore`, KV-store utilities) |
+| `liquidity.rs` | Liquidity provider integration (LSPS1/LSPS2) |
+| `connection.rs` | Peer connection management and reconnection logic |
+| `gossip.rs` | Gossip data source management (RGS, P2P) |
+| `graph.rs` | Network graph querying and channel/node information |
+| `types.rs` | Type aliases for LDK components (`ChannelManager`, `PeerManager`, etc.) |
+| `ffi/` | UniFFI bindings for cross-language support |
+
+### Key Design Patterns
+
+- **Arc-based Shared Ownership**: Extensive use of `Arc` for thread-safe shared components enabling background task spawning
+- **Event-Driven Architecture**: Events flow from LDK → `EventHandler` → `EventQueue` → User application
+- **Trait-based Abstraction**: `KVStore`/`KVStoreSync` for storage, `ChainSource` for chain backends, `StorableObject` for persistence
+- **Builder Pattern**: Fluent configuration with sensible defaults and validation during build phase
+- **Background Tasks**: Multiple categories (wallet sync, gossip updates, peer reconnection, fee updates, event processing)
+
+### Lifecycle
+
+**Startup (`node.start()`)**: Acquires lock → starts chain source → updates fee rates → spawns background sync tasks → sets up gossip/listeners/peer reconnection → starts event processor → marks running
+
+**Shutdown (`node.stop()`)**: Acquires lock → signals stop → aborts cancellable tasks → waits for background tasks → disconnects peers → persists final state
+
+### Type Aliases (from `types.rs`)
+
+Key LDK type aliases used throughout the codebase:
+- `ChannelManager` - LDK channel management
+- `ChainMonitor` - LDK chain monitoring
+- `PeerManager` - LDK peer connections
+- `OnionMessenger` - LDK onion messaging
+- `Router` - LDK pathfinding (`DefaultRouter`)
+- `Scorer` - Combined probabilistic + external scoring
+- `Graph` - `NetworkGraph`
+- `Sweeper` - `OutputSweeper`

Cargo.toml

@@ -39,30 +39,31 @@ default = []
 #lightning-liquidity = { version = "0.2.0", features = ["std"] }
 #lightning-macros = { version = "0.2.0" }
 
-lightning = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "5bf0d1e2427d759fc1ba4108ddc7e9b32e8bacfc", features = ["std"] }
-lightning-types = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "5bf0d1e2427d759fc1ba4108ddc7e9b32e8bacfc" }
-lightning-invoice = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "5bf0d1e2427d759fc1ba4108ddc7e9b32e8bacfc", features = ["std"] }
-lightning-net-tokio = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "5bf0d1e2427d759fc1ba4108ddc7e9b32e8bacfc" }
-lightning-persister = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "5bf0d1e2427d759fc1ba4108ddc7e9b32e8bacfc", features = ["tokio"] }
-lightning-background-processor = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "5bf0d1e2427d759fc1ba4108ddc7e9b32e8bacfc" }
-lightning-rapid-gossip-sync = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "5bf0d1e2427d759fc1ba4108ddc7e9b32e8bacfc" }
-lightning-block-sync = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "5bf0d1e2427d759fc1ba4108ddc7e9b32e8bacfc", features = ["rest-client", "rpc-client", "tokio"] }
-lightning-transaction-sync = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "5bf0d1e2427d759fc1ba4108ddc7e9b32e8bacfc", features = ["esplora-async-https", "time", "electrum-rustls-ring"] }
-lightning-liquidity = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "5bf0d1e2427d759fc1ba4108ddc7e9b32e8bacfc", features = ["std"] }
-lightning-macros = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "5bf0d1e2427d759fc1ba4108ddc7e9b32e8bacfc" }
+lightning = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "7fe3268475551b0664d315bfbc860416ca8fc774", features = ["std"] }
+lightning-types = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "7fe3268475551b0664d315bfbc860416ca8fc774" }
+lightning-invoice = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "7fe3268475551b0664d315bfbc860416ca8fc774", features = ["std"] }
+lightning-net-tokio = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "7fe3268475551b0664d315bfbc860416ca8fc774" }
+lightning-persister = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "7fe3268475551b0664d315bfbc860416ca8fc774", features = ["tokio"] }
+lightning-background-processor = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "7fe3268475551b0664d315bfbc860416ca8fc774" }
+lightning-rapid-gossip-sync = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "7fe3268475551b0664d315bfbc860416ca8fc774" }
+lightning-block-sync = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "7fe3268475551b0664d315bfbc860416ca8fc774", features = ["rest-client", "rpc-client", "tokio"] }
+lightning-transaction-sync = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "7fe3268475551b0664d315bfbc860416ca8fc774", features = ["esplora-async-https", "time", "electrum-rustls-ring"] }
+lightning-liquidity = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "7fe3268475551b0664d315bfbc860416ca8fc774", features = ["std"] }
+lightning-macros = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "7fe3268475551b0664d315bfbc860416ca8fc774" }
 
 bdk_chain = { version = "0.23.0", default-features = false, features = ["std"] }
 bdk_esplora = { version = "0.22.0", default-features = false, features = ["async-https-rustls", "tokio"]}
 bdk_electrum = { version = "0.23.0", default-features = false, features = ["use-rustls-ring"]}
-bdk_wallet = { version = "2.2.0", default-features = false, features = ["std", "keys-bip39"]}
+bdk_wallet = { version = "2.3.0", default-features = false, features = ["std", "keys-bip39"]}
 
-reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
+bitreq = { version = "0.3", default-features = false, features = ["async-https"] }
 rustls = { version = "0.23", default-features = false }
 rusqlite = { version = "0.31.0", features = ["bundled"] }
 bitcoin = "0.32.7"
 bip39 = { version = "2.0.0", features = ["rand"] }
 bip21 = { version = "0.5", features = ["std"], default-features = false }
 
+async-trait = {version = "0.1.89"}
 base64 = { version = "0.22.1", default-features = false, features = ["std"] }
 rand = { version = "0.9.2", default-features = false, features = ["std", "thread_rng", "os_rng"] }
 chrono = { version = "0.4", default-features = false, features = ["clock"] }
@@ -77,15 +78,18 @@ log = { version = "0.4.22", default-features = false, features = ["std"]}
 
 vss-client = { package = "vss-client-ng", version = "0.4" }
 prost = { version = "0.11.6", default-features = false}
+#bitcoin-payment-instructions = { version = "0.6" }
+bitcoin-payment-instructions = { git = "https://github.com/tnull/bitcoin-payment-instructions", rev = "b9f9991b42e9d71b3ca966818a93b158cf8f6c40" }
 
 [target.'cfg(windows)'.dependencies]
 winapi = { version = "0.3", features = ["winbase"] }
 
 [dev-dependencies]
-lightning = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "5bf0d1e2427d759fc1ba4108ddc7e9b32e8bacfc", features = ["std", "_test_utils"] }
+lightning = { git = "https://github.com/lightningdevkit/rust-lightning", rev = "7fe3268475551b0664d315bfbc860416ca8fc774", features = ["std", "_test_utils"] }
 proptest = "1.0.0"
 regex = "1.5.6"
 criterion = { version = "0.7.0", features = ["async_tokio"] }
+ldk-node-062 = { package = "ldk-node", version = "=0.6.2" }
 
 [target.'cfg(not(no_download))'.dev-dependencies]
 electrsd = { version = "0.36.1", default-features = false, features = ["legacy", "esplora_a33e97e1", "corepc-node_27_2"] }
@@ -101,9 +105,6 @@ clightningrpc = { version = "0.3.0-beta.8", default-features = false }
 lnd_grpc_rust = { version = "2.10.0", default-features = false }
 tokio = { version = "1.37", features = ["fs"] }
 
-[target.'cfg(vss_test)'.dev-dependencies]
-ldk-node-062 = { package = "ldk-node", version = "=0.6.2" }
-
 [build-dependencies]
 uniffi = { version = "0.28.3", features = ["build"], optional = true }
 
@@ -122,6 +123,7 @@ check-cfg = [
 	"cfg(tokio_unstable)",
 	"cfg(cln_test)",
 	"cfg(lnd_test)",
+	"cfg(cycle_tests)",
 ]
 
 [[bench]]
@@ -167,3 +169,16 @@ harness = false
 
 #vss-client-ng = { path = "../vss-client" }
 #vss-client-ng = { git = "https://github.com/lightningdevkit/vss-client", branch = "main" }
+#
+#[patch."https://github.com/lightningdevkit/rust-lightning"]
+#lightning = { path = "../rust-lightning/lightning" }
+#lightning-types = { path = "../rust-lightning/lightning-types" }
+#lightning-invoice = { path = "../rust-lightning/lightning-invoice" }
+#lightning-net-tokio = { path = "../rust-lightning/lightning-net-tokio" }
+#lightning-persister = { path = "../rust-lightning/lightning-persister" }
+#lightning-background-processor = { path = "../rust-lightning/lightning-background-processor" }
+#lightning-rapid-gossip-sync = { path = "../rust-lightning/lightning-rapid-gossip-sync" }
+#lightning-block-sync = { path = "../rust-lightning/lightning-block-sync" }
+#lightning-transaction-sync = { path = "../rust-lightning/lightning-transaction-sync" }
+#lightning-liquidity = { path = "../rust-lightning/lightning-liquidity" }
+#lightning-macros = { path = "../rust-lightning/lightning-macros" }

README.md

@@ -67,7 +67,7 @@ LDK Node currently comes with a decidedly opinionated set of design choices:
 - Entropy for the Lightning and on-chain wallets may be sourced from raw bytes or a [BIP39](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki) mnemonic. In addition, LDK Node offers the means to generate and persist the entropy bytes to disk.
 
 ## Language Support
-LDK Node itself is written in [Rust][rust] and may therefore be natively added as a library dependency to any `std` Rust program. However, beyond its Rust API it also offers language bindings for [Swift][swift], [Kotlin][kotlin], and [Python][python] based on the [UniFFI](https://github.com/mozilla/uniffi-rs/). Moreover, [Flutter bindings][flutter_bindings] are also available.
+LDK Node itself is written in [Rust][rust] and may therefore be natively added as a library dependency to any `std` Rust program. However, beyond its Rust API it also offers language bindings for [Swift][swift], [Kotlin][kotlin], and [Python][python] based on the [UniFFI](https://github.com/mozilla/uniffi-rs/).
 
 ## MSRV
 The Minimum Supported Rust Version (MSRV) is currently 1.85.0.
@@ -85,4 +85,3 @@ The Minimum Supported Rust Version (MSRV) is currently 1.85.0.
 [swift]: https://www.swift.org/
 [kotlin]: https://kotlinlang.org/
 [python]: https://www.python.org/
-[flutter_bindings]: https://github.com/LtbLightning/ldk-node-flutter

benches/payments.rs

@@ -127,6 +127,7 @@ fn payment_benchmark(c: &mut Criterion) {
 		true,
 		false,
 		common::TestStoreType::Sqlite,
+		common::TestStoreType::Sqlite,
 	);
 
 	let runtime =