Skip to content

Commit 2f1a7f8

Browse files
arik-soarik-so
committed
use flags for deviations from standards within non-incremental updates
1 parent 60d0c56 commit 2f1a7f8

File tree

3 files changed

+255
-240
lines changed

3 files changed

+255
-240
lines changed

‎lightning-graph-sync/README.md

+47-46
Original file line numberDiff line numberDiff line change
@@ -1,46 +1,57 @@
11
# lightning-graph-sync
22

3-
This crate exposes functionality for rapid gossip graph syncing, aimed primarily
4-
at mobile clients.
3+
This crate exposes functionality for rapid gossip graph syncing, aimed primarily at mobile clients.
54

65
## Mechanism
76

8-
The (presumed) server sends a compressed gossip response containing gossip data.
9-
The gossip data is formatted compactly, omitting signatures and opportunistically
10-
incremental.
7+
The (presumed) server sends a compressed gossip response containing gossip data. The gossip data is formatted compactly,
8+
omitting signatures and opportunistically incremental.
119

1210
Essentially, the serialization structure is as follows:
1311

1412
1. Fixed prefix bytes `76, 68, 75, 1` (the first three bytes are ASCII for `LDK`)
15-
- The purpose of this prefix is to identify the serialization format, should other
16-
rapid gossip sync formats arise in the future.
13+
- The purpose of this prefix is to identify the serialization format, should other rapid gossip sync formats arise
14+
in the future.
1715
- The fourth byte is the protocol version in case our format gets updated
18-
2. A `CompactSize` indicating the number of channel announcement messages to follow
19-
3. `[CustomChannelAnnouncement]` (array of significantly stripped down channel announcements)
20-
4. A `CompactSize` indicating the number of channel update messages to follow
21-
5. `[CustomChannelUpdate]`
16+
2. Chain hash (32 bytes)
17+
3. Latest seen timestamp (`u32`)
18+
4. A `u64` indicating the number of node IDs to follow
19+
5. `[PublicKey]` (array of compressed 33-byte node IDs)
20+
6. A `u64` indicating the number of channel announcement messages to follow
21+
7. `[CustomChannelAnnouncement]` (array of significantly stripped down channel announcements)
22+
8. A `u64` indicating the number of channel update messages to follow
23+
9. A `u8` flagging whether non-incremental updates are present (if yes, it's set to `1`)
24+
- If present, the following values are added:
25+
1. `default_cltv_expiry_delta`: `u16`
26+
2. `default_htlc_minimum_msat`: `u64`
27+
3. `default_fee_base_msat`: `u32`
28+
4. `default_fee_proportional_millionths`: `u32`
29+
5. `default_htlc_maximum_msat`: `u64` (if the default is no maximum, `u64::MAX`)
30+
- The defaults are calculated by the server based on the frequency within non-incremental updates within this
31+
particular message
32+
10. `[CustomChannelUpdate]`
2233

2334
You will also notice that `NodeAnnouncement` messages are skipped altogether.
2435

25-
The data is then applied to the current network graph, artifically backdated 7 days from the current time to
26-
make sure more recent updates obtained directly from gossip are not accidentally overwritten.
36+
The data is then applied to the current network graph, artifically backdated 7 days from the current time to make sure
37+
more recent updates obtained directly from gossip are not accidentally overwritten.
2738

2839
### CustomChannelAnnouncements
2940

30-
To achieve compactness and avoid data repetition, we're sending a significantly stripped down version of the
31-
channel announcement message, which contains only the following data:
41+
To achieve compactness and avoid data repetition, we're sending a significantly stripped down version of the channel
42+
announcement message, which contains only the following data:
3243

3344
1. `channel_features`: `u16` + `n`, where `n` is the number of bytes indicated by the first `u16`
34-
2. `short_channel_id`: `u64`
35-
3. `node_id_1`: `ECPoint`
36-
4. `node_id_2`: `ECPoint`
45+
2. `short_channel_id`: `u64`/`CompactSize` (starts with `u64`, then incremental `CompactSize` deltas)
46+
3. `node_id_1_index`: `CompactSize` (index of node id within the previously sent sequence)
47+
4. `node_id_2_index`: `CompactSize` (index of node id within the previously sent sequence)
3748

3849
### CustomChannelUpdate
3950

40-
For the purpose of rapid syncing, we have deviated from the channel update format
41-
specified in BOLT 7 significantly. Our custom channel updates are structured as follows:
51+
For the purpose of rapid syncing, we have deviated from the channel update format specified in BOLT 7 significantly. Our
52+
custom channel updates are structured as follows:
4253

43-
1. `short_channel_id`: `u64`
54+
1. `short_channel_id`: `u64`/`CompactSize` (starts with `u64`, then incremental `CompactSize` deltas)
4455
2. `custom_channel_flags`: `u8`
4556
3. `update_data`
4657

@@ -50,41 +61,31 @@ Specifically, our custom channel flags break down like this:
5061
|---------------------|----|----|----|---|---|------------------|-----------|
5162
| Incremental update? | | | | | | Disable channel? | Direction |
5263

53-
If the most significant bit is set to `0`, indicating a non-incremental update,
54-
`update_data` is formatted as follows:
55-
56-
1. `cltv_expiry_delta`: `u16`
57-
2. `htlc_minimum_msat`: `u64`
58-
3. `fee_base_msat`: `u32`
59-
4. `fee_proportional_millionths`: `u32`
60-
5. `has_htlc_maximum_msat`: `u8`
61-
6. `htlc_maximum_msat`: `u64` (only if `has_htlc_maximum_msat`'s least significant bit is set to `1`)
62-
63-
If, however, the update is incremental, the intermediate bits of the channel flags
64-
assume new meaning:
64+
If the most significant bit is set to `1`, indicating an incremental update, the intermediate bit flags assume the
65+
following meaning:
6566

6667
| 64 | 32 | 16 | 8 | 4 |
6768
|---------------------------------|---------------------------------|-----------------------------|-------------------------------------------|---------------------------------|
6869
| `cltv_expiry_delta` has changed | `htlc_minimum_msat` has changed | `fee_base_msat` has changed | `fee_proportional_millionths` has changed | `htlc_maximum_msat` has changed |
6970

70-
In this case, `update_data` only contains the fields that are indicated to have changed by the channel flags.
71+
If the most significant bit is set to `0`, the meaning is almost identical, except instead of a change the flags now
72+
represent a deviation from the defaults sent at the beginning of the update sequence.
7173

72-
If `htlc_maximum_msat` has changed, there is an additional `u8` preceding the new value
73-
indicating whether or not a maximum is present.
74+
In both cases, `update_data` only contains the fields that are indicated by the channel flags to be non-standard or to
75+
have changed.
7476

7577
## Delta Calculation
7678

77-
The way a server is meant to calculate this rapid gossip sync data is by using two
78-
data points as a reference that are meant to be provided by the client:
79+
The way a server is meant to calculate this rapid gossip sync data is by using two data points as a reference that are
80+
meant to be provided by the client:
7981
`latest_announcement_blockheight` and `latest_update_timestamp`.
8082

81-
Based on `latest_announcement_blockheight`, the server only sends channel announcements
82-
that occurred at or after that block height.
83+
Based on `latest_announcement_blockheight`, the server only sends channel announcements that occurred at or after that
84+
block height.
8385

84-
Based on `latest_update_timestamp`, the server fetches all channel updates that occurred
85-
at or after the timestamp. Then, the server also checks for each update whether there
86-
had been a previous one prior to the given timestamp.
86+
Based on `latest_update_timestamp`, the server fetches all channel updates that occurred at or after the timestamp.
87+
Then, the server also checks for each update whether there had been a previous one prior to the given timestamp.
8788

88-
If a particular channel update had never occurred before, the full update is sent.
89-
If an channel has had updates prior to the provided timestamp, the latest update
90-
prior to the timestamp is taken as a reference, and the delta is calculated against it.
89+
If a particular channel update had never occurred before, the full update is sent. If an channel has had updates prior
90+
to the provided timestamp, the latest update prior to the timestamp is taken as a reference, and the delta is calculated
91+
against it.

‎lightning-graph-sync/src/lib.rs

+65-66
Original file line numberDiff line numberDiff line change
@@ -30,75 +30,74 @@ pub mod processing;
3030
/// `sync_path`: Path to the file where the gossip update data is located
3131
///
3232
pub fn sync_network_graph_with_file_path(
33-
network_graph: &network_graph::NetworkGraph,
34-
sync_path: &str,
33+
network_graph: &network_graph::NetworkGraph,
34+
sync_path: &str,
3535
) -> Result<(), GraphSyncError> {
36-
let mut file = File::open(sync_path)?;
37-
processing::read_network_graph(&network_graph, &mut file)
36+
let mut file = File::open(sync_path)?;
37+
processing::read_network_graph(&network_graph, &mut file)
3838
}
3939

4040
#[cfg(test)]
4141
mod tests {
42-
use std::fs;
43-
44-
use bitcoin::blockdata::constants::genesis_block;
45-
use bitcoin::Network;
46-
47-
use lightning::routing::network_graph::NetworkGraph;
48-
49-
use crate::sync_network_graph_with_file_path;
50-
51-
#[test]
52-
fn test_sync_from_file() {
53-
// same as incremental_only_update_fails_without_prior_same_direction_updates
54-
let valid_response = vec![
55-
76, 68, 75, 1, 111, 226, 140, 10, 182, 241, 179, 114, 193, 166, 162, 70, 174, 99, 247,
56-
79, 147, 30, 131, 101, 225, 90, 8, 156, 104, 214, 25, 0, 0, 0, 0, 0, 97, 227, 98, 218,
57-
0, 0, 0, 4, 2, 22, 7, 207, 206, 25, 164, 197, 231, 230, 231, 56, 102, 61, 250, 251,
58-
187, 172, 38, 46, 79, 247, 108, 44, 155, 48, 219, 238, 252, 53, 192, 6, 67, 2, 36, 125,
59-
157, 176, 223, 175, 234, 116, 94, 248, 201, 225, 97, 235, 50, 47, 115, 172, 63, 136,
60-
88, 216, 115, 11, 111, 217, 114, 84, 116, 124, 231, 107, 2, 158, 1, 242, 121, 152, 106,
61-
204, 131, 186, 35, 93, 70, 216, 10, 237, 224, 183, 89, 95, 65, 3, 83, 185, 58, 138,
62-
181, 64, 187, 103, 127, 68, 50, 2, 201, 19, 17, 138, 136, 149, 185, 226, 156, 137, 175,
63-
110, 32, 237, 0, 217, 90, 31, 100, 228, 149, 46, 219, 175, 168, 77, 4, 143, 38, 128,
64-
76, 97, 0, 0, 0, 2, 0, 0, 8, 153, 192, 0, 2, 27, 0, 0, 0, 1, 0, 0, 255, 2, 68, 226, 0,
65-
6, 11, 0, 1, 2, 3, 0, 0, 0, 2, 8, 153, 192, 0, 2, 27, 0, 0, 1, 0, 40, 0, 0, 0, 0, 0, 0,
66-
3, 232, 0, 0, 0, 1, 0, 0, 0, 125, 0, 0, 0, 0, 58, 85, 116, 216, 255, 2, 68, 226, 0, 6,
67-
11, 0, 1, 1, 0, 40, 0, 0, 0, 0, 0, 0, 3, 232, 0, 0, 3, 232, 0, 0, 0, 1, 0, 0, 0, 0, 29,
68-
129, 25, 192,
69-
];
70-
71-
fs::create_dir_all("./tmp/graph-sync-tests").unwrap();
72-
fs::write("./tmp/graph-sync-tests/test_data.lngossip", valid_response).unwrap();
73-
74-
let block_hash = genesis_block(Network::Bitcoin).block_hash();
75-
let network_graph = NetworkGraph::new(block_hash);
76-
77-
let before = network_graph.to_string();
78-
assert_eq!(before.len(), 31);
79-
80-
let sync_result = sync_network_graph_with_file_path(
81-
&network_graph,
82-
"./tmp/graph-sync-tests/test_data.lngossip",
83-
);
84-
85-
assert!(sync_result.is_ok());
86-
87-
let after = network_graph.to_string();
88-
assert_eq!(after.len(), 1727);
89-
assert!(
90-
after.contains("021607cfce19a4c5e7e6e738663dfafbbbac262e4ff76c2c9b30dbeefc35c00643:")
91-
);
92-
assert!(
93-
after.contains("02247d9db0dfafea745ef8c9e161eb322f73ac3f8858d8730b6fd97254747ce76b:")
94-
);
95-
assert!(
96-
after.contains("029e01f279986acc83ba235d46d80aede0b7595f410353b93a8ab540bb677f4432:")
97-
);
98-
assert!(
99-
after.contains("02c913118a8895b9e29c89af6e20ed00d95a1f64e4952edbafa84d048f26804c61:")
100-
);
101-
assert!(after.contains("channels: [619737530008010752]"));
102-
assert!(after.contains("channels: [783241506229452801]"));
103-
}
42+
use std::fs;
43+
44+
use bitcoin::blockdata::constants::genesis_block;
45+
use bitcoin::Network;
46+
47+
use lightning::routing::network_graph::NetworkGraph;
48+
49+
use crate::sync_network_graph_with_file_path;
50+
51+
#[test]
52+
fn test_sync_from_file() {
53+
// same as incremental_only_update_fails_without_prior_same_direction_updates
54+
let valid_response = vec![
55+
76, 68, 75, 1, 111, 226, 140, 10, 182, 241, 179, 114, 193, 166, 162, 70, 174, 99, 247,
56+
79, 147, 30, 131, 101, 225, 90, 8, 156, 104, 214, 25, 0, 0, 0, 0, 0, 97, 227, 98, 218,
57+
0, 0, 0, 4, 2, 22, 7, 207, 206, 25, 164, 197, 231, 230, 231, 56, 102, 61, 250, 251,
58+
187, 172, 38, 46, 79, 247, 108, 44, 155, 48, 219, 238, 252, 53, 192, 6, 67, 2, 36, 125,
59+
157, 176, 223, 175, 234, 116, 94, 248, 201, 225, 97, 235, 50, 47, 115, 172, 63, 136,
60+
88, 216, 115, 11, 111, 217, 114, 84, 116, 124, 231, 107, 2, 158, 1, 242, 121, 152, 106,
61+
204, 131, 186, 35, 93, 70, 216, 10, 237, 224, 183, 89, 95, 65, 3, 83, 185, 58, 138,
62+
181, 64, 187, 103, 127, 68, 50, 2, 201, 19, 17, 138, 136, 149, 185, 226, 156, 137, 175,
63+
110, 32, 237, 0, 217, 90, 31, 100, 228, 149, 46, 219, 175, 168, 77, 4, 143, 38, 128,
64+
76, 97, 0, 0, 0, 2, 0, 0, 8, 153, 192, 0, 2, 27, 0, 0, 0, 1, 0, 0, 255, 2, 68, 226, 0,
65+
6, 11, 0, 1, 2, 3, 0, 0, 0, 2, 1, 0, 40, 0, 0, 0, 0, 0, 0, 3, 232, 0, 0, 3, 232, 0, 0,
66+
0, 125, 0, 0, 0, 0, 29, 129, 25, 192, 8, 153, 192, 0, 2, 27, 0, 0, 21, 0, 0, 0, 1, 0,
67+
0, 0, 0, 58, 85, 116, 216, 255, 2, 68, 226, 0, 6, 11, 0, 1, 9, 0, 0, 0, 1,
68+
];
69+
70+
fs::create_dir_all("./tmp/graph-sync-tests").unwrap();
71+
fs::write("./tmp/graph-sync-tests/test_data.lngossip", valid_response).unwrap();
72+
73+
let block_hash = genesis_block(Network::Bitcoin).block_hash();
74+
let network_graph = NetworkGraph::new(block_hash);
75+
76+
let before = network_graph.to_string();
77+
assert_eq!(before.len(), 31);
78+
79+
let sync_result = sync_network_graph_with_file_path(
80+
&network_graph,
81+
"./tmp/graph-sync-tests/test_data.lngossip",
82+
);
83+
84+
assert!(sync_result.is_ok());
85+
86+
let after = network_graph.to_string();
87+
assert_eq!(after.len(), 1727);
88+
assert!(
89+
after.contains("021607cfce19a4c5e7e6e738663dfafbbbac262e4ff76c2c9b30dbeefc35c00643:")
90+
);
91+
assert!(
92+
after.contains("02247d9db0dfafea745ef8c9e161eb322f73ac3f8858d8730b6fd97254747ce76b:")
93+
);
94+
assert!(
95+
after.contains("029e01f279986acc83ba235d46d80aede0b7595f410353b93a8ab540bb677f4432:")
96+
);
97+
assert!(
98+
after.contains("02c913118a8895b9e29c89af6e20ed00d95a1f64e4952edbafa84d048f26804c61:")
99+
);
100+
assert!(after.contains("channels: [619737530008010752]"));
101+
assert!(after.contains("channels: [783241506229452801]"));
102+
}
104103
}

0 commit comments

Comments
 (0)