Traffic management module to tame large meshes

[h3lix1]

Summary

As meshes get larger, including 0-hop cost routing, the amount of broadcast traffic a mesh needs to handle increases a lot. While the mesh I'm apart of (bayme.sh) has migrated to faster presets, we are likely to stay on MediumFast for the forseeable furture.

As an alternative to faster presets, I propose a few changes to the code that will implement a "Traffic Management" module. This module will give router owners (and to a degree client owners) the ability to reduce the hops for certain traffic types, and reduce traffic utilization in the process.

The features..

Creation of a node 'cache' entry

Given the heavy weight of the nodedb, increasing the size is not easilly possible without causing problems with the client or other services. The module, when enabled, will create an entry for nodes it hears as a cache, with slight differences between ESP32 and NRF52 that will be discussed later.

For esp32, it looks like this

struct UnifiedCacheEntry {
    NodeNum node;              // 4 bytes - Node identifier (0 = empty)
    int32_t lat_truncated;     // 4 bytes - Precision-truncated latitude
    int32_t lon_truncated;     // 4 bytes - Precision-truncated longitude
    uint16_t pos_time_secs;    // 2 bytes - Seconds since epoch for position
    uint16_t rate_window_secs; // 2 bytes - Window start for rate limiting
    uint16_t unknown_secs;     // 2 bytes - Window start for unknown tracking
    uint8_t rate_count;        // 1 byte  - Packet count (saturates at 255)
    uint8_t unknown_count;     // 1 byte  - Unknown packet count
};
static_assert(sizeof(UnifiedCacheEntry) == 20);

For nrf52 it looks like...

struct UnifiedCacheEntry {
    NodeNum node;              // 4 bytes - Node identifier (0 = empty)
    uint8_t pos_hash;          // 1 byte  - 8-bit hash of truncated lat/lon
    uint8_t pos_time_mins;     // 1 byte  - Minutes since epoch (~4h range)
    uint16_t rate_window_secs; // 2 bytes - Window start for rate limiting
    uint16_t unknown_secs;     // 2 bytes - Window start for unknown tracking
    uint8_t rate_count;        // 1 byte  - Packet count (saturates at 255)
    uint8_t unknown_count;     // 1 byte  - Unknown packet count
};
static_assert(sizeof(UnifiedCacheEntry) == 12);

The expectation is to move from 20 bytes per node in the cache to 12 for memory constrained devices. This is accomplished saving positions as 8 bit hashes. As for the different fields, those are spoilers for later.

Position Deduplication

This is fairly straightforward. If a position for a node hasn't changed a configurable distance since the last time it has sent a posiion, don't repeat it.

This is implemented for both esp32 and nrf52 nodes, but with subtle differences for both. ESP32 will save the full position data in PSRAM but nrf52 will save a hash of the data in DRAM. This allows for the module to store 1024 nodes with about 12KB of memory for nrf52 and 20KB of memory for esp32.

The hash is only 256 entries, so there is a chance of a hash collisions for nrf52, but given the number of solar routers in the wild, this seems an acceiptable compromise, and the chance is really low. (0.4%)

The configs for this would be

traffic_management.position_dedup_enabled  (true/false)
traffic_management.position_precision_bits  (1-32)
traffic_management.position_min_interval_secs (seconds between intervals)

Nodeinfo Direct Response

If a node exists in the router's nodeinfo database, spoof a response to the requestor with nodeinfo.

This is meant for roof nodes or local nodes that may have a full nodedb table, but someone local (between 0-3 hops away for routers, only 0 hops away for clients) will respond back with the information (with 0 max_hops). The expectation is that the message will not need to traverse the network to get this information from the remote node.

The configs for this would be

traffic_management.nodeinfo_direct_response (true/false)
traffic_management.nodeinfo_direct_response_max_hops (1-3 for routers, 0 for clients)

Rate Limiting

This is about as direct as one can expect. It rate limits how much a node can send at one time with multiple types of rate limiting.

First is rate limit packets in general for a timeframe. Any packets.
Second is to limit packets for unknown traffic (encrypted, unknown) traffic for a timeframe.

The expectation here is to make it fair for everyone to have a share of airtime. If a node is making too many requests, don't forward it.

The configs for this would be

traffic_management.rate_limit_enabled (true/false)
traffic_management.rate_limit_window_secs (1-65535)
traffic_management.drop_unknown_enabled (true/false)
traffic_management.unknown_packet_threshold (1-255)

Both unknown and known packets use the same rate_limit_window_secs value to determine time.

Exhausting Hops (Telemetry and Position)

The protocol leaves the number of hops open for modification as part of the protobuf. This code uses that to help exhaust the hops on the packet for telemetry and position data when it re-sends the message.

The expectation is to send the message with hop_limit = 0 for these packets.

When local nodes receive the packets, it is added to recentPackets, and nodes will not re-send the packet even if another packet comes in with hop_limit >= 1. This effectively stops other nodes from repeating this packet fruther.

The configs for this would be

traffic_management.exhaust_hop_telemetry (true/false)
traffic_management.exhaust_hop_position (true/false)

Preserving Hops

Last, but not least, the feature to preserve hops for messages. While many of the above optoins are to reduce traffic, this one allows all traffic to pass without decreasing hop_limit, allowing for packets to go further. This was a very popular feature for meshes covering long distgances, but it required manually adding each node individually. There was also discussion about routers accidentally favoriting other nodes, allowing packets to travel further than the network operators would like them to travel. This is a single switch to turn on or off this feature.

A router in this instance is either ROUTER, ROUTER_LATE, or CLIENT_BASE. It has many of the same rules as the 0-cost hop route feature, but it allows for CLIENT_BASE to not decrement hops for incoming traffic as well.

The config for this would be

traffic_management.router_preserve_hops (true/false)

Finally...

For all the above, I'm interested in keeping all the changes necessary within the module code itself. If something breaks or does not work as intended, it is easy to turn off the module.

There is still work to be done for this. For example, it might be best to treat esp32 and nrf52 noders similiarly to keep the behavior consistent between the two platforms. This will also cut down on the amount of code needed to treat each one differently.

Some default timeouts need to be set with reasonable values. Right now, they're placeholders.

This needs testing. I have the code running on a couple routers today with expected results, but not all meshes are the same.

PRs

the code PRs to support this change are still in draft (except for the protobufs - need to get that one in first)

#9358
meshtastic/protobufs#849
meshtastic/python#890

[stephen304]

The preserving hops ideas is really interesting to me since I have a well positioned client base that seems to see a lot more nodes and get a lot more public messages than my favorited handheld - I assume this would fix that by ensuring my handheld gets all of the same info as the base? And if I understand correctly that would also allow you to not incur a 1 hop penalty when sending out messages through the client base as well?

[h3lix1]

If you can favorite your local routers, https://meshtastic.org/blog/zero-cost-hops-favorite-routers/ might work for you as well, but it won't forward many broadcast packets. I'd say your best bet might be ROUTER_LATE If your indoor nodes can't hear the local routers. (Or put your client base on wifi and use that to connect via android/ios.. but it does require an esp32 node)

[TheDomanNow]

Hello h3lix1!

I have some nodes in prime locations in big city. We are still on MEDIUM_FAST, and with a reason - on SHORT_FAST we started to loose a lot of range and reliability. In few places ChUtil is up to 50%, but normally it's 25-35%. We have some spammers. So... I'm VERY interested in traffic management and grateful that someone did this. However, as it stands it's hard to implement something that I don't understand and the information is... sparse, and for a person that didn't write the code and didn't do any reverse engineering... it's almost black magic ;-)
I'm really hoping that you could clarify some things.

Let's start from the beginning. To simplify things; node A = my node with traffic management enabled, node B = some random client node.

Turn on module.
traffic_management.enabled: true
That's simple.

Nodeinfo Direct Response. Set:

nodeinfo_direct_response: true
nodeinfo_direct_response_max_hops: 1

Do I understand correctly that; if my A node will hear that B node that's 1 hop away wants NodeInfo of some far away node, my A node will step in and spoof the packet and send requested NodeInfo by itself?

Position Deduplication. Set:

position_dedup_enabled: true
position_precision_bits: 15
position_min_interval_secs: 60

Do I understand correctly that; if node B sends position every 45 seconds and/or position within 15 bit range compared to the last recorded position, my A node will ignore and drop this packet?
Do I understand correctly that; if node B sends position every 90 seconds and/or position outside 15 bit range compared to the last recorded position, my A node will accept and forward this packet?

Exhausting Hops (Telemetry and Position). Set:

exhaust_hop_position: true
exhaust_hop_telemetry: true

Do I understand correctly that; with this setting my A node just forwards a packet with 0 hop set, regardless of everything else?

Rate Limiting. Set:

rate_limit_enabled: true
rate_limit_max_packets: 10
rate_limit_window_secs: 60
drop_unknown_enabled: true
unknown_packet_threshold: 1

Do I understand correctly that; if node B sends 13 packets in 60 seconds window, my A node will ignore and drop 4 last packets? Meaning that my A node will accept packet traffic up to 9 packets per 60 seconds?
Do I understand correctly that; if node B sends even 1 unknown packet in 60 seconds window, my A node will ignore and drop this packet?
What values should I set if I want to limit B node to 250 packets per 24h?
What values should I set if I want to completely block B node from sending unknown packets?
What values should I set if I want to limit B node to sending max 10 unknown packets in 24h?
Unknown packet = encrypted or unknown from other channels that I don't have?

Preserving Hops. Set:
router_preserve_hops: true
How does this work; roles ROUTER, ROUTER_LATE, CLIENT_BASE will just always leave hops as they came (no decrease in hop_limit), or does the CLIENT_BASE still require favorite nodes to do this?

Uff... I know that's a lot of questions...

Best regards, Doman.

PS: I'm on FW 2.7.20 Alpha

[h3lix1]

HI @TheDomanNow - thank you checking this out. :)

The documentation is pending a human to re-write it, but the MR for that is here.. It should explain everything..

https://github.com/meshtastic/meshtastic/pull/2295/changes

Your intuition is really good at this without having documentation. Well done.

[h3lix1]

Thre was a blog post for this, too https://github.com/meshtastic/meshtastic/pull/2297/changes