Unicast and other networking improvements (UBC-Thunderbots#3417)

* project vimrc for FZF

* filter out .swp files

* udp listener refactoring in-progress

* in-progress: add latency_tester module

* latency scripts compile

* formatting

* Revert "Logger now creates a log directory if doesn't exist (UBC-Thunderbots#3143)"

This reverts commit 0e8bf7e.

* cleanup test script

* wip

* add tracy symbols

* working on ansibleing

* init

* wip

* wip

* mayhaps it works

* wip

* wip

* wip

* wip

* debugging

* should clean this up

* wip

* various thunderscope improvements

* add some unit tests

* wip

* wip

* wip

* wip

* wip & cleanup

* wip

* cleanup

* revert unintentional change

* [pre-commit.ci lite] apply automatic fixes

* bug fix and update getting_started guide

* [pre-commit.ci lite] apply automatic fixes

* formatting + fix CI

* [pre-commit.ci lite] apply automatic fixes

* address PR comments

* [pre-commit.ci lite] apply automatic fixes

* [pre-commit.ci lite] apply automatic fixes

* update getting started

* further improve docs

* doc cleanup

* simplify getting_started more

* remove unnecessary socket closing

* cleanup edge case handling of robot_communication binding

* [pre-commit.ci lite] apply automatic fixes

* cleanup edge cases

* formatting

* cleanup logic

* simplify logic for selecting interface

* further simplify selection logic

* remove redundant caching

* delete duplicated code

* [pre-commit.ci lite] apply automatic fixes

* address PR comments

* update documentation for getLocalIp

* [pre-commit.ci lite] apply automatic fixes

* add initial delay before starting the test

* maybe fix ci

* [pre-commit.ci lite] apply automatic fixes

* standardize latency testing

* add unicast support to benchmarking scripts

* [pre-commit.ci lite] apply automatic fixes

* wip

* fix communication bug

* Add documentation, tests and cleanup code in benchmarking_utils

* Add documentation and cleanup some code

* initial changes for unicast comm support

* WIP network service implementation

* network service implemented

* wip python changes

* begin outlines of wifi communication manager

* wip

* wip refactor RobotCommunication to use new WifiNetworkManager module

* wip

* wip but Thunderscope doesn't yet run

* wip debugged thunderscope and seems to work

* working on host mode

* Finish up changes

* constexpr evaluation in Thunderloop constructor

* doc update

* Change order of tracy calls

* [pre-commit.ci lite] apply automatic fixes

* debugging integration problems

* seems to work?

* wip

* wip

* wip

* formatting yo!

* update ansible

* getting ready for PR

* docs for benchmarking utils

* ci + documentation

* [pre-commit.ci lite] apply automatic fixes

* fix ansible linting

* wip not yet done

* [pre-commit.ci lite] apply automatic fixes

* wip

* [pre-commit.ci lite] apply automatic fixes

* wip

* [pre-commit.ci lite] apply automatic fixes

* C++ appears to compile, working through Python

* [pre-commit.ci lite] apply automatic fixes

* update wifi_communication_manager

* some cleanup + stray forgotten type hinting return types

* wip

* [pre-commit.ci lite] apply automatic fixes

* fixing network log listener

* [pre-commit.ci lite] apply automatic fixes

* wip

* [pre-commit.ci lite] apply automatic fixes

* network log listener is finally done

* wip

* fix broken compile and tests

* [pre-commit.ci lite] apply automatic fixes

* minor nit from field testing

---------

Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com>

Author: itsarune

docs/getting-started.md

@@ -230,13 +230,11 @@ Now that you're setup, if you can run it on the command line, you can run it in
 
     - If we want to run it with real robots:
         - Open your terminal, `cd` into `Software/src` and run `ifconfig`.
-            - Pick the network interface you would like to use:
-                1. If you are running things locally, you can pick any interface that is not `lo`
-                2. If you would like to communicate with robots on the network, make sure to select the interface that is connected to the same network as the robots.
+            - Pick the network interface you would like to use. If you would like to communicate with robots on the network, make sure to select the interface that is connected to the same network as the robots.
             - For example, on a sample machine, the output may look like this:
 
                 ```
-                enp0s5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
+                wlp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
                         ...
                         [omitted]
                         ...
@@ -247,22 +245,30 @@ Now that you're setup, if you can run it on the command line, you can run it in
                         ...
                 ```
 
-            - An appropriate interface we could choose is `enp0s5`
+            - An appropriate interface we could choose is `wlp3s0`
+            - Hint: If you are using a wired connection, the interface will likely start with `e-`. If you are using a WiFi connection, the interface will likely start with `w-`.
         - If we are running the AI as "blue": `./tbots.py run thunderscope_main --interface=[interface_here] --run_blue`
         - If we are running the AI as "yellow": `./tbots.py run thunderscope_main --interface=[interface_here] --run_yellow`
         - `[interface_here]` corresponds to the `ifconfig` interfaces seen in the previous step
-            - For instance, a call to run the AI as blue on wifi could be: `./tbots.py run thunderscope_main --interface=enp0s5 --run_blue`
+            - For instance, a call to run the AI as blue on WiFi could be: `./tbots.py run thunderscope_main --interface=wlp3s0 --run_blue`. This will start Thunderscope and set up communication with robots over the wifi interface. It will also listen for referee and vision messages on the same interface.
+        - **Note: You do not need to include the `--interface=[interface_here]` argument!** You can run Thunderscope without it and use the dynamic configuration widget to set the interfaces for communication to send and receive robot, vision and referee messages.
+            - If you choose to include `--interface=[interface_here]` argument, Thunderscope will listen for and send robot messages on this port as well as receive vision and referee messages.
+            - Using the dynamic configuration widget is recommended at Robocup. To reduce latencies, it is recommended to connect the robot router to the AI computer via ethernet and use a separate ethernet connection to receive vision and referee messages. In this configuration, Thunderscope will need to bind to two different interfaces, each likely starting with a "e-".
+            - If you have specified `--run_blue` or `--run_yellow`, navigate to the "Parameters" widget. In "ai_config" > "ai_control_config" > "network_config", you can set the appropriate interface using the dropdowns for robot, vision and referee message communication.
         - This command will set up robot communication and the Unix full system binary context manager. The Unix full system context manager hooks up our AI, Backend and SensorFusion
 2. Run AI along with Robot Diagnostics:
     - The Mechanical and Electrical sub-teams use Robot Diagnostics to test specific parts of the Robot.
     - If we want to run with one AI and Diagnostics
-        - `./tbots.py run thunderscope_main [--run_blue | --run_yellow] --run_diagnostics` will start Thunderscope
+        - `./tbots.py run thunderscope_main [--run_blue | --run_yellow] --run_diagnostics --interface=[interface_here]` will start Thunderscope
             - `[--run_blue | --run_yellow]` indicate which FullSystem to run
             - `--run_diagnostics` indicates if diagnostics should be loaded as well
         - Initially, the robots are all connected to the AI and only receive input from it
         - To change the input source for the robot, use the drop-down menu of that robot to change it between None, AI, and Manual
         - None means the robots are receiving no commands
         - More info about Manual control below
+        - `--interface=[interface_here]` corresponds to the `ifconfig` interfaces seen in the previous step
+            - For instance, a call to run the AI as blue on WiFi could be: `./tbots.py run thunderscope_main --interface=wlp3s0 --run_blue --run_diagnostics`
+            - The `--interface` flag is optional. If you do not include it, you can set the interface in the dynamic configuration widget. See above for how to set the interface in the dynamic configuration widget.
 3. Run only Diagnostics
     - To run just Diagnostics
         - `./tbots.py run thunderscope --run_diagnostics --interface <network_interface>`

docs/wifi-communication-readme.md

@@ -0,0 +1,69 @@
+# WiFi Communication README
+
+## Lessons Learned So Far
+1. For optimal performance, make sure that any packets sent over the network are below MTU size (1500 bytes). Packets larger than MTU size require multiple transmissions for a single send event and multiple retransmissions in the case of packet loss. Overall, these packets contribute to greater utilization of the network and increased latency.
+2. Connect the host computer to the network via ethernet cable when possible. By minimizing the utilization of the WiFi network, this change significantly improves the round-trip time.
+3. Use unicast communication over WiFi for frequent, low-latency communication over multicast. [RFC 9119](https://www.rfc-editor.org/rfc/rfc9119.html#section-3.1.2) provides a good overview of the limitations of multicast communication over WiFi. In short, routers are forced to transmit at the lowest common data rate of all devices on the network to ensure that all devices receive the packet, meaning that the network is slowed down by the slowest device. In addition, router features such as Multiple Input Multiple Output (MIMO) may not be available when using multicast communication. We have found a 24% improvement in round-trip time when switching from multicast to unicast communication with some benchmarking tests.
+4. On embedded Linux devices, WiFi power management seems to cause significant latency spikes. To disable power management, run the following command: `sudo iw dev {wifi_interface} set power_save off` where `{wifi_interface}` is the name of the WiFi interface (e.g. `wlan0`).
+
+
+## Debugging
+
+We have built some tools to help diagnose network latency problems without the confounding effects of running Thunderloop and Thunderscope and their associated overheads.
+
+The latency tester tests the round trip time between two nodes. The primary node sends a message to the secondary node, which sends back the same message as soon as it receives it. The primary node then measures the round trip time.
+
+Typically, the primary node is the host computer and the secondary node is the robot.
+
+## Running the latency tester with the robot
+### Prerequisites
+You must know:
+- The IP address of the robot. We will refer to this address as `{robot_ip}`.
+- The WiFi interface of the robot. We will refer to this interface as `{robot_wifi_interface}`. This interface is typically found by running `ifconfig` or `ip a` on the robot.
+- The network interface of the host computer. We will refer to this interface as `{host_interface}`. This interface is typically found by running `ifconfig` or `ip a` on the host computer.
+
+1. Build the latency tester secondary node: `./tbots.py build latency_tester_secondary_node --platforms=//cc_toolchain:robot`
+2. Copy the binary to the robot: `scp bazel-bin/software/networking/benchmarking_utils/latency_tester_secondary_node robot@{robot_ip}:/home/robot/latency_tester_secondary_node`
+3. SSH into the robot: `ssh robot@{robot_ip}`
+4. There are two test modes: multicast or unicast
+    1. For multicast:
+        1. Run the latency tester secondary node: `./latency_tester_secondary_node --interface {robot_wifi_interface}`
+            - You may optionally also provide the following arguments:
+                - `--runtime_dir` to specify the directory where log files are stored
+                - `--listen_port` to specify the port on which the secondary node listens for messages.
+                - `--send_port` to specify the port on which the secondary node sends messages
+                - `--listen_channel` to specify the channel on which the secondary node listens for messages
+                - `--send_channel` to specify the channel on which the secondary node sends back replies
+        2. On a different terminal on the host computer, run the latency tester primary node: `./tbots.py run latency_tester_primary_node -- --interface {host_interface}`
+            - You may optionally also provide the following arguments:
+                - `--runtime_dir` to specify the directory where log files are stored
+                - `--listen_port` to specify the port on which the primary node listens for replies to messages. This port must match the `--send_port` argument provided to the secondary node.
+                - `--send_port` to specify the port on which the primary node sends messages. This port must match the `--listen_port` argument provided to the secondary node.
+                - `--listen_channel` to specify the channel on which the primary node listens for replies to messages. This channel must match the `--send_channel` argument provided to the secondary node.
+                - `--send_channel` to specify the channel on which the primary node sends messages. This channel must match the `--listen_channel` argument provided to the secondary node.
+                - `--num_messages` to specify the number of messages to send
+                - `--message_size_bytes` to specify the size of the message payload in bytes
+                - `--timeout_duration_ms` to specify the duration in milliseconds to wait for a reply before retransmitting the message
+                - `--initial_delay_s` to specify the delay in seconds before sending the first message
+    2. For unicast:
+        1. Run the latency_tester_secondary_node: `./latency_tester_secondary_node --interface {robot_wifi_interface} --unicast`
+            - You may optionally also provide the following arguments:
+                - `--runtime_dir` to specify the directory where log files are stored
+                - `--listen_port` to specify the port on which the secondary node listens for messages.
+                - `--send_port` to specify the port on which the secondary node sends messages
+                - `--send_ip` to specify the IP address of the primary node to send replies to
+        2. On a different terminal on the host computer, run the latency_tester_primary_node: `./tbots.py run latency_tester_primary_node -- --interface {host_interface} --unicast`
+            - You may optionally also provide the following arguments:
+                - `--runtime_dir` to specify the directory where log files are stored
+                - `--listen_port` to specify the port on which the primary node listens for replies to messages. This port must match the `--send_port` argument provided to the secondary node.
+                - `--send_port` to specify the port on which the primary node sends messages. This port must match the `--listen_port` argument provided to the secondary node.
+                - `--send_ip` to specify the IP address of the secondary node to send messages to (`{robot_ip}`)
+                - `--num_messages` to specify the number of messages to send
+                - `--message_size_bytes` to specify the size of the message payload in bytes
+                - `--timeout_duration_ms` to specify the duration in milliseconds to wait for a reply before retransmitting the message
+                - `--initial_delay_s` to specify the delay in seconds before sending the first message
+3. This tool can also be run with Tracy, a profiling tool, which provides some nice performance visualizations and histograms. To do so:
+    1. Make sure Tracy has been installed. Run `./environment_setup/install_tracy.sh` to install Tracy.
+    2. On a new terminal in the host computer run Tracy: `./tbots.py run tracy`
+    3. When running the latency tester primary node, add the `--tracy` flag to the command before the `--`. For example: `./tbots.py run latency_tester_primary_node --tracy -- --interface {host_interface}`
+    4. Tracy will allow you to select the binary to profile and provide detailed performance information after the tester has run.

environment_setup/install_tracy.sh

@@ -6,7 +6,7 @@ CURR_DIR=$(dirname -- "$(readlink -f -- "$BASH_SOURCE")")
 cd "$CURR_DIR" || exit
 
 # Install dependences
-sudo apt install libglfw3-dev libfreetype-dev libdbus-1-dev
+sudo apt install -y libglfw3-dev libfreetype-dev libdbus-1-dev
 
 # Install capstone-next
 wget -nc https://github.com/capstone-engine/capstone/archive/refs/tags/5.0.1.zip -O /tmp/capstone.zip

environment_setup/setup_software.sh

@@ -211,4 +211,8 @@ sudo ln -s ~/.platformio/penv/bin/platformio /usr/local/bin/platformio
 
 print_status_msg "Done PlatformIO Setup"
 
+print_status_msg "Set up ansible-lint"
+/opt/tbotspython/bin/ansible-galaxy collection install ansible.posix
+print_status_msg "Finished setting up ansible-lint"
+
 print_status_msg "Done Software Setup, please reboot for changes to take place"

environment_setup/ubuntu20_requirements.txt

@@ -1,3 +1,4 @@
+ansible-lint==24.12.2
 pyqtgraph==0.13.7
 pyqt6==6.6.1
 PyQt6-Qt6==6.6.1

environment_setup/ubuntu22_requirements.txt

@@ -1,3 +1,4 @@
+ansible-lint==24.12.2
 pyqtgraph==0.13.7
 pyqt6==6.6.1
 PyQt6-Qt6==6.6.1

environment_setup/ubuntu24_requirements.txt

@@ -1,3 +1,4 @@
+ansible-lint==24.12.2
 pyqtgraph==0.13.7
 thefuzz==0.19.0
 iterfzf==0.5.0.20.0

scripts/lint_and_format.sh

@@ -119,12 +119,25 @@ function run_eof_new_line(){
     fi
 }
 
+function run_ansible_lint(){
+    printf "Running ansible-lint...\n\n"
+
+    /opt/tbotspython/bin/ansible-lint $CURR_DIR/../src/software/embedded/ansible/**/*.yml --fix
+
+    if [[ "$?" != 0 ]]; then
+        printf "\n***Failed to lint and format Ansible files!***\n\n"
+        exit 1
+    fi
+}
+
+
 # Run formatting
 run_code_spell
 run_clang_format
 run_bazel_formatting
 run_ruff
 run_eof_new_line
 run_git_diff_check
+run_ansible_lint
 
 exit 0

src/proto/BUILD

@@ -11,6 +11,7 @@ proto_library(
         "game_state.proto",
         "geneva_slot.proto",
         "geometry.proto",
+        "ip_notification.proto",
         "parameters.proto",
         "play.proto",
         "power_frame_msg.proto",
@@ -50,6 +51,7 @@ py_proto_library(
         "game_state.proto",
         "geneva_slot.proto",
         "geometry.proto",
+        "ip_notification.proto",
         "parameters.proto",
         "play.proto",
         "primitive.proto",

src/proto/ip_notification.proto

@@ -0,0 +1,11 @@
+syntax = "proto3";
+
+import "proto/tbots_timestamp_msg.proto";
+
+package TbotsProto;
+
+message IpNotification
+{
+    int32 robot_id    = 1;
+    string ip_address = 2;
+}

src/proto/parameters.proto

@@ -53,6 +53,9 @@ message AiControlConfig
 
     // Override the existing play with the Play enum provided
     required PlayName override_ai_play = 2 [default = UseAiSelection];
+
+    // Interfaces for various network listeners
+    required NetworkConfig network_config = 3;
 }
 
 message AiParameterConfig
@@ -623,6 +626,18 @@ message PossessionTrackerConfig
     ];
 }
 
+message NetworkConfig
+{
+    // The robot communication interface
+    required string robot_communication_interface = 1 [default = "lo"];
+
+    // The referee interface
+    required string referee_interface = 2 [default = "lo"];
+
+    // The vision interface
+    required string vision_interface = 3 [default = "lo"];
+}
+
 message CreaseDefenderConfig
 {
     // The additional buffer length for each side of the goal

src/proto/primitive.proto

@@ -2,9 +2,10 @@ syntax = "proto3";
 
 package TbotsProto;
 
+import "google/protobuf/descriptor.proto";
 import "proto/geometry.proto";
 import "proto/geneva_slot.proto";
-import "google/protobuf/descriptor.proto";
+import "proto/tbots_timestamp_msg.proto";
 
 extend google.protobuf.EnumValueOptions
 {
@@ -100,13 +101,20 @@ message PowerControl
 
 message Primitive
 {
-    reserved 5;
     oneof primitive
     {
         MovePrimitive move                    = 1;
         StopPrimitive stop                    = 2;
         DirectControlPrimitive direct_control = 3;
     }
+
+    // Sequence number for the primitive
+    uint64 sequence_number = 4;
+
+    reserved 5;
+
+    // Epoch timestamp when primitives were assigned
+    Timestamp time_sent = 6;
 }
 
 message MovePrimitive

src/shared/constants.h

@@ -43,10 +43,12 @@ static const short unsigned int REDIS_DEFAULT_PORT      = 6379;
 static const short unsigned int PRIMITIVE_PORT = 42070;
 
 // the port the AI receives msgs from the robot
-static const short unsigned int ROBOT_STATUS_PORT      = 42071;
-static const short unsigned int ROBOT_LOGS_PORT        = 42072;
-static const short unsigned int ROBOT_CRASH_PORT       = 42074;
-static const short unsigned int NETWORK_COMM_TEST_PORT = 42075;
+static constexpr short unsigned int ROBOT_STATUS_PORT                         = 42071;
+static constexpr short unsigned int ROBOT_LOGS_PORT                           = 42072;
+static constexpr short unsigned int ROBOT_CRASH_PORT                          = 42074;
+static constexpr short unsigned int NETWORK_COMM_TEST_PORT                    = 42075;
+static constexpr short unsigned int ROBOT_TO_FULL_SYSTEM_IP_NOTIFICATION_PORT = 42073;
+static constexpr short unsigned int FULL_SYSTEM_TO_ROBOT_IP_NOTIFICATION_PORT = 42076;
 
 // maximum transfer unit of the network interface
 // this is an int to avoid Wconversion with lwip

src/shared/test_util/tbots_gtest_main.cpp

@@ -11,7 +11,7 @@ bool TbotsGtestMain::help                = false;
 bool TbotsGtestMain::enable_visualizer   = false;
 bool TbotsGtestMain::run_sim_in_realtime = false;
 bool TbotsGtestMain::stop_ai_on_start    = false;
-std::string TbotsGtestMain::runtime_dir  = "/tmp/tbots/gtest_logs";
+std::string TbotsGtestMain::runtime_dir  = "/tmp/tbots/yellow_test";
 double TbotsGtestMain::test_speed        = 1.0;
 
 

src/software/BUILD

@@ -46,6 +46,8 @@ cc_binary(
         "//proto:tbots_cc_proto",
         "//shared:constants",
         "//software/networking/udp:threaded_proto_udp_listener",
+        "//software/networking/udp:threaded_proto_udp_sender",
+        "//software/world:robot_state",
         "@boost//:program_options",
     ],
 )

src/software/embedded/ansible/playbooks/deploy_powerboard.yml

@@ -1,35 +1,35 @@
 ---
-
 - name: Deploy to the powerboard
   hosts: THUNDERBOTS_HOSTS
 
   tasks:
     - name: Sync powerboard files
       become: true
-      become_method: sudo
+      become_method: ansible.builtin.sudo
       ansible.posix.synchronize:
         src: ../../../../../power/powerloop.tar.gz
         dest: ~/
-        recursive: yes
-        copy_links: yes
+        recursive: true
+        copy_links: true
 
     - name: Untar powerboard files
-      shell: 'mkdir -p powerloop && tar -xvf ~/powerloop.tar.gz -C powerloop'
+      ansible.builtin.shell: "mkdir -p powerloop && tar -xvf ~/powerloop.tar.gz -C powerloop"
       register: result
+      changed_when: true
       args:
         chdir: ~/
 
     - name: Put the powerboard in bootloader mode
       ansible.builtin.pause:
         prompt: "Press enter to continue"
-        echo: no
+        echo: false
 
     - name: Flashing... (this will take a while on the first run)
-      shell: '/opt/tbotspython/bin/platformio run --disable-auto-clean -t nobuild -t upload -d ~/powerloop/power'
+      ansible.builtin.command: "/opt/tbotspython/bin/platformio run --disable-auto-clean -t nobuild -t upload -d ~/powerloop/power"
       register: result
+      changed_when: true
 
     - name: Reset powerboard to finish flashing
       ansible.builtin.pause:
         prompt: "Press enter to continue"
-        echo: no
-
+        echo: false