Update to Python 3.10, Bazel 5.4 and replace Black/Autoflake with Ruff (UBC-Thunderbots#3292)

* Update to python 3.10.0 and bazel 5.4.0

* No need to import `typing` for collections as of python 3.9

* Update black formatter to 24.4.2

* Run fix_formatting.sh

* Remove PyOpenGL-accelerate dependency

* Install latest pip in setup_software.sh

* Install latest pip in setup_software.sh

* Testing

* Testing

* Remove testing

* Bump pyqtgraph to 0.13.3

* Update setup_software.sh

* Try fixing precommit

* Try fixing precommit

* Try fixing precommit

* Replace black/autoflake with ruff

* Remove black binary

* Remove git attribute

* Remove unused check_formatting_ci.sh

* Run fix_formatting.sh

* Add simulated/field test fixtures to conftest.py

* Address UBC-Thunderbots#3251

* Python code cleanup and fix Thunderscope

* Enable pydocstyle rules in ruff

* Add compile pip requirements to pre-commit

* Formatting and fix pre-commit

* Fix pre-commit

* Fix pre-commit

* Fix pre-commit and fsm_diagram_generator.py

* Update docstrings

* Formatting

* Fix type errors

* Nuke jetson_nano/display :(

* Switch to pyqtdarktheme

* Bump pytqtgraph to 0.13.7

* If build fails, download autoref from Google Drive mirror

* Fix setup_software.sh

* If build fails, download autoref from releases in github fork

* Rename formatting_scripts to scripts

* Fix docstring

* FIx docstring

* [pre-commit.ci lite] apply automatic fixes

* Fix pyqtdarktheme dep missing

* Nits

* Use python dataclasses, remove unused stuff, nits

* [pre-commit.ci lite] apply automatic fixes

* Use dataclass

* Nits

---------

Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com>

Author: williamckha

.gitattributes

@@ -7,10 +7,7 @@
 * text=auto
 
 # Our clang-format executable is a binary
-formatting_scripts/clang-format-*    binary
-
-# Our black executable is a binary
-formatting_scripts/black*    binary
+scripts/clang-format-*    binary
 
 # Treat images as binaries
 *.svg           binary

.github/workflows/pre-commit.yml

@@ -24,6 +24,9 @@ jobs:
         run: |
           "${GITHUB_WORKSPACE}"/environment_setup/setup_software.sh
       
+      - name: Install pip
+        run: curl -sS https://bootstrap.pypa.io/get-pip.py | python
+
       - uses: pre-commit/[email protected]
 
       - uses: pre-commit-ci/[email protected]

.gitignore

@@ -148,7 +148,7 @@ src/config
 *.ipynb
 
 # codespell dictionary
-formatting_scripts/dictionary/
+scripts/dictionary/
 
 # clangd for nvim lsp
 src/.clangd

.pre-commit-config.yaml

@@ -1,12 +1,17 @@
 repos:
 -   repo: local
     hooks:
-    -   id: fix_formatting
+    -   id: compile_pip_requirements
+        name: compile pip requirements
+        language: system
+        entry: scripts/compile_pip_requirements.sh
+
+    -   id: lint_and_format
         name: fix common trivial code errors
         language: system
-        entry: formatting_scripts/fix_formatting.sh 
+        entry: scripts/lint_and_format.sh 
 
     -   id: generate_fsm_diagrams
         name: generate new FSM diagrams
         language: python
-        entry: ./src/software/ai/fsm_diagram_generator.py
+        entry: scripts/fsm_diagram_generator.py

docs/fsm-diagrams.md

@@ -17,12 +17,9 @@ Stop --> SetPlay : [gameStateSetupRestart]\n<i>setupSetPlay</i>
 Playing --> Halt : [gameStateHalted]\n<i>setupHaltPlay</i>
 Playing --> Stop : [gameStateStopped]\n<i>setupStopPlay</i>
 Playing --> SetPlay : [gameStateSetupRestart]\n<i>setupSetPlay</i>
-SetPlay --> SetPlay : [gameStateHalted]\n<i>(resetSetPlay</i>
-setupHaltPlay_A) --> Halt
-SetPlay --> SetPlay : [gameStateStopped]\n<i>(resetSetPlay</i>
-setupStopPlay_A) --> Stop
-SetPlay --> SetPlay : [gameStatePlaying]\n<i>(resetSetPlay</i>
-setupOffensePlay_A) --> Playing
+SetPlay --> Halt : [gameStateHalted]\n<i>resetSetPlay, setupHaltPlay</i>
+SetPlay --> Stop : [gameStateStopped]\n<i>resetSetPlay, setupStopPlay</i>
+SetPlay --> Playing : [gameStatePlaying]\n<i>resetSetPlay, setupOffensePlay</i>
 SetPlay --> SetPlay : [gameStateSetupRestart]\n<i>setupSetPlay</i>
 Terminate:::terminate --> Terminate:::terminate
 

docs/getting-started.md

@@ -215,8 +215,8 @@ Now that you're setup, if you can run it on the command line, you can run it in
         - `[--run_blue | --run_yellow]` indicate which FullSystem to run
         - `[--run_diagnostics]` indicates if diagnostics should be loaded as well
       - If FullSystem is running, the robots receive input from the AI
-      - If Diagnostics is enabled, the robots can also receive input from Manual controls or XBox controls
-      - This mode allows us to test and debug the robots by setting each robot's input to be either AI, Manual Control or XBox Control
+      - If Diagnostics is enabled, the robots can also receive input from Manual controls or Xbox controls
+      - This mode allows us to test and debug the robots by setting each robot's input to be either AI, Manual Control or Xbox Control
       - Control mode for each robot can be set with each one's drop down menu in the Robot View widget
 
     - If we want to run it with real robots:
@@ -262,7 +262,7 @@ Now that you're setup, if you can run it on the command line, you can run it in
     - Manual Control
       - When a robot is in Manual control mode, the commands it receives depend on the radio buttons to the top-right
         - Diagnostics Control allows us to use the on-screen sliders and buttons to control the robot
-        - XBox control allows us to use a connected XBox controller to control the robots
+        - Xbox control allows us to use a connected Xbox controller to control the robots
 4. Run our SimulatedPlayTests in Thunderscope
     - This will launch the visualizer and simulate AI Plays, allowing us to visually see the robots acting according to their roles.
     1. For legacy C++ tests (#2581) with the visualizer:
@@ -443,7 +443,7 @@ If you do rebase or merge and get conflicts, you'll need to resolve them manuall
 
 We use [clang-format](https://electronjs.org/docs/development/clang-format) to automatically format our code. Using an automatic tool helps keep things consistent across the codebase without developers having to change their personal style as they write. See the [code style guide](code-style-guide.md) for more information on exactly what it does.
 
-To format the code, from the `Software` directory run `./formatting_scripts/fix_formatting.sh`.
+To format the code, from the `Software` directory run `./scripts/lint_and_format.sh`.
 
 We recommend running the formatting script and then committing all your changes, so that your commits can more easily pass CI.
 

environment_setup/setup_software.sh

@@ -62,14 +62,14 @@ host_software_packages=(
     protobuf-compiler # This is required for the "NanoPb" library, which does not
                       # properly manage this as a bazel dependency, so we have
                       # to manually install it ourselves
-    python3.8       # Python 3
-    python3.8-dev # Python 3 headers
-    python3.8-venv # Virtual Environment
-    python3-pip   # Required for bazel to install python dependencies for build targets
-    python3-protobuf # This is required for the "NanoPb" library, which does not
-                    # properly manage this as a bazel dependency, so we have
-                    # to manually install it ourselves
-    python3-yaml 	# Load dynamic parameter configuration files
+    python3.10        # Python 3
+    python3.10-dev    # Python 3 headers
+    python3.10-venv   # Virtual Environment
+    python3-pip       # Required for bazel to install python dependencies for build targets
+    python3-protobuf  # This is required for the "NanoPb" library, which does not
+                      # properly manage this as a bazel dependency, so we have
+                      # to manually install it ourselves
+    python3-yaml 	  # Load dynamic parameter configuration files
     valgrind # Checks for memory leaks
     libsqlite3-dev # needed to build Python 3 with sqlite support
     libffi-dev # needed to use _ctypes in Python3
@@ -114,7 +114,12 @@ print_status_msg "Setting Up Virtual Python Environment"
 # delete tbotspython first
 sudo rm -rf /opt/tbotspython
 
-if ! sudo /usr/bin/python3.8 -m venv /opt/tbotspython ; then
+if ! curl -sS https://bootstrap.pypa.io/get-pip.py | sudo /usr/bin/python3.10 ; then
+    print_status_msg "Error: Installing pip failed"
+    exit 1
+fi
+
+if ! sudo /usr/bin/python3.10 -m venv /opt/tbotspython ; then
     print_status_msg "Error: Setting up virtual environment failed"
     exit 1
 fi
@@ -149,8 +154,15 @@ print_status_msg "Compiling TIGERS AutoRef"
 sudo wget -N https://github.com/TIGERs-Mannheim/AutoReferee/archive/refs/heads/autoref-ci.zip -O /tmp/autoref-ci.zip
 unzip -q -o -d /tmp/ /tmp/autoref-ci.zip
 touch /tmp/AutoReferee-autoref-ci/.git # a hacky way to make gradle happy when it tries to find a dependency
-/tmp/AutoReferee-autoref-ci/./gradlew installDist -p /tmp/AutoReferee-autoref-ci/ -Dorg.gradle.java.home=/usr/lib/jvm/jdk-17/
-cp -r /tmp/AutoReferee-autoref-ci/build/install/autoReferee/ /opt/tbotspython/autoReferee
+
+if ! /tmp/AutoReferee-autoref-ci/./gradlew installDist -p /tmp/AutoReferee-autoref-ci/ -Dorg.gradle.java.home=/usr/lib/jvm/jdk-17/; then
+    print_status_msg "Building TIGERS AutoRef failed. Downloading mirror"
+    
+    wget https://github.com/UBC-Thunderbots/AutoReferee/releases/download/autoref-ci/autoReferee.tar.gz -O /tmp/autoReferee.tar.gz
+    tar -xzf /tmp/autoReferee.tar.gz -C /opt/tbotspython/
+else
+    cp -r /tmp/AutoReferee-autoref-ci/build/install/autoReferee/ /opt/tbotspython/autoReferee
+fi
 
 sudo chmod +x "$CURR_DIR/../src/software/autoref/run_autoref.sh"
 sudo cp "$CURR_DIR/../src/software/autoref/DIV_B.txt" "/opt/tbotspython/autoReferee/config/geometry/DIV_B.txt"
@@ -160,8 +172,11 @@ print_status_msg "Finished setting up AutoRef"
 # Install Bazel
 print_status_msg "Installing Bazel"
 
+# Uninstall Bazel first
+sudo rm -rf $HOME/.bazel
+
 # Adapted from https://docs.bazel.build/versions/main/install-ubuntu.html#install-with-installer-ubuntu
-sudo wget -nc https://github.com/bazelbuild/bazel/releases/download/5.0.0/bazel-5.0.0-installer-linux-x86_64.sh -O /tmp/bazel-installer.sh
+sudo wget -nc https://github.com/bazelbuild/bazel/releases/download/5.4.0/bazel-5.4.0-installer-linux-x86_64.sh -O /tmp/bazel-installer.sh
 sudo chmod +x /tmp/bazel-installer.sh
 sudo /tmp/bazel-installer.sh --bin=/usr/bin --base=$HOME/.bazel
 echo "source ${HOME}/.bazel/bin/bazel-complete.bash" >> ~/.bashrc
@@ -184,8 +199,8 @@ sudo service udev restart
 # allow user access to serial ports
 sudo usermod -a -G dialout $USER
 
-# installs PlatformIO to global environment
-if ! sudo /usr/bin/python3.8 -m pip install --prefix /usr/local platformio==6.1.13; then
+# install PlatformIO to global environment
+if ! sudo /usr/bin/python3.10 -m pip install platformio==6.1.13; then
     print_status_msg "Error: Installing PlatformIO failed"
     exit 1
 fi

environment_setup/ubuntu20_requirements.txt

@@ -1,15 +1,12 @@
 protobuf==3.20.2
-pyqtgraph==0.12.4
-autoflake==1.4
+pyqtgraph==0.13.7
+pyqtdarktheme==2.1.0
 pyqt6==6.5.0
 PyQt6-Qt6==6.5.0
 thefuzz==0.19.0
 iterfzf==0.5.0.20.0
-pyqtdarktheme==1.1.0
 python-Levenshtein==0.12.2
 psutil==5.9.0
-qt-material==2.12
 PyOpenGL==3.1.6
-PyOpenGL-accelerate==3.1.5
-psutil==5.9.0
 numpy==1.24.4
+ruff==0.5.5

environment_setup/ubuntu22_requirements.txt

@@ -1,14 +1,12 @@
 protobuf==3.20.2
-pyqtgraph==0.12.4
-autoflake==1.4
+pyqtgraph==0.13.7
+pyqtdarktheme==2.1.0
 pyqt6==6.5.0
 PyQt6-Qt6==6.5.0
 thefuzz==0.19.0
 iterfzf==0.5.0.20.0
-pyqtdarktheme==1.1.0
 python-Levenshtein==0.12.2
 psutil==5.9.0
-qt-material==2.12
 PyOpenGL==3.1.6
-psutil==5.9.0
 numpy==1.24.4
+ruff==0.5.5

formatting_scripts/black_19-10b0

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
+# This script runs all the `compile_pip_requirements` update targets in the repo
+# to generate requirements lock files from requirement.in files. 
+#
+# Whenever we update requirements.in with new dependencies/versions, we need to 
+# regenerate its associated requirements lock file and check it into git.
+#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~#
+
+# The directory this script is in
+CURR_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+# The root bazel directory
+BAZEL_ROOT_DIR="$CURR_DIR/../src"
+
+cd $BAZEL_ROOT_DIR
+bazel run //extlibs:nanopb_requirements.update
+bazel run //software/thunderscope:requirements.update
+bazel run //software/jetson_nano/ansible:requirements.update
+bazel run //software/simulated_tests:requirements.update

formatting_scripts/check_formatting_ci.sh

@@ -1,4 +1,4 @@
-#!/opt/tbotspython/bin/python3.8
+#!/opt/tbotspython/bin/python3
 
 import re
 import os
@@ -8,17 +8,14 @@
 
 
 def generate_diagram(fsm):
-    """
-    Generates a Mermaid text definition for a state diagram 
-    representing the FSM.
+    """Generates a Mermaid text definition for a state diagram representing the FSM.
 
     Details on Mermaid UML syntax:
     https://mermaid.js.org/syntax/stateDiagram.html
 
     :param fsm: the FSM code containing its transition table
-    :returns: the mermaid.js text definition for FSM diagram
+    :return: the mermaid.js text definition for FSM diagram
     """
-
     # Regex match to extract transition table from the FSM
     transition_table_match = re.search(r"make_transition_table\(([\s\S]*?)\);", fsm)
     if not transition_table_match:
@@ -33,34 +30,45 @@ def generate_diagram(fsm):
 
     diagram = ""
 
-    transitions = transition_table.split(",")
-    for transition in transitions:
+    # Get the transitions in the transition table
+    # Split by commas NOT in between parentheses (commas in parentheses separate actions)
+    transitions = re.split(r",(?![^()]*\))", transition_table)
 
+    for transition in transitions:
         # Transitions have the following format:
         # src_state + event [guard] / action = dest_state
 
         # Extract dest_state from transition, which is after '=' sign
-        transition, *rest = transition.split("=")
-        dest_state = rest[0] if rest else ""
+        transition, _, dest_state = transition.partition("=")
 
         # Extract action from transition, which is after '/' sign
-        transition, *rest = transition.split("/")
-        action = rest[0] if rest else ""
+        transition, _, action = transition.partition("/")
 
         # Extract guard contained between square brackets
         guard = next(iter(re.findall(r"\[(.*?)\]", transition)), "")
 
         # Extract src_state from transition, which is before '+' sign
-        src_state = transition.split("+")[0]
+        src_state, _, _ = transition.partition("+")
 
-        def remove_suffix(str, suffix):
-            return str[: -len(suffix)] if str.endswith(suffix) else str
+        # There may be multiple guards, so get a list of the guards and the
+        # operators separating them
+        guard_list = re.split(r"(&&|\|\|)", guard)
 
-        # Remove suffixes from states, guard, action
-        src_state = remove_suffix(src_state, "_S")
-        dest_state = remove_suffix(dest_state, "_S")
-        guard = " && ".join([remove_suffix(g, "_G") for g in guard.split("&&")])
-        action = remove_suffix(action, "_A")
+        # There may be multiple actions, so get a list of all of them
+        if action.startswith("(") and action.endswith(")"):
+            action_list = action.strip("()").split(",")
+        else:
+            action_list = [action]
+
+        # Remove suffixes from states, guards, actions
+        src_state = src_state.removesuffix("_S")
+        dest_state = dest_state.removesuffix("_S")
+        guard_list = [g.removesuffix("_G") for g in guard_list]
+        action_list = [a.removesuffix("_A") for a in action_list]
+
+        # Convert guard and action lists into strings
+        guard = " ".join(guard_list)
+        action = ", ".join(action_list)
 
         # Terminate state is marked with X in transition table.
         # Give terminate state custom styling with the
@@ -97,17 +105,15 @@ def remove_suffix(str, suffix):
 
 
 if __name__ == "__main__":
-
-    root_dir = Path(os.path.abspath(__file__)).parents[3]
+    root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+    ai_dir = os.path.join(root_dir, "src/software/ai")
 
     fsm_diagrams = {}
     fsm_file_paths = {}
 
-    ai_dir = os.path.join(root_dir, "src/software/ai")
     for root, dirs, files in os.walk(ai_dir):
         dirs.sort()
         for file in sorted(files):
-
             if not file.endswith("fsm.h"):
                 continue
 
@@ -119,7 +125,6 @@ def remove_suffix(str, suffix):
             diagram = generate_diagram(fsm)
 
             if diagram:
-
                 # Regex match to extract FSM struct name
                 fsm_name = re.findall(r"struct\s+(\w+)\s(?:\s:\s.)?{?", fsm)[0]
 
@@ -131,11 +136,9 @@ def remove_suffix(str, suffix):
 
     output_file_path = os.path.join(root_dir, OUTPUT_FILE_PATH)
     with open(output_file_path, "w") as output_file:
-
         print("# Play and Tactic FSM Diagrams\n", file=output_file)
 
         for fsm_name, diagram in fsm_diagrams.items():
-
             file_path = fsm_file_paths[fsm_name]
 
             # Format diagram in Markdown syntax and write to output file