|
| 1 | +# Testing strategy |
| 2 | + |
| 3 | +## Status |
| 4 | + |
| 5 | +**DRAFT** |
| 6 | + |
| 7 | +## Scope |
| 8 | + |
| 9 | +This document specifies how candumpr (and other tools in this workspace) are tested, given that they |
| 10 | +depend on Linux socketcan interfaces that require either real hardware or elevated permissions to |
| 11 | +create. |
| 12 | + |
| 13 | +## Problem |
| 14 | + |
| 15 | +candumpr interacts directly with CAN sockets. Testing requires CAN interfaces, but: |
| 16 | + |
| 17 | +* Real CAN hardware is not available in CI. |
| 18 | +* Virtual CAN (vcan) interfaces require `CAP_NET_ADMIN` to create. |
| 19 | +* vcan interfaces are system-global resources, so parallel tests using shared interfaces cause |
| 20 | + interference. |
| 21 | +* Tests must run in CI (GitHub Actions) and locally without requiring root. |
| 22 | + |
| 23 | +## Solution: user + network namespaces |
| 24 | + |
| 25 | +Each test process enters its own isolated Linux network namespace using |
| 26 | +`unshare(CLONE_NEWUSER | CLONE_NEWNET)`. Inside the namespace, the process has `CAP_NET_ADMIN` |
| 27 | +without real root privileges, vcan interfaces are private and isolated, and everything is cleaned up |
| 28 | +when the process exits. See the [vcan-fixture](../../vcan-fixture/) crate for the implementation. |
| 29 | + |
| 30 | +Constraint: `unshare(CLONE_NEWUSER)` requires a single-threaded process. The Rust test harness is |
| 31 | +multi-threaded, so namespace entry happens in a `ctor` constructor before `main()`. |
| 32 | + |
| 33 | +## Test tiers |
| 34 | + |
| 35 | +### Unit tests |
| 36 | + |
| 37 | +No sockets, no namespaces. Config parsing, filter compilation, output formatting, filename template |
| 38 | +expansion, duration/size parsing. |
| 39 | + |
| 40 | +### Integration tests |
| 41 | + |
| 42 | +Run inside user + network namespaces with vcan interfaces. Socket binding, filter application, |
| 43 | +multi-interface capture, file rotation, ZSTD streaming, address claim, device resilience. |
| 44 | + |
| 45 | +### End-to-end tests |
| 46 | + |
| 47 | +Run the actual binary inside a network namespace. Launch candumpr, send frames with cangenr, verify |
| 48 | +output files, signal handling, config file loading. |
| 49 | + |
| 50 | +## CI |
| 51 | + |
| 52 | +Tests that require vcan use `#[cfg_attr(feature = "ci", ignore = "requires vcan")]`. In CI, |
| 53 | +`--all-features` enables the `ci` feature, making them `#[ignore]`. They are then run as a separate |
| 54 | +step gated on whether vcan setup succeeded: |
| 55 | + |
| 56 | +A separate canary job (`vcan-available`) with `continue-on-error: true` shows yellow when the vcan |
| 57 | +module is unavailable on the runner, rather than silently skipping the tests. |
| 58 | + |
| 59 | +See [lint.yml](/.github/workflows/lint.yml) for the implementation. |
| 60 | + |
| 61 | +## Benchmarking |
| 62 | + |
| 63 | +Benchmarks compare candumpr against candump on 4 vcan interfaces with J1939 traffic. |
| 64 | + |
| 65 | +### Metrics |
| 66 | + |
| 67 | +* **Frame loss** (primary): frames sent vs. frames in output |
| 68 | +* **Throughput ceiling**: send rate at which frames start dropping |
| 69 | +* **CPU usage**: total CPU time (user + system) |
| 70 | +* **Memory usage**: peak RSS |
| 71 | + |
| 72 | +### Simulating the target environment |
| 73 | + |
| 74 | +The target is a ~4 core ~1 GHz ARM CPU. Use `taskset` to pin benchmarks to 4 cores: |
| 75 | + |
| 76 | +```sh |
| 77 | +taskset -c 0-3 cargo bench |
| 78 | +``` |
| 79 | + |
| 80 | +Core count is the important variable for comparing architecture options (dedicated thread pairs vs. |
| 81 | +shared threads). Clock speed matters less for relative comparison. Final validation must happen on |
| 82 | +real target hardware. |
| 83 | + |
| 84 | +### Acceptance criteria |
| 85 | + |
| 86 | +candumpr must not drop frames at the realistic J1939 rate (2000 frames/s per interface, 8000 |
| 87 | +frames/s aggregate). At higher rates, candumpr should drop fewer frames than candump. |
0 commit comments