Nailed native test numbers so PRs don't forget to increment by NomDeTom · Pull Request #11117 · meshtastic/firmware

NomDeTom · 2026-07-21T11:11:08Z

There's a manual count of the number of native tests - this is needed because previously tests were skipped silently, and so local testing wouldn't mirror the CI testing.

Now more tests are being added, but it's not being incremented - presumably people are running their own test suites. This PR prevents people adding tests without incrementing the number.

🤝 Attestations

I have tested that my proposed changes behave as described.
I have tested that my proposed changes do not cause any obvious regressions on the following devices:
- Heltec (Lora32) V3
- LilyGo T-Deck
- LilyGo T-Beam
- RAK WisBlock 4631
- Seeed Studio T-1000E tracker card
- Other (please specify below)

Summary by CodeRabbit

Tests
- Added an automated workflow check to verify the stored native test suite count matches the actual number of native test_* suites.
- The native test pipeline now fails early with clear error annotations when the count file is missing, malformed, or out of sync.
- Simulator and platform-based test jobs are gated to run only when suite counts are consistent, improving reliability of test results.

platformio test discovers and runs whatever test_* directories exist, so it never notices when test/native-suite-count drifts from the actual directory count. That reconciliation lived only in bin/run-tests.sh, which CI does not invoke - so a stale or unbumped count sailed through PRs (develop itself shipped 38 dirs against a 37 file). Add a standalone suite-count-check job to test_native.yml that mirrors run-tests.sh's exact counting logic and fails on a mismatch. It runs on every PR via main_matrix's test-native job, with no build step so it fails fast. Bump native-suite-count to 38 to match the current suites. clod helped too

coderabbitai · 2026-07-21T11:11:27Z

📝 Walkthrough

Walkthrough

Adds a native test workflow job that validates test/native-suite-count against test/test_* directories, gates simulator and PlatformIO tests, and pins affected checkout actions to a specific v7 commit.

Changes

Native suite count validation

Layer / File(s)	Summary
Validate and gate native test suites `.github/workflows/test_native.yml`	The workflow validates the count file, counts native test directories, reports mismatches, gates simulator and PlatformIO tests, and pins checkout actions for the affected jobs.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

meshtastic/firmware#11059: Updates the native suite count value checked by this workflow.
meshtastic/firmware#11130: Modifies native test suite discovery in the same workflow.

Suggested labels: github_actions

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly matches the main change: enforcing the native test count so it stays incremented.
Description check	✅ Passed	The description summarizes the change and includes the required attestations section, so it is mostly complete.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/test_native.yml:
- Line 24: Update the actions/checkout step to set persist-credentials to false,
ensuring the workflow does not retain the GITHUB_TOKEN in local Git
configuration while preserving the existing checkout behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 6c589e66-b0c8-4fd2-b7a4-a0ada549d1c0

📥 Commits

Reviewing files that changed from the base of the PR and between 1872e5b and ed96b0a.

📒 Files selected for processing (1)

.github/workflows/test_native.yml

Copilot

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Adds a CI guard to ensure the manually maintained test/native-suite-count stays in sync with the actual number of test/test_* suite directories, preventing PRs from adding/removing suites without updating the count.

Changes:

Introduces a new GitHub Actions job to reconcile test/native-suite-count with discovered test/test_* directories.
Fails CI with a targeted ::error message when the count drifts, including actionable hints.

coderabbitai

🧹 Nitpick comments (1)

.github/workflows/test_native.yml (1)
60-60: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Skip report generation when the suite-count gate fails.

When suite-count-check fails, both dependent test jobs are skipped. Because generate-reports still runs with if: always(), it can attempt to download artifacts that were never produced and fail with a secondary, misleading error. Add suite-count-check to generate-reports.needs and require that gate to succeed while preserving always() for actual test failures.
Suggested fix
     needs:
+      - suite-count-check
       - simulator-tests
       - platformio-tests
-    if: always()
+    if: ${{ always() && needs.suite-count-check.result == 'success' }}
Also applies to: 135-135
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/test_native.yml at line 60, Update the generate-reports
job’s needs configuration to include suite-count-check, and adjust its condition
so the suite-count gate must succeed while always() still allows report
generation after test-job failures. Preserve the existing dependent test-job
requirements and artifact-reporting behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/workflows/test_native.yml:
- Line 60: Update the generate-reports job’s needs configuration to include
suite-count-check, and adjust its condition so the suite-count gate must succeed
while always() still allows report generation after test-job failures. Preserve
the existing dependent test-job requirements and artifact-reporting behavior.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 0a04584f-92ac-48ca-b972-bd23530b15c1

📥 Commits

Reviewing files that changed from the base of the PR and between d811835 and d5bb2c2.

📒 Files selected for processing (1)

.github/workflows/test_native.yml

github-actions · 2026-07-21T21:51:33Z

⚡ Try this PR in the Web Flasher

Warning

This is an automated, unreviewed CI test build. Back up your device configuration
before flashing, and only flash devices you are able to recover.

Supported boards built by this PR (30)

Device	Board	Platform
Crowpanel Adv 3.5 TFT	`elecrow-adv-35-tft`	esp32-s3
Heltec HT62	`heltec-ht62-esp32c3-sx1262`	esp32-c3
Heltec Mesh Node 096	`heltec-mesh-node-t096`	nrf52840
Heltec Mesh Node T1	`heltec-mesh-node-t1`	nrf52840
Heltec Mesh Node T114	`heltec-mesh-node-t114`	nrf52840
Heltec V3	`heltec-v3`	esp32-s3
Heltec V4	`heltec-v4`	esp32-s3
Meshnology W10	`meshnology_w10`	esp32-s3
Raspberry Pi Pico	`pico`	rp2040
Raspberry Pi Pico W	`picow`	rp2040
RAK WisMesh Pocket V3	`rak_wismesh_pocket`	nrf52840
RAK WisMesh Pod	`rak_wismesh_pod`	nrf52840
RAK WisMesh Repeater Mini V2	`rak_wismesh_repeater_mini`	nrf52840
RAK WisMesh Tag	`rak_wismeshtag`	nrf52840
RAK WisBlock 11200	`rak11200`	esp32
RAK WisBlock 11310	`rak11310`	rp2040
RAK3312	`rak3312`	esp32-s3
RAK WisBlock 4631	`rak4631`	nrf52840
Seeed SenseCAP Mesh-Tracker-X1	`seeed_mesh_tracker_X1`	nrf52840
Seeed Wio Tracker L1	`seeed_wio_tracker_L1`	nrf52840
Seeed Xiao NRF52840 Kit	`seeed_xiao_nrf52840_kit`	nrf52840
Seeed Xiao ESP32-S3	`seeed-xiao-s3`	esp32-s3
Station G2	`station-g2`	esp32-s3
Station G3	`station-g3`	esp32-s3
LILYGO T-Deck	`t-deck-tft`	esp32-s3
LILYGO T-Echo	`t-echo`	nrf52840
LILYGO T-Echo Plus	`t-echo-plus`	nrf52840
LILYGO T-Impulse Plus	`t-impulse-plus`	nrf52840
LilyGo T3-C6	`tlora-c6`	esp32-c6
Seeed SenseCAP T1000-E	`tracker-t1000-e`	nrf52840

Build artifacts expire on 2026-08-21. Updated for 99efee5.

NomDeTom and others added 2 commits July 21, 2026 09:39

Merge branch 'meshtastic:develop' into nailed-test-number

ed96b0a

coderabbitai Bot reviewed Jul 21, 2026

View reviewed changes

Comment thread .github/workflows/test_native.yml Outdated

NomDeTom requested a review from Copilot July 21, 2026 11:14

NomDeTom added enhancement New feature or request tech debt Code or lib references that are not up to date or propper standards labels Jul 21, 2026

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread .github/workflows/test_native.yml

Comment thread .github/workflows/test_native.yml

Comment thread .github/workflows/test_native.yml

NomDeTom and others added 3 commits July 21, 2026 21:46

nitpicks and guards

d811835

Merge branch 'develop' into nailed-test-number

9f6fb6b

fix a nitpicky semgrep

d5bb2c2

coderabbitai Bot reviewed Jul 21, 2026

View reviewed changes

Merge branch 'meshtastic:develop' into nailed-test-number

99efee5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Nailed native test numbers so PRs don't forget to increment#11117

Nailed native test numbers so PRs don't forget to increment#11117
NomDeTom wants to merge 6 commits into
meshtastic:developfrom
NomDeTom:nailed-test-number

NomDeTom commented Jul 21, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 21, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

github-actions Bot commented Jul 21, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

NomDeTom commented Jul 21, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤝 Attestations

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚡ Try this PR in the Web Flasher

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

NomDeTom commented Jul 21, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 21, 2026 •

edited

Loading

github-actions Bot commented Jul 21, 2026 •

edited

Loading