fix(ci): AUT-2282 exact coverage gate + test PR heads against current main - #3338
Open
svcnemo-autobot wants to merge 2 commits into
Open
svcnemo-autobot wants to merge 2 commits into
svcnemo-autobot wants to merge 2 commits into
Conversation
… against current main PR NVIDIA-NeMo#3330 merged with exact coverage 94.97% because the 95% fail_under gate rounded to 95% at zero-decimal precision, and CI tested the stale PR head (d9b2122) instead of the six-commits-advanced main, so the merged main state (94.46%) was never checked pre-merge. - set [tool.coverage.report] precision = 2 so fail_under compares the exact percentage (94.97 < 95 now exits nonzero) - add scripts/ci/merge_current_main.sh: on mirrored pull-request/* checkouts, merge current origin/main into the tested tree so every PR job validates the synthetic merge with current main; a conflict fails validation - wire the merge step into every unit-tests.yml checkout and drop the github.event_name guard on force-run-all so full suites run when the base-ref is empty - cover all of the above in tests/unit_tests/test_ci_environment.py Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
svcnvidia-nemo-ci
added
area:ci
|
/ok to test d81b5da |
… guards, sandbox image rewrites Raises exact coverage above the now-exact 95% gate (94.98% on the prior head): covers nemo_gym.decorators.experimental, judge.py's JudgeError passthrough and failsafe missing-body guard, _verifier_runner's absolute-path and update_expected guards, and sandbox/utils cpu_cap_env/rewrite_image. Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
|
/ok to test 6c5e69a |
yaoyu-33
added
bug
3 tasks
kajalj22
added a commit
that referenced
this pull request
Sep 14, 2026
## Background The full test suite failed on `main` with exact coverage **94.46%**. Root cause (per triage on PR #3330's merge): the coverage gate compares at zero-decimal precision, so an actual **94.97%** rounded to 95% and incorrectly passed the 95% gate before merge. This PR addresses only that rounding gap. It intentionally excludes the stale-PR-head / merge-current-main fix tracked separately in #3338, and does not add tests to close the pre-existing coverage gap (a separate concern). ## What changed - `pyproject.toml`: set `[tool.coverage.report] precision = 2` so `fail_under` compares the exact percentage — 94.97 < 95 now exits nonzero. - `tests/unit_tests/test_ci_environment.py`: assert `precision == 2` and `fail_under == 95.0` in the coverage config. ## Note With this change alone, the repo's aggregate coverage is still below 95%, so the coverage-gated CI job on this PR is expected to fail until coverage is separately brought up. This PR only fixes the gate's precision, not the underlying coverage shortfall. ## Test plan - [x] `pytest tests/unit_tests/test_ci_environment.py` passes locally - [x] `pre-commit run --files pyproject.toml tests/unit_tests/test_ci_environment.py` passes - [x] DCO sign-off on the commit --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com>
github-actions
Bot
added
the
sla:triage-overdue
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The full test suite failed on main (run 34703944848 at
1f478743e) with exact coverage 94.46%. Root cause: PR #3330 merged despite the 95% gate because (1) the gate compares at zero-decimal precision, so 94.97% rounded to 95% and passed, and (2) CI tested the stale PR headd9b2122instead of the six-commits-advanced main, so the merged main state was never checked pre-merge.What changed
pyproject.toml: set[tool.coverage.report] precision = 2sofail_undercompares the exact percentage — 94.97 < 95 now exits nonzero.scripts/ci/merge_current_main.sh(new): on mirroredpull-request/*checkouts, merge currentorigin/maininto the tested tree (--no-ff, no contributor branch rewrite); a conflict fails validation; no-op on any other ref..github/workflows/unit-tests.yml: run the merge step after everyTest/server-suitecheckout (fetch-depth: 0) and drop thegithub.event_nameguard onforce-run-allso full suites run wheneverbase-refis empty.tests/unit_tests/test_ci_environment.py: cover the merge-script semantics (synthetic-merge parents, no-op on non-mirrored refs), the workflow wiring (3 merge steps, 3fetch-depth: 0), and the coverage config (precision == 2,fail_under == 95.0).Details
flowchart LR A[PR checkout] --> B{refs/heads/pull-request/*?} B -- no --> N[test checked-out commit] B -- yes --> C[fetch origin/main] C --> D{main ancestor of HEAD?} D -- yes --> N D -- no --> E[merge --no-ff origin/main] E -- conflict --> F[fail validation] E --> G[test synthetic merge vs current main] G --> H[coverage precision=2, fail_under=95.0]Tested
d81b5dafddirectly on currentorigin/main1f478743e; remote head verified equal after push.bash -n scripts/ci/merge_current_main.shpasses.nemo-gymunit:core-unit-testsrow could not launch in the authoring environment: the platform checkpoint consumer rejected the materialized source with status 409 on two distinct checkpoint refs after healthy materialization convergence (pre-materialization infrastructure failure; no workload was ever allocated). Local pytest is unavailable (workstation image has no python3) and local git-fixture validation is blocked (its git build disallowsfile://transport). GitHub checks remain authoritative and will run on this PR.