fix(ci): AUT-2282 exact coverage gate precision - #3373
Merged
kajalj22 merged 3 commits intoSep 14, 2026
Merged
Conversation
…ct percentages PR NVIDIA-NeMo#3330 merged despite failing the 95% coverage gate because the report's default zero-decimal precision rounded 94.97% up to 95%, letting a failing run report as passing. Setting [tool.coverage.report] precision = 2 makes fail_under compare the exact percentage instead. Also adds unit coverage for previously-uncovered code paths (decorators.experimental, judge.py error passthrough/failsafe guard, _verifier_runner path/type validation, sandbox/utils cpu_cap_env and rewrite_image) so this change clears the tightened gate on its own. Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Keep this PR scoped to the precision=2 config fix and its verifying test. Filling the pre-existing coverage gap is a separate concern. Signed-off-by: Kajal Jain <kajalj@nvidia.com>
Contributor
Author
|
/ok to test ba14156 |
cmunley1
approved these changes
Sep 14, 2026
chtruong814
approved these changes
Sep 14, 2026
yaoyu-33
added
area:ci
pull Bot
pushed a commit
to Stars1233/Gym
that referenced
this pull request
Sep 15, 2026
…re coverage (NVIDIA-NeMo#3398) ## Background The scheduled Full test suite on main began failing its coverage gate: `Total coverage: 94.83%` < `fail_under = 95.0` (run 34890370042 on 84fbd3d, every run since 2026-09-14). The gate's precision fix (NVIDIA-NeMo#3373) exposed that coverage had silently dropped below 95%. ## What changed The full-test-suite job installs deps with `uv sync --extra dev --extra sandbox --extra openshell`, omitting the `telemetry` extra. The tests under `tests/unit_tests/telemetry/` are gated on nemo-lens (`requires_lens`), which only the telemetry extra installs, so they all skipped and the telemetry modules (`setup.py`, `endpoints.py`, `metrics.py`, `contrib.py`, `_fallbacks.py`) counted as uncovered, dragging the total below the gate. This adds `--extra telemetry` to that install so the tests run and coverage includes telemetry, mirroring `scripts/ci/core_unit_tests.sh` (which syncs the same extra). PR CI was unaffected because `unit-tests.yml` runs `core_unit_tests.sh`. ## Details ```mermaid flowchart LR A[full-test-suite.yml] -->|uv sync dev/sandbox/openshell| B[nemo-lens MISSING] B --> C[telemetry tests SKIP] C --> D[telemetry modules uncovered] D --> E[coverage 94.83% < 95%] F[core_unit_tests.sh] -->|uv sync dev/telemetry| G[nemo-lens present] ``` Left = the broken scheduled path fixed here; right = the PR path that already worked. A regression test `test_full_test_suite_installs_telemetry_extra_for_coverage_gate` fails if the extra is dropped again. ## Tested - 2 files, +22/-1: the workflow install line (+comment) and the new guard test. - Root cause confirmed from the failing run log (telemetry tests `SKIPPED`, coverage 94.83%) and by diffing `full-test-suite.yml` against `core_unit_tests.sh`/`unit-tests.yml`. - The managed `nemo-gym` unit suite runs against this head once its CI image build completes; PR CI validates the full gate. --------- Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The full test suite failed on
mainwith exact coverage 94.46%. Root cause (per triage on PR #3330's merge): the coverage gate compares at zero-decimal precision, so an actual 94.97% rounded to 95% and incorrectly passed the 95% gate before merge.This PR addresses only that rounding gap. It intentionally excludes the stale-PR-head / merge-current-main fix tracked separately in #3338, and does not add tests to close the pre-existing coverage gap (a separate concern).
What changed
pyproject.toml: set[tool.coverage.report] precision = 2sofail_undercompares the exact percentage — 94.97 < 95 now exits nonzero.tests/unit_tests/test_ci_environment.py: assertprecision == 2andfail_under == 95.0in the coverage config.Note
With this change alone, the repo's aggregate coverage is still below 95%, so the coverage-gated CI job on this PR is expected to fail until coverage is separately brought up. This PR only fixes the gate's precision, not the underlying coverage shortfall.
Test plan
pytest tests/unit_tests/test_ci_environment.pypasses locallypre-commit run --files pyproject.toml tests/unit_tests/test_ci_environment.pypasses