Skip to content

throw if no test items found#216

Merged
nickrobinson251 merged 11 commits into
mainfrom
npr-throw-no-tests
Jul 18, 2025
Merged

throw if no test items found#216
nickrobinson251 merged 11 commits into
mainfrom
npr-throw-no-tests

Conversation

@nickrobinson251
Copy link
Copy Markdown
Member

@nickrobinson251 nickrobinson251 commented Jun 30, 2025

It's always seemed slightly odd that running 0 tests is a "success", in #215 we at least made it look less successful by logging a warning, but ultimately this doesn't catch issues (like we've seen at RAI) where e.g. changes in CI test setup leads to some job running 0 tests and therefore the job succeeding and therefore the issue (that we're no longer running tests) being missed.

If you runtests and fail to run any tests, i think that should be an error. So this PR makes it an error.

Here's how it looks:

julia> runtests("test/testfiles/_happy_tests.jl"; name="blahahah_nope")
[ Info: Scanning for test items in project `ReTestItems` at paths: test/testfiles/_happy_tests.jl
[ Info: Finished scanning for test items in 0.04 seconds.
ERROR: No test items found.

Note we follow the precendent of Test.TestSetException and print just the ERROR: ... line without a subsequent backtrace

e.g. compare this where there's no backtrace after the final Error: ... line

julia> @testset "foo" begin
           error("barf")
       end
foo: Error During Test at REPL[31]:1
  Got exception outside of a @test
  barf
  Stacktrace:
    [1] error(s::String)
      @ Base ./error.jl:35
    ...snip...
   [16] _start()
      @ Base ./client.jl:552
Test Summary: | Error  Total  Time
foo           |     1      1  0.0s
ERROR: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.

julia>

This PR also updates the error when the file doesn't exist (when validate_paths=true, or no valid files were given, resulting in no testitems):

julia> runtests("no_such_tests.jl"; validate_paths=true)
ERROR: No such path "no_such_tests.jl"

Base automatically changed from npr-warn-no-tests to main July 7, 2025 10:50
@nickrobinson251 nickrobinson251 marked this pull request as ready for review July 18, 2025 13:30
@nickrobinson251 nickrobinson251 requested review from Drvi and NHDaly July 18, 2025 14:28
Copy link
Copy Markdown
Member

@Drvi Drvi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, thanks!

@nickrobinson251 nickrobinson251 merged commit 82bc433 into main Jul 18, 2025
8 checks passed
@nickrobinson251 nickrobinson251 deleted the npr-throw-no-tests branch July 18, 2025 17:34
@nickrobinson251 nickrobinson251 mentioned this pull request Jul 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants