"It is not a test it is a sbt.testing.SuiteSelector" in JUnit XML test reports

[gaeljw]

Hello folks,

Upgrading from JUnit 4 to JUnit 5 in our test suite using Cucumber, we noticed that the test reports (in JUnit XML format) have changed and the testcase name is now a meaningless message "It is not a test it is a sbt.testing.SuiteSelector".

Apart from the name to be incorrect, the side effect is that all test cases have the same name and this can be misinterpreted by some reporting UIs (like GitLab) where a single testcase is shown instead of several.

A reproduction scenario is:

• clone https://github.com/cucumber/cucumber-jvm-scala repo
• run sbt (interactive)
• then compile
• then project examplesJunit4
• then test
• then project examplesJunit5
• then test
• retrieve the test reports files at:
  • examples/examples-junit4/target/jvm-2.13/test-reports/TEST-cucumber.examples.scalacalculator.RunCukesTest.xml
  • examples/examples-junit5/target/jvm-2.13/test-reports/TEST-cucumber.examples.scalacalculator.RunCukesTest.xml

Notice the difference between JUnit 4 and JUnit 5:

- <testcase classname="cucumber.examples.scalacalculator.RunCukesTest" name="Basic Arithmetic.Adding" time="0.055">
+ <testcase classname="cucumber.examples.scalacalculator.RunCukesTest" name="(It is not a test it is a sbt.testing.SuiteSelector)" time="0.051">

The offending text comes from https://github.com/sbt/sbt/blob/f284e809d41f27ceefadc58534c07bf41405a25d/testing/src/main/scala/sbt/JUnitXmlTestsListener.scala#L129C1-L136C16 but it sounds like this might be an issue in this plugin?

---

This has been initially reported in cucumber/cucumber-jvm-scala#403.

[gaeljw]

Regarding this one, I've had a look at the code but I'm a bit confused how things integrate with each other. If someone with more knowledge can give some guidance, I'll be happy to spend some time on a PR. For now I'm a bit lost :(

Should sbt support JUnit XML reporting with SuiteSelector and enrich SuiteSelector with details about nested tests? Or should this plugin somehow not use SuiteSelector but build something different?