API to reliably detect if a test is running under Swift Testing or XCTest #475
Comments
|
Hard part: Swift Testing and XCTest don't know about each other, so it's not clear how to write a "one true test library" function or something like that. You can check if I could see us adding a // increment when runner starts, decrement when it ends
private static let _isRunning = Locked(rawValue: 0)
public static var isRunning: Bool {
_isRunning.rawValue > 0
}That should be sufficient API to tell you if Swift Testing is running; if it's not, you could then assume XCTest is running. Would that work? |
|
@grynspan This would definitely be nice to have! Our test tools are currently having to resort to inspecting the process info to determine if the current process is running tests, which is failure-prone. It'd be nice for XCTest to also expose such an API, but I'm guessing that won't happen at this point 😉 |
That would work well enough for now. The next question is being able to record issues without crashing. Last I checked, swift testing throws a runtime error if you try to record an issue when |
This is a known bug in Xcode 16 Beta 1 and is not a bug in Swift Testing (which simply records the issue with no associated test.) |
This PR adds a `Test.isRunning` static property that tells the caller if _any_ thread/task/queue/etc. is currently running Swift Testing code. This property is distinct from `Test.current` because that property has a value of `nil` on detached tasks. Resolves #475.
Description
Like #474, this was also discussed in a lab.
This issue is related to the second half of Quick/Nimble#1145: Determining whether to report errors to Swift Testing or XCTest.
Background about how Nimble determines the test runner
Nimble nominally supports working with other test runners, so that if you wanted, you could use non-XCTest test runners.
The way Nimble works is it dynamically determines the test runner environment the first time an assertion fails. At that point, Nimble checks whether the test runner is XCTest, and then reports the issue to XCTest using the "appropriate" API1. Then, Nimble will cache that check.
The way Nimble checks if the test runner is XCTest is by checking if the
XCTestCaseclass can be found. Which probably needs to be updated. For example, I would expect that check to still pass if Swift Testing is the test runner, just because I would also expect XCTest to be loaded in to the test bundle even if it's not actually used. Especially during this period when Swift Testing is just being introduced, and I can expect a lot of wonkiness going on. So, I'llprobablydefinitely update that check to be more like "is there a currently running XCTestCase"?The actual ask here
In the lab I had, it was mentioned that the only API for determining if a Swift Test is running is
Test.current. However, as you mentioned,Test.currentis a task local value, so if someone uses a detached Task inside of the test, thenTest.currentwill incorrectly return nil (as far as the semantics of checking "am I running inside of a test?" go).What Nimble needs here is a way to reliably and positively identify whether or not it should report the issue to Swift Testing.
Other Possible Approaches for reporting issues (Nimble Specific)
Also in Quick/Nimble#1145, I wrote up that another way that Nimble could determine whether or not to report to Swift Testing or XCTest is to assume that if there is no current running XCTestCase, then it should fall back to Swift Testing. Which probably would work in the interim, but I'd certainly appreciate a way to positively verify that Nimble should report the failure to Swift Testing.
Another approach I wrote up is to try to report to both Swift Testing and XCTest. So, just blasting that a test failure happened (or an error was encountered, or a polling expectation timed out, etc.) to both of them, and hoping that only one of them records the failure. That's also unappealing to me because, again, I'd like to positively verify the right place to report the issue and only report it once. But it's certainly an approach.
Gosh, Rachel, that's a lot. Is there a TL;DR?
I need a way to reliably and positively identify if Nimble (and other testing assertion tools) should report issues to Swift Testing or XCTest.
Expected behavior
No response
Actual behavior
No response
Steps to reproduce
No response
swift-testing version/commit hash
No response
Swift & OS version (output of
swift --version && uname -a)No response
Footnotes
Which just inspired me to write up https://github.com/Quick/Nimble/issues/1146, because Nimble should be using
XCTestCase.recordFailure. ↩The text was updated successfully, but these errors were encountered: