Challenges with `XCTAssert` in `swift-testing` and 3rd Party Libraries

Hi there!

I've encountered a few instances where developers are using helper methods within the swift-testing land that depend on XCTAssert. Unfortunately, these assertions don't seem to be executing at all in swift-testing-based tests.

It’s important to note that there are many third-party libraries offering testing helpers that utilize XCTAssert under the hood. This becomes particularly problematic when developers transition from XCTest to swift-testing, as there's a significant risk that these tests won't run assertions at all, and worse, developers might not even notice.

Do you think there's a need to address this issue more broadly so that people are aware of this constraint?

Additionally, I'm really curious to learn why XCTAssert is incompatible with swift-testing.

Thanks,
Mateusz

I think you may have a misunderstanding about the relationship between XCTest and Swift Testing?

Do you think there's a need to address this issue more broadly so that people are aware of this constraint?

There's no constraint — the XCTAssert functions are all part of the older XCTest framework, not part of Swift Testing. They are completely separate frameworks, and require completely different annotations in code to expose tests and check expectations/requirements.

New third party libraries can (and should) be written to fill the roles of the old XCTest-backed approaches where they still make sense.

If you haven't already, you should watch the "Meet Swift Testing" session from WWDC24.

@tonyarnold, thanks for your response. I apologize if my initial message was unclear.

I understand the distinct nature of XCTest and swift-testing. But, my concern is rooted in observing both public forums and private discussions where developers appear not to be aware of that.

You mentioned that new third-party libraries could fill the roles previously handled by XCTest helpers, which I agree with. However, there seems to be a lack of guidance on how these libraries can support both frameworks effectively–is it ok to implement both types of assertions or do we need to recognize which framework runs tests?. In an ideal world, libraries would seamlessly support both XCTest and swift-testing, but we're not quite there yet and I strongly believe that some very popular ones will not add support for months. It’s not uncommon for developers to inadvertently use outdated libraries, unaware of compatibility issues until they encounter non-functioning tests.

I've started this discussion to consider if more explicit warnings or some sort of compatibility checks could be integrated into the new library to alert developers when they attempt to use XCTest functions in swift-testing environments since I'm convinced that that kind of situation will occurs multiplie times.

1 Like

Hey no worries, and no need for apologies!

I don't see any technical reason why a library couldn't support both XCTest and Swift Testing. Apple have been clear that it's possible to use both in a single target so that we can incrementally migrate to Swift Testing in time.

Given that XCTest functions aren't available unless you import XCTest, I'm not sure that misuse of the manner you're describing needs calling out. The call sites to XCTest functions when the framework isn't imported won't compile.

Can you explain in more detail what you think the warnings/errors might look like?

If a function is marked @Test, it is possible for Swift Testing to detect uses of XCTest functions at compile time and emit warnings with LLVM/swiftc Fix-Its. Would that help?

As we discussed outside the forums—this could cover only a scenario when the XCTest function is called directly in the body of the test method. If the "assertion" part of the method is implemented outside the body of the test method, we are not able to detect it at compile time. :/ Great example of this case is calling assertMacroExpansion function from the swift-syntax package.

1 Like