[Accepted] A New Direction for Testing in Swift

[grynspan (Jonathan Grynspan)]

It was very hard not to reply to everyone's comments over the weekend! You all had a lot of thoughts to share, and I've tried to answer a selection of points/questions here.

@wadetregaskis

It's not very elegant, though, is it? Lots of boilerplate.

I don't think there's more boilerplate than necessary. This contrived example just shows a specific possible use case, and has no real test content, so the structure of the test dominates. Real-world tests using swift-testing have a better boilerplate-to-content ratio.

@wadetregaskis

How will this new testing framework treat that? Will it deduce the parameterised nature of the test and run all cases rather than failing at the first; will it present the results in a suitable form (e.g. a table) to help make clear which specific test cases are problematic?

No, the testing library is not going to try to turn a test function into a parameterized test just because it has a for-loop in it, nor is that a direction we'd likely pursue in the future. Imagine a test like this one that is looking for a non-ASCII character in a string:

@Test func stringIsAllASCII() {
  for c in someString {
    #expect(c.isASCII)
  }
}

It would likely be a mistake to turn this into a parameterized test over the collection someString. This example is, again, contrived, so the danger may not be apparent: imagine that the for loop is over some non-sendable generated sequence rather than over a constant sequence. Because a parameterized test can run its individual test cases in parallel, automatically turning this for loop into a parameterized test could produce unexpected downstream effects.

@kainjow

• Will tests be skippable, based on a runtime action? For example OS version?

Yes! Use the .disabled(), .disabled(if:), or .enabled(if:) traits to conditionalize test execution.

@kainjow

• Can suites have setup and tear down? I’ve found some tests can have significant speed ups by doing setup once per suite and XCTests supports this well.

We're leveraging the features of the Swift language to support setup and teardown with init/deinit as well as with static let. We recognize there are usage patterns that may not be covered here, and we're still looking at how we can cover those patterns in a way that isn't ersatz. @smontgomery and I would be happy to discuss this more with you in a dedicated thread if you'd like.

@tera

FWIW in C the use sites of macros are not polluted with those @ or #, only the declaration sites. The above would look, say:

We've discussed this particular topic with the core Swift team at length. I don't presume to speak for them (@hborla or @Douglas_Gregor might be able to shed more light than I can) but in a nutshell, it's important that Swift macro invocations be visible at their usage sites rather than looking like function calls. This is why you have to write #expect() instead of expect(). As for @, well, that's how Swift spells attributes anyway like @Sendable or @MainActor or @TaskLocal, and we're consistent with the (rest of the) language here.

The macros used by swift-testing are not just wrappers around function calls. Having @Test and #expect() be spelled Test and/or expect() would require changes to the Swift compiler and the introduction of new language- and compiler-level features just to support test compilation, which is a non-goal for the Swift project at large.

@hassila

It feels nice to have the parameterization on the outside, but agree that the contrived test one was a bit hard to read.

If it helps, you can use the same sort of pattern with swift-testing:

let parameterization = (0...15).map { 1 << $0 } // 1, 2, 4, ...

@Test(arguments: parameterization)
func iterations(count: Int) {
  ...
}

You could go further and have the inner loop be part of the parameterization as well:

let parameterization = (0...15)
  .map { 1 << $0 } // 1, 2, 4, ...
  .map { 0 ..< $0 } // 0..<1, 0..<2, 0..<4, ...
@Test(arguments: parameterization)
func iterations(range: ClosedRange<Int>) {
  ...
}

@wadetregaskis

Consider also something like:

@Test(0 -> 0)
@Test(1 -> 10)
@Test(10 -> 100)
func timesTen(i: Int) -> Int {
    i * 10
}

I think that -> specifically is reserved by the language, but I would be happy to be wrong there. A pattern like this would be an interesting area to explore. We have intentionally not added any sort of support for return types other than Void so far, but that doesn't mean we couldn't in the future. (I imagine in this case that the custom -> operator would need its right-hand operand to conform to Equatable.) We can start a separate thread to discuss further if you'd like.

@taylorswift

this feels pretty closely related to testing fatal errors which is something that would fill a huge gap in our tooling.

This is something we're exploring for a future release. We call these sorts of tests "exit tests" and we have an experimental implementation already. See my post here for more info.