Paramaterised Suites?

Development Swift Testing
1

I've just migrated a project over to Swift Testing from XCTest, and the process was fairly simple, save one feature that is relatively simple to do in XCTest and somewhat less elegant in Swift Testing.

In my original set of XCTests, I had a suite of tests that would test multiple subclasses with different strategies for all common expected behaviour. Using inheritance in the XCTestCase hierarchy and overriding a template method to provide the specific SUT, this was fairly trivial to accomplish.

The same sort of thing doesn't appear to work with Swift Testing, because of the use of macros rather than runtime inference on what functions are and are not test functions. If I have understood this correctly, Swift Testing cannot 'see' the @Test functions from class Test { @Test func something() {} } when evaluating class TestChild: Test { }. The only way to replicate the XCTest behaviour that I have found is therefore to use parameterised tests. This works perfectly well - I have an array of all the subclasses that I want to pass in, and the test runs on them all.

The problem is, that I have an entire suite of tests like this. Every single test function in that suite therefore has to be paramaterised in exactly the same way, with the same arguments, and the same signature. It also makes it hard/messy for different helper functions within the suite to share the same instance of the SUT, as it's passed in at a function level rather as a parameter than at a class/suite level.

Have there been any ideas on how to move forward with this sort of testing in Swift Testing? If just following existing patterns, it seems like there could perhaps be a way to parameterise a suite, to accept an array of arguments that can be applied across the entire suite?

2

It really depends what you mean by "parameterized suites." The topic has come up before, and I don't think we can universally agree on a definition of the term, let alone figure out how to implement it.

Based on your post here, I don't think "parameterized suites" is necessarily the right solution so much as something that sounds like it might be shoehornable. So I'm going to try to address this problem instead:

Hypothetically, if the following compiled, would it solve your problem?

@Suite protocol CommonTests {
  @Test func f()
  @Test func g()
}

extension CommonTests {
  func f() { ... }
  func g() { ... }
}

@Suite struct Tests1: CommonTests {
  // inherits f() and g() automatically
}

As of right now, we can't build anything like the above because our current mechanism for emitting metadata into a binary doesn't let us specialize generic types such as any/some CommonTests at runtime. I'm working on replacing that mechanism with a new one that is more flexible and ought to let us perform that sort of protocol specialization.

3

Yes, that’s exactly the sort of solution I would want, all the better for being in a protocol and not requiring an inheritance hierarchy.

Being able to apply a set of common tests to any suite in this manner would be a great addition in my opinion.

4

(post deleted by author)