Paramaterised Suites?

Development Swift Testing
1

I've just migrated a project over to Swift Testing from XCTest, and the process was fairly simple, save one feature that is relatively simple to do in XCTest and somewhat less elegant in Swift Testing.

In my original set of XCTests, I had a suite of tests that would test multiple subclasses with different strategies for all common expected behaviour. Using inheritance in the XCTestCase hierarchy and overriding a template method to provide the specific SUT, this was fairly trivial to accomplish.

The same sort of thing doesn't appear to work with Swift Testing, because of the use of macros rather than runtime inference on what functions are and are not test functions. If I have understood this correctly, Swift Testing cannot 'see' the @Test functions from class Test { @Test func something() {} } when evaluating class TestChild: Test { }. The only way to replicate the XCTest behaviour that I have found is therefore to use parameterised tests. This works perfectly well - I have an array of all the subclasses that I want to pass in, and the test runs on them all.

The problem is, that I have an entire suite of tests like this. Every single test function in that suite therefore has to be paramaterised in exactly the same way, with the same arguments, and the same signature. It also makes it hard/messy for different helper functions within the suite to share the same instance of the SUT, as it's passed in at a function level rather as a parameter than at a class/suite level.

Have there been any ideas on how to move forward with this sort of testing in Swift Testing? If just following existing patterns, it seems like there could perhaps be a way to parameterise a suite, to accept an array of arguments that can be applied across the entire suite?

2

It really depends what you mean by "parameterized suites." The topic has come up before, and I don't think we can universally agree on a definition of the term, let alone figure out how to implement it.

Based on your post here, I don't think "parameterized suites" is necessarily the right solution so much as something that sounds like it might be shoehornable. So I'm going to try to address this problem instead:

Hypothetically, if the following compiled, would it solve your problem?

@Suite protocol CommonTests {
  @Test func f()
  @Test func g()
}

extension CommonTests {
  func f() { ... }
  func g() { ... }
}

@Suite struct Tests1: CommonTests {
  // inherits f() and g() automatically
}

As of right now, we can't build anything like the above because our current mechanism for emitting metadata into a binary doesn't let us specialize generic types such as any/some CommonTests at runtime. I'm working on replacing that mechanism with a new one that is more flexible and ought to let us perform that sort of protocol specialization.

3

Yes, that’s exactly the sort of solution I would want, all the better for being in a protocol and not requiring an inheritance hierarchy.

Being able to apply a set of common tests to any suite in this manner would be a great addition in my opinion.

5

Maybe to get a bit more of an idea about what parameterized suites could be, let me give another example.

I'm doing the Elevator Kata. These elevators share some behaviour that both should adhere to. So I'm creating a suite to collect all shared behaviour. Here is a snippet:

@Suite("Every elevator should") struct EveryElevatorTests {
    let clock: MockClock
    let elevators: [any Elevator]
    
    init() {
        clock = MockClock()
        elevators = [
            SlowElevator(clock: clock),
            FastElevator(clock: clock)
        ]
    }

    @Test("start at the ground floor with closed doors") func startAtGroundFloorWithClosedDoors() {
        for elevator in elevators {
            #expect(elevator.elevatorState == .closedDoor)
        }
    }

    @Test("after calling on the ground floor, have doors stay open for three seconds") func doorsStayOpenForThreeSeconds() {
        for elevator in elevators {
            elevator.callAtFloor(.groundFloor)

            clock.advanceTime(by: 2)

            #expect(elevator.elevatorState == .openDoor(floor: .groundFloor))
        }
    }
}

Notice the use of a for loop to drive each elevator.

Ideally, I'd like to write something like:

@Suite("...", arguments: [
    FastElevator(clock: MockClock()),
    SlowElevator(clock: MockClock())
] struct EveryElevatorTests {
    let elevator: any Elevator 

    @Test("...") func someTest() {
         elevator.callAtFloor(...)
         ...
         #expect(elevator....)
    }
    
    ...
}

To generalize: when testing behaviour of instances of a type that adheres to a protocol, validate that they all exhibit the expected behaviour.

6

So the main reason we can't do this today is that the @Test attribute's macro expansion logic can't reliably "see" the @Suite argument and members of the suite type well enough to correctly infer everything. For instance, given your EveryElevatorTests example:

  • The glue code we emit around someTest() would need to know there's an initializer on EveryElevatorTests that takes an instance of any Elevator (or some Elevator or T: Elevator or T where T: Elevator or…), but you haven't declared such an init member;
  • Even if you did declare such an init member, we can't see it. The lexical context that's visible to a macro during expansion is basically just the declaration's signature: @Suite(...) struct EveryElevatorTests {} and nothing else. So we don't know what init to call.
  • If a test is declared in an extension to a suite type, the available lexical context is for the extension only, so the macro can't even see the @Suite attribute.

It's also worth thinking about the algorithmic complexity of such a test suite. Most suite types are going to have more than 2 stored properties, but we really don't want to introduce hidden/accidental… uh… O(n1n2…m) i.e. polynomial complexity here (where n1 is the number of elements in the first arguments collection, etc., and where m is the number of test functions in the test suite.)

This is not to say we aren't interested in exploring this sort of functionality, just that it isn't as straightforward as it may seem.

2 Likes