I've just migrated a project over to Swift Testing from XCTest, and the process was fairly simple, save one feature that is relatively simple to do in XCTest and somewhat less elegant in Swift Testing.
In my original set of XCTests, I had a suite of tests that would test multiple subclasses with different strategies for all common expected behaviour. Using inheritance in the XCTestCase hierarchy and overriding a template method to provide the specific SUT, this was fairly trivial to accomplish.
The same sort of thing doesn't appear to work with Swift Testing, because of the use of macros rather than runtime inference on what functions are and are not test functions. If I have understood this correctly, Swift Testing cannot 'see' the @Test functions from class Test { @Test func something() {} } when evaluating class TestChild: Test { }. The only way to replicate the XCTest behaviour that I have found is therefore to use parameterised tests. This works perfectly well - I have an array of all the subclasses that I want to pass in, and the test runs on them all.
The problem is, that I have an entire suite of tests like this. Every single test function in that suite therefore has to be paramaterised in exactly the same way, with the same arguments, and the same signature. It also makes it hard/messy for different helper functions within the suite to share the same instance of the SUT, as it's passed in at a function level rather as a parameter than at a class/suite level.
Have there been any ideas on how to move forward with this sort of testing in Swift Testing? If just following existing patterns, it seems like there could perhaps be a way to parameterise a suite, to accept an array of arguments that can be applied across the entire suite?
It really depends what you mean by "parameterized suites." The topic has come up before, and I don't think we can universally agree on a definition of the term, let alone figure out how to implement it.
Based on your post here, I don't think "parameterized suites" is necessarily the right solution so much as something that sounds like it might be shoehornable. So I'm going to try to address this problem instead:
Hypothetically, if the following compiled, would it solve your problem?
As of right now, we can't build anything like the above because our current mechanism for emitting metadata into a binary doesn't let us specialize generic types such as any/some CommonTests at runtime. I'm working on replacing that mechanism with a new one that is more flexible and ought to let us perform that sort of protocol specialization.
Maybe to get a bit more of an idea about what parameterized suites could be, let me give another example.
I'm doing the Elevator Kata. These elevators share some behaviour that both should adhere to. So I'm creating a suite to collect all shared behaviour. Here is a snippet:
@Suite("Every elevator should") struct EveryElevatorTests {
let clock: MockClock
let elevators: [any Elevator]
init() {
clock = MockClock()
elevators = [
SlowElevator(clock: clock),
FastElevator(clock: clock)
]
}
@Test("start at the ground floor with closed doors") func startAtGroundFloorWithClosedDoors() {
for elevator in elevators {
#expect(elevator.elevatorState == .closedDoor)
}
}
@Test("after calling on the ground floor, have doors stay open for three seconds") func doorsStayOpenForThreeSeconds() {
for elevator in elevators {
elevator.callAtFloor(.groundFloor)
clock.advanceTime(by: 2)
#expect(elevator.elevatorState == .openDoor(floor: .groundFloor))
}
}
}
Notice the use of a for loop to drive each elevator.
Ideally, I'd like to write something like:
@Suite("...", arguments: [
FastElevator(clock: MockClock()),
SlowElevator(clock: MockClock())
] struct EveryElevatorTests {
let elevator: any Elevator
@Test("...") func someTest() {
elevator.callAtFloor(...)
...
#expect(elevator....)
}
...
}
To generalize: when testing behaviour of instances of a type that adheres to a protocol, validate that they all exhibit the expected behaviour.
So the main reason we can't do this today is that the @Test attribute's macro expansion logic can't reliably "see" the @Suite argument and members of the suite type well enough to correctly infer everything. For instance, given your EveryElevatorTests example:
The glue code we emit around someTest() would need to know there's an initializer on EveryElevatorTests that takes an instance of any Elevator (or some Elevator or T: Elevator or T where T: Elevator or…), but you haven't declared such an init member;
Even if you did declare such an init member, we can't see it. The lexical context that's visible to a macro during expansion is basically just the declaration's signature: @Suite(...) struct EveryElevatorTests {} and nothing else. So we don't know what init to call.
If a test is declared in an extension to a suite type, the available lexical context is for the extension only, so the macro can't even see the @Suite attribute.
It's also worth thinking about the algorithmic complexity of such a test suite. Most suite types are going to have more than 2 stored properties, but we really don't want to introduce hidden/accidental… uh… O(n1n2…m) i.e. polynomial complexity here (where n1 is the number of elements in the first arguments collection, etc., and where m is the number of test functions in the test suite.)
This is not to say we aren't interested in exploring this sort of functionality, just that it isn't as straightforward as it may seem.
I've been struggling with how to test that conforming implementations of a protocol meet certain requirements; I think "parameterized suite" describes this idea well.
I have a protocol, let's call it RegularLanguage, that specifies a certain interface that conforming types must implement. (For example, an epsilon property and a concatenate function.) It makes sense to me that I should be able to define an associated protocol RegularLanguageTests that I can assign to my test suites, which will automatically provide a certain number of tests for an implementation of RegularLanguage. For example, that T.epsilon.concatenate(T.epsilon) == T.epsilon must always be true.
However, Testing expands the macros to include static properties, so I can't use @test inside a protocol extension.
It would also work if I could write a generic struct, so I could define all my tests in struct RegularLanguageTests<T: RegularLanguage> { ... }. I would expect Swift Testing to only run tests in the concrete expansions of the generic struct, but apparently it performs the @suite macro expansion before Swift expands the generic, and it throws an error saying "Attribute 'Test' cannot be applied to a function within generic structure"
Yes, that's expected and unavoidable. It is not possible to expand @Test without making use of static members on the current type (where one exists) because we can only emit peers of test functions and, if they are member functions, peers must also be members of that type.
This diagnostic is emitted by the @Test macro itself and is emitted because we cannot produce a valid expansion in a generic type for the same reasons we can't produce one in a protocol extension: static storage is not available and even if it were, we wouldn't know how to specialize your generic type at runtime.
I would recommend creating a helper structure to contain the specialization information you need. Something like this:
protocol P {
static var epsilon: T { get }
...
}
struct MyProtocolTests {
struct PHelper {
var type: any P.Type
init<T: P>(_ type: T.Type) {
self.type = type
}
}
@Test(arguments: [PHelper(ConcreteType1.self), PHelper(ConcreteType2.self), ...])
func epsilonConcatenatedWithSelf(_ helper: PHelper) {
func open<T: P>(_ type: T.Type) {
#expect(T.epsilon.concatenate(T.epsilon) == T.epsilon)
}
open(helper.type)
}
}
The nested function open() "opens" the existential type inside each instance of PHelper, then does the actual test work.
So why is static storage is not available on the expanded type? I get that you can't use @Test on a generic type like RegularLanguageTests<T>, as that's not actually a type that's going to be in the build output. But the expansion RegularLanguageTests<RegularLanguageOne> is a concrete type (except the name will be mangled). Why is Swift Testing trying to apply the macros before expanding the generics, instead of after?
I can probably make use of this technique in a few places. I don't believe it's going to be a complete solution, as I use parameterized tests in several places. For example, taking several different strings and converting them back and forth, once for each file format, to ensure there's no loss of information.
Macros in Swift are syntax-based and do not have access to type information. We can only go by the characters you've typed in your source file, parsed into a Swift AST.
Generic specialization generally takes place at runtime (although compile-time optimizations can and do occur under various conditions).
The @Test macro emits a type at compile time that conforms to a special protocol. At runtime, Swift Testing walks Swift's type metadata section looking for types that conform to this protocol and "realizing" them. This approach to test discovery cannot be extended to generic types because "realizing" them requires more metadata than is available to us. Hence, we cannot discover tests in generic types even if our macros produce valid expansions at compile time.
We have a solution to this problem pending which will move test discovery into a dedicated Mach-O/ELF/COFF section, but we can't enable it until support for the @section attribute is completed/approved.
Somehow I was under the impression that macro expansion is fundamentally the same mechanism as generics. Also you can "export" generics, but this is an illusion performed with witnesses and other mechanisms behind the scenes that transforms protocols and generics into related concrete types and with some runtime performance cost. I'll take another look at this.
For better or for worse, the mechanisms you're thinking of aren't available to us at macro expansion time. Any nested types we emit are necessarily embedded inside your generic suite type and are themselves considered generic in the type metadata section that we look at—meaning we still don't have the information necessary to "realize" them.