On the road to seeing how I could implement a custom macro for snapshot tests I managed to get a form of MVP working which wasn't too bad, but along the way found some limitations of the fact I was making use of Swift Testing instead of adding directly to it.
In a spike I looked in to if I could add to what already exists in Swift Testing to see how I would have written this directly in the source code. This led me to the below draft pitch - an expectation argument on @Test and @Suite that makes use of a @Test function's return type...
It's my first pitch, but I'm excited to hear any feedback and suggestions on it.
[Pitch] - new expectation argument on @Test and @Suite
Motivation
Today, writing a test that relies on producing lazily computed, opaque values is cumbersome and arguments can quickly get bloated when we think about more complex types.
We have to rely on a @Test function which either:
- takes an arguments of closures or
- relies on separate functions/helpers to create what we need
This is especially true for opaque types which are currently not supported within @Test arguments.
One way to implement this is to have two functions with names that are similar/overloaded so one can create an instance, and the other can be a @Test that does the assertion.
These naming conventions aren't required techinically but do hint at the functions being intrinsically linked.
However, this requires bookkeeping, a lot of copy/paste, quickly becomes tedious and crucially, is not maintainable or scalable.
// ⚠️ Today's implementation
struct MotivationMirroredNameTests {
@Test
func myView() {
#expect(assertSnapshot(makeMyView()) == true)
}
func makeMyView() -> some View {
Text("Some text")
}
@Test
func anotherView() {
#expect(assertSnapshot(makeAnotherView()) == true)
}
func makeAnotherView() -> some View {
Text("Some other text")
}
}
func assertSnapshot<V: View>(_ view: V) -> Bool {
.random()
}
And while we could make use of arguments for parameterised testing here, if the functions return different views, we'd have to still create separate functions.
Using opaque types produces a compiler error.
// ❌ Doesn't compile due to `arguments` now returning generics in the form of `some View`.
struct MotivationArgumentsTests {
@Test( // ❌ Type 'Any' does not conform to the 'Sendable' protocol
arguments: [
makeMyView,
makeAnotherView
]
)
func myView(makeView: () -> some View) { // ❌ Attribute 'Test' cannot be applied to a generic function
#expect(assertSnapshot(makeView()) == true)
}
func makeMyView() -> some View {
Text("Some text")
}
func makeAnotherView() -> some View {
Text("Some other text")
}
}
Proposal
Add a new expect parameter on both @Test and @Suite to encapsulate all the above boilerplate down to one function which has a return value.
This is a new type which can be used in more complex cases where custom expectations are needed, can be implemented once and reused many times, similar to traits.
Taking inspiration from custom Traits on both @Test and @Suite, developers would be able to implement a new Expectation protocol to handle custom evaluation.
expect would likely be the last parameter on @Test and @Suite which have many arguments.
This would follow the current format of reading the #expect clause last when glancing at a test, but there is potential to have this parameter before arguments which could be a very long list and might read better.
// Using new `expect` parameter on @Suite
@Suite(expect: .snapshots) // 👈🏻 New parameter
struct ProposalTests {
@Test
func myView() -> some View {
Text("Some text")
}
@Test
func anotherView() -> some View {
Text("Some other text")
}
}
// ... 👆🏻 would be equivalent to today's code below 👇🏻 ...
struct ProposalEquivalentTests {
@Test
func myView() {
#expect(assertSnapshot(makeMyView()) == true)
}
func makeMyView() -> some View {
Text("Some text")
}
@Test
func anotherView() {
#expect(assertSnapshot(makeAnotherView()) == true)
}
func makeAnotherView() -> some View {
Text("Some other text")
}
}
Simplified (removed boilerplate)
Valid examples
// ✅ Can be attached to an individual Test
@Suite
struct ProposalSingularTests {
@Test(expect: .snapshots)
func myView() -> some View {
Text("Some text")
}
}
// ✅ Can be attached to a Suite for all children to inherit
@Suite(expect: .snapshots)
struct ProposalMultipleTests {
@Test // Implicitly @Test(expect: .snapshots)
func myView() -> some View {
Text("Some text")
}
@Test // Implicitly @Test(expect: .snapshots)
func anotherView() -> some View {
Text("Some other text")
}
@Suite // Implicitly @Suite(expect: .snapshots)
struct ProposalChildTests {
@Test // Implicitly @Test(expect: .snapshots)
func myChildView() -> some View {
Text("Child text")
}
@Test // Implicitly @Test(expect: .snapshots)
func anotherChildView() -> some View {
Text("Some other child text")
}
}
}
Invalid examples
Return types
// ❌ Mix and match return types
@Suite(expect: .snapshots)
struct ProposalReturnTypesTests {
// ✅ Valid test
@Test
func myView() -> some View {
Text("Some text")
}
/*
❌ Invalid test.
Generates a diagnostic error due to mismatching return type `String` (as defined by `SnapshotsExpectation`)
*/
@Test
func aMismatchingReturnType() -> String {
"bar"
}
/*
❌ Invalid test.
Generates a diagnostic error due to missing return type (as defined by `SnapshotsExpectation`)
*/
@Test
func aMissingReturnType() {
#expect("foo" == "bar")
}
}
Explicit inheritance
@Suite(expect: .snapshots)
struct ProposalInheritanceTests {
@Test(expect: .snapshots) // Redundant, so would compile
func myView() -> some View {
Text("Some text")
}
/*
❌ Invalid test.
Generates a diagnostic error due to the child having different expectation than parent.
Designed for readability and maintainability of test targets, this isn't allowed.
People glancing at tests may see the @Suite with `.snapshots` at the top of the file which inside has
potentially dozens or even hundreds of funcs returning `some View`.
Somewhere inside of that Suite could be a test with a different expectation perhaps accidentally left behind after a
refactor, or added here to quickly test some implementation.
This test could get lost to time and doesn't really belong in that Suite of snapshot tests as defined by the `@Suite(expect:)`.
*/
@Test(expect: .successfulHttpStatusCodes)
func isValidHttpStatusCode() -> Int {
400
}
/*
❌ Invalid test.
Generates a diagnostic error due to the child having different expectation than parent, even though the inner functions are all `.snapshots` too
*/
@Suite(expect: .successfulHttpStatusCodes)
struct ProposedInheritanceChildTests {
@Test(expect: .snapshots)
func anotherView() -> some View {
Text("Some other text")
}
}
}
Explicit #expect
@Suite(expect: .snapshots)
struct ExplicitExpectTests {
// ❌ Invalid test? Potentially generates a diagnostic error due to an explicit #expect in the @Test's function body.
@Test
func myView() -> some View {
#expect(Bool(false))
return Text("Some text")
}
}
Detailed design
Valid @Test with new Expectation
All existing code is still valid, this change doesn't break existing tests.
For a @Test to be considered to have a valid expectation it must:
- Have an
expectparameter- Explicitly added to a
@Testfunction or inherited from a@Suite
- Explicitly added to a
- Have a return type this is both:
- Non-void (Void functions with an expect will cause a diagnostic to be thrown)
- Exactly matches the
Expectation.Valuefrom its expect parameter, again, either explicity or inherited from a parent Suite.
New types
Standard types
/// Sits alongside protocols like `Trait`/`TestTrait`/`SuiteTrait` for developers to inherit from and create custom expectations behaviour.
public protocol Expectation {
associatedtype Value
/// We possibly want to mimic `__checkValue()`'s `Result<Void, any Error>` return type here, but for now throw if there's an error (and subsequently fail)
func checkValue(_ value: Value) async throws
}
In our macro definitions, expect is a singular, optional, scalar value.
By default these will be nil, in which case the test will proceed as it does today.
@attached(member) @attached(peer)
public macro Suite(expect: (any Expectation)? = nil) = #externalMacro(module: "TestingMacros", type: "SuiteDeclarationMacro")
// ... Plus other variants for displayName, traits, etc.
@attached(peer)
public macro Test(expect: (any Expectation)? = nil) = #externalMacro(module: "TestingMacros", type: "TestDeclarationMacro")
// ... Plus other variants for displayName, traits, etc.
User defined examples of potential custom expectations
// Snapshots
extension Expectation where Self == SnapshotsExpectation<AnyView> {
static var snapshots: Self {
Self()
}
}
struct SnapshotsExpectation<C: View>: Expectation {
typealias Value = C
func checkValue(_ value: some View) async throws {
#expect(assertSnapshotUsingThirdPartyLibrary(value) == true)
}
func assertSnapshotUsingThirdPartyLibrary(_ value: some View) -> Bool {
.random()
}
}
// HTTP Status Codes
extension Expectation where Self == HttpStatusCodesExpectation {
static var successfulHttpStatusCodes: Self {
Self(range: 200 ..< 300)
}
static var clientErrorHttpStatusCodes: Self {
Self(range: 400 ..< 500)
}
static var serverErrorHttpStatusCodes: Self {
Self(range: 500 ..< 600)
}
}
struct HttpStatusCodesExpectation: Expectation {
typealias Value = Int
let range: Range<Int>
func checkValue(_ value: Int) async throws {
#expect(range.contains(value))
}
}
Changes needed
Return types
Currently Swift Testing throws a warning and discards a functions return type as it's unused.
There is a code comment for this currently having no use cases so it seems this idea of making use of the return types is a valid one technically speaking.
Equally, those return types need to be passed around and various places assume Void, these will need to be updated to the Evaluation.Type / return type of the @Test function.
There's two options here:
- Pass around Any? which will have minimal impact but doesn't have the type safety of the Expectation until later down the chain
- Update Test to take a generic to pass around the specific type from the function
Despite minimal knowledge of the code base I was able to quickly do the former to get a spike working, but the latter is potentially a much bigger change with bigger knockon effects as all the types using Test and Test.Case etc. will all need updating, potentially creating breaking changes.
Expectation
The new Expectation type will need to be extracted and used in place when tests are run.
Isolation access
Specific types like UIKit and SwiftUI will need processing on the main actor.
The configuration type has a defaultSynchronousIsolationContext property which could be made use of here, where developers could/should set their configuration during when creating their Expectation conformance.
However, changing this configuration could have negative impacts on other traits which have also set their own configuration.
It's more likely we'll need to consider something more implicit such as guaranteeing the @Test function is called on the main actor if the function explicitly says it should, or inherits it from a suite/parent suite.
Benefits
Grouping tests with similar expectations in to child suites.
While the primary examples are relating to snapshot testing, this idea can be used in any way the developer sees fit to create a custom expectation that can encapsulate any logic they want for any type returned by the @Test function.
This reduces the need for global helper functions for doing complex assertions which are not very discoverable or maintainable.
Equally, if adding new tests to a Suite, the expectation will already be set on the Suite so a developer only needs to go in and add a function that returns their sut and all the plumbing will automatically be set up.
if they try to return a type not supported by the expectation, they can be lead with diagnostics within Xcode to guide them with very little knowledge of an existing code base's helper functions.
For example, a developer might write a @Suite of tests for a state machine and break this down to having child suites for each possible state.
A developer can write a custom expectation for their state machine .state(.waiting), .state(.processing), .state(.complete) and attach them to each child state, with each function of the child suite returning a state machine in the various states.
While a developer might think about parameterised testing, this could be difficult to read for many states, or even impossible if their state machine makes use of generics and opaque types.
Functions that return a specific type can be far more concise and easier to read.
@Suite
struct StateMachineTests {
@Suite(expect: .stateMachine(.waiting))
struct WaitingTests {
@Test
func myView() -> some StateMachine {
MyStateMachine(state: .waiting) // ✅ Passes
}
@Test
func myView() -> some StateMachine {
MyStateMachine(state: .complete) // ❌ Failes
}
}
@Suite(expect: .stateMachine(.complete))
struct CompleteTests {
@Test
func myView() -> some StateMachine {
MyStateMachine(state: .complete) // ✅ Passes
}
}
}
protocol StateMachine {
associatedtype State
var state: State { get }
}
// (Declarations for MyStateMachine, State etc...)
Passing more complex expectations
With our own Expectation protocol conformances, we can create complex logic (similar to traits), passing in more information for customisation of the expectation when checkValue() is called.
In the below example we are explicitly saying we 'expect the views to match snapshot sizes of these specific devices'.
You could imagine checkValue() then going on use this configuration when implementing the logic in the custom Expectation.
@Suite(expect: .snapshots(sizes: .iPhone16, .iPadPro11))
struct EnhancementsTests {
@Test
func myView() -> some View {
Text("Some text")
}
}
Working with traits
There's a potential here for the new expects to be able to work with traits.
In the above example we pass the sizes to the expectation. This is specific to what we 'expect' our test to be, like arguments for an #expect, or multiple #expect values encapsulated in our .snapshots type.
This creates a clear distinction between traits and the new proposed expectations.
However there may be use cases where traits have value to use alongside these expectations, like the below example where we want to force record all our snapshots:
This doesn't make sense as part of our expect here and would sit more comfortably alongside '.disabled()' or '.tag()'.
In the below example, we can pass in a trait to force record all the snapshots that failed.
@Suite(
.recordSnapshots(.failed),
expect: .snapshots(sizes: .iPhone16, .iPadPro11)
)
struct EnhancementsWithTraitsTests {
@Test
func myView() -> some View {
Text("Some text")
}
}
Xcode/IDE UI
Expectations can be extracts by IDE UIs to make it clear to the user at a glance that a collection of tests expects a specific value.
There's also a potential to run all tests with a specific expectation type, simular to tags, to only run a subset of tests most likely to have broken.
Eg as a developer makes source code changes and quickly iterates, they can run only the tests their changes would likely have affected for a quicker turn around.
Alternatives considered
Arguments
The biggest question here is where arguments come in.
Adding the new proposed expect, it might seem like we're making arguments redundant.
Currently a test function takes data in from a parameter (arguments) and perfoms the assertion inside the test function with #expect.
In this proposal, a test function would return data and performs the assertion using the expect: parameter.
The main thing here is the proposed expect compliments the existing arguments parameter and can be used alongside it.
// Parameterised expect
@Suite
struct ArgumentTests {
@Test(
arguments: [
"One", "Two", "Three"
],
expect: .snapshots
)
func myView(input: String) -> some View {
Text(input)
}
}
As detailed above under the Motivation, currently parameterised testing doesn't support opaque types.
They can also lead to very difficult to read code which itself has to be refactored out in to fixtures and managed as its own code.
Make use of Trait/TestScoping
While there is potential for adding this under traits, I don't believe this fits the idea for what traits are under the Vision Document
We're specifically dealing with evaluation the product of a function here instead of any metadata, or the setup/teardown which to me feels like a separate and isolated part of the test.
Having said that, I did implement my spike using existing traits work (with a few extra bodges) in order to see what's possible at the moment from a technical point of view:
Breaking changes
This would be an entirely new additive feature so wouldn't break anything for existing written code.
)