A New Approach to Testing in Swift

Evolution Discussion
, , ,
80

Always very happy to see improvements to Swift’s testing! Thanks for driving this forward!

We maintain a number of Swift packages and many of them share a common theme: improving Swift’s testing story. Hopefully some of our learnings can be shared. I'll try to itemize each issue and how our libraries have been solving for them.

Assertion readability

XCTAssertEqual’s output is not very readable for larger data structures. Even smaller ones are difficult to scrutinize:

var other = user
other.name += "!"

XCTAssertEqual(user, other)

:x: XCTAssertEqual failed: ("User(favoriteNumbers: [42, 1729], id: 2, name: "Blob")") is not equal to ("User(favoriteNumbers: [42, 1729], id: 2, name: "Blob!")")

We try to offer an improvement in our swift-custom-dump library, which provides a custom “dump” function that is a customizable, pretty-printed version of Swift's dump, and it uses this function in custom XCTAssertNoDifference and XCTAssertDifference functions:

XCTAssertNoDifference(user, other)

:x: XCTAssertNoDifference failed: …

  User(
    favoriteNumbers: […],
    id: 2,
-   name: "Blob"
+   name: "Blob!"
  )

(First: −, Second: +)

(XCTAssertDifference also provides a nice way of writing assertions for how larger structures are modified, either exhaustively over the course of an operation, or non-exhaustively asserting against just a few given fields.)

We're not the only library to address this problem. Difference is another popular solution. And the recent swift-power-assert also tries to provide better failure messages.

Ideally, Swift would provide a good story here by default. SwiftSyntax's macro testing helpers are already providing a nicer experience in assertMacroExpansion, which will print a difference when the expected expansion doesn't match. I hope Apple will follow these examples more generally!

I took #expect for a spin and found that it does not currently address this problem.

Testing concurrency

Writing reliable async tests in Swift is notoriously difficult.

Our swift-concurrency-extras surfaces a global override to allow concurrent code to run serially, making it possible to write reliable async tests:

await withMainSerialExecutor {
  // Code in here runs serially
}

It's a hack, but it demonstrates a real deficiency in Swift testing that ought to be resolved.

Testing time-based asynchrony

Swift doesn't provide any out-of-the-box helpers for testing APIs that can sleep for a duration. So we provide swift-clocks (and combine-schedulers) with a number of helpers for testing time-based asynchrony, particularly a Test{Clock,Scheduler} that can manipulate the flow of time, an Immediate{Clock,Scheduler} that can sub into tests that don’t care about waiting for time to pass, and an Unimplemented{Clock,Scheduler} that is a stub that will emit test failures when code interacts with it (a good default value to help notify you if a test interacts with time-based asynchrony in a non-testable way).

While Combine is a private framework, Clock is a standard library protocol that is not super testable out of the box. While I’m not sure if swift-testing should be responsible for closing the gap, I do think Swift should ideally provide first-class helpers for writing tests that interact with the Clock protocol.

Snapshot testing

Snapshot testing has become more and more popular, but Swift doesn't offer any solutions out of the box. It's been mentioned a few times elsewhere in this thread, but we provide some libraries for it:

  • swift-snapshot-testing provides helpers for recording references and testing against them, either on disk or inline in the test case.
  • swift-macro-testing extends the inline snapshotting capability of swift-snapshot-testing for testing macros.

Writing assertions in application/library code

It can be helpful to write test helpers for your application/library code, but one must tediously introduce a brand new MyLibraryTestSupport module to do so. This is because the act of importing XCTest into an app or library target leads to linker issues.

But it is also handy to write test assertions directly in application/library code: they can act as softer preconditions than fatal errors and immediately integrate with your test suite.

To work around these limitations we built xctest-dynamic-overlay, which allows you to call XCTFail from application and library code by dynamically loading XCTest. And so it's possible to instrument application and library code with test failures that will be caught by your suite. We also log purple runtime warnings when XCTFail is invoked outside a test suite, making it easier for developers to catch issues while they run their code in a simulator or on device. While it may seem strange to want to emit test failures from application and library code, we’ve found many useful applications, and XCTest Dynamic Overlay is a dependency of many of our libraries. A few use cases:

  • swift-case-paths defines XCTUnwrap(case:) and XCTModify helpers for better testing enums and their associated values.

  • swift-composable-architecture uses it to provide a TestStore for testing features written with the library without introducing another module. While we originally shipped a separate ComposableArchitectureTestSupport module, it often led to downstream build issues in user land due to bugs in Apple's tooling. It's also less ergonomic to remind users to import ComposableArchitectureTestSupport.

    In addition to the TestStore, the library generally uses XCTFail to emit test failures and runtime warnings when we detect and communicate issues with how the library is being used to the developer.

  • swift-custom-dump uses it to define its XCTAssertNoDifference and XCTAssertDifference functions.

  • swift-dependencies invokes XCTFail when an unstubbed dependency is called from a test, among other failures.

These are just a few high level issues that come to mind, and we'd be happy to dive deeper into any of the above topics if folks have questions.

38 Likes