[Accepted] A New Direction for Testing in Swift

[mbrandonw (Brandon Williams)]

This is very exciting stuff! Thanks to all who worked on it.

There is an issue we brought up in the original proposal that was never addressed, and now that we have access to an Xcode beta that uses swift-testing we can see it is indeed a problem.

And so we wanted to bring it up again:

@stephencelis

Writing assertions in application/library code

It can be helpful to write test helpers for your application/library code, but one must tediously introduce a brand new MyLibraryTestSupport module to do so. This is because the act of importing XCTest into an app or library target leads to linker issues.

But it is also handy to write test assertions directly in application/library code: they can act as softer preconditions than fatal errors and immediately integrate with your test suite.

I just want to reiterate that being able to import Testing and execute Issue.record from app and library code can be incredibly powerful.

For example, if you like to control dependencies in your app (e.g. date initializers, API clients, location managers, etc.), then a powerful pattern to follow is to make the test version of those dependencies perform an XCTFail whenever it is accessed. This makes it so that while testing a feature, if the dependency is used it will cause a test failure. That forces you to override the dependency, and makes tests stronger by allowing you to prove which dependencies are and are not used. And catches you in the future if you start using a new dependency.

Now, one way to do this without using XCTFail in an app/library target is to move the test dependency to its own “test support” module that is only ever imported in tests. However, this is not actually practical in real world projects for a few reasons:

• First of all, it leads to a proliferation of “test support” modules. Nearly every module you create ends up needing a “test support” version, and so that begs the question: why can’t I just include my test helpers directly in the main module?

• Second, due to a confluence of bugs in Xcode and SPM, it’s not possible for a SPM package to ship a library and a test support library at the same time. This means you actually need to ship a whole separate SPM package (hence separate repo) if you want to provide these testing helpers.

• And third, sometimes it is just not possible to have a separate “test support” library. This is true when you want to build a library that requires one to integrate it in the app target, but also provides defaults that cause test failures.

  This is how our swift-dependencies library works. You can register a dependency with the library to use in your features’ logic, but if you access that dependency in a test without overriding it, you get a test failure. This XCTFail cannot be provided in some kind of “test support” library because it is an integral part of how the library works.

This problem was so pervasive that we created a dedicated SPM package just for dynamically loading XCTFail so that it could be used in app/library targets. And according to GitHub, it is getting over 13,000 unique clones per 2 weeks, and so there are a good number of people who are also finding this useful.

A “new direction for testing in Swift” seems like the perfect opportunity to move beyond this limitation of XCTest, and would also help prepare swift-testing for future requested functionality, like that of writing tests alongside application code in the same file.

So, is there any reason to continue to prevent application code from linking to the Testing framework?