[Accepted] A New Direction for Testing in Swift

I’m excited about the support for testing fatalError().

Something I haven’t seen mentioned yet, is whether there are any plans to help folks migrate a codebase away from XCTest? Or is that out of scope?

So the efficiency of using macros is a distribution / integration problem, and this will be (or already is?) resolved, right? (I did not really understand from the link what the current situation is and how it will be resolved in the long term.) So anyway, there seems to be no “big“ inherent efficiency problem with macros. Good to know, thanks.

Keep in mind exit tests are an experimental feature and future direction, not a feature we're planning to deploy with the Swift 6 toolchain or Xcode 16.

We have not implemented any sort of automatic translation from XCTest to Swift Testing.

As of late, you can use swift test --quiet and it will suppress most output from Swift Testing. If you're using Xcode 16, check out the Test Report.

XCTest and Swift Testing should be equally capable when it comes to XML output. We're also working on a stable JSON-based event stream API/ABI that can be used with external tools.

I'm not entirely sure what you're looking for here. Mind spinning up a separate thread in the forums for us to discuss further?

3 Likes

Unfortunately this is not true, as far as I can tell.

The distribution/integration problem of macros and SwiftSyntax has not been solved for 3rd parties. Apple's frameworks get the benefit of being bundled directly into the toolchain so that users of their frameworks do not feel the pain of needing to compile SwiftSyntax. It's why we've been able to use @Observable for the past year without incurring 30+ second DEBUG build times, or many minutes of build times in RELEASE.

The document linked above is calling out that ideally the Testing framework would not need to be bundled with the toolchain, and maybe in the future will not be:

Since this project relies on Swift Macros, it depends on swift-syntax and that is a large project known to have lengthy build times as of this writing.

Due to these practical concerns, this project will be distributed as part of the Swift toolchain, at least initially. [...] Longer-term, if the practical concerns described above are resolved, the library could be removed from the toolchain, and doing so could yield other benefits such as more explicit dependency tracking and more portable toolchains.

It would be great to get an update from the core Swift team about their plans to make macros more viable for the ecosystem. Perhaps in a new forum thread?

8 Likes

Separately from the distribution overhead problem, there's also the problem that there's meaningful per-macro-expansion overhead. It's not an issue for 100 @Observables, but it's pretty concerning for 100k #expects.

2 Likes

would it ever become possible to compile swift-testing (including its macros) from source without also compiling swift-syntax from source? i’m not sure if i’m up to speed on why both frameworks are joining the toolchain, as opposed to just swift-syntax.

1 Like

Since this vision document was only recently accepted, Swift Testing is not yet part of Swift 6.0 toolchains, but we plan to propose adding it there soon, along with some accompanying changes to SwiftPM to enable using it. It is included in Xcode 16 Beta, however, so if you are trying that out, you will not need to declare a dependency on the swift-testing package and can freely import Testing.

I think we'll need someone more familiar with the Swift macros infrastructure, SwiftPM, and/or swift-syntax to comment on this topic more authoritatively, possibly in a new thread, since it's a combination of factors involving those components. But one consideration is that the copy of swift-syntax included in the toolchain is coupled to the version of the compiler in that toolchain, so an arbitrary version of a package may not be compatible with the version of swift-syntax in the toolchain.

Another consideration is that when a package with macros is built by SwiftPM, its macro implementation is built for debug rather than release, and it may incur more overhead than macros built and included in the toolchain. As @tgoyne mentioned above, there can potentially be many macro usages such as #expect in Swift Testing tests, so we want to maximize the performance of these macros.

4 Likes

is this copy of swift-syntax internal to the toolchain? meaning, will client code be able to import SwiftSyntax the way it will be able to import Testing?

Something I haven't seen talked about much is the possibility of having Tests in the same files as source code.

I'm not personally a big fan of it, but some really like it and I can see use cases for it, especially in a very small codebase.

e.g.

extension Date {
    func isWithinFiveMinutes(of otherDate: Date) -> Bool { /* Implementation */ }
}

import Testing

@Test func dateWithinFiveMinutesMatches() {
    let startDate = Date()
    #expect(startDate. isWithinFiveMinutes(of: startDate)
    #expect(startDate. isWithinFiveMinutes(of: startDate.advanced(by: 300)
    #expect(startDate. isWithinFiveMinutes(of: startDate.advanced(by: -300)
}

@Test func dateOutsideFiveMinutesFails() {
    let startDate = Date()
    #expect(startDate. !isWithinFiveMinutes(of: startDate.advanced(by: 301)
    #expect(startDate. !isWithinFiveMinutes(of: startDate.advanced(by: -301)
}

Also more generally if Tests were allowed in the same target and files, users could organizing their code in whichever way they choose. This would especially make Packages more approachable and simpler to set up / use.

I'm curious if this would be considered for a current or future direction, or whether the sentimant in the core team and/or community is against the idea generally

2 Likes

I'm afraid we don't yet have a good technical solution for the general problem of not needing to compile swift syntax from source in order to compile macros outside of the toolchain. It is well understood though that this is an important feature for the community and one where solutions are being actively investigated, but there aren't any updates to share as of right now. Starting a new thread on the forums won't accelerate this process.

18 Likes

It seems to be standard practice to have SwiftUI previews in the same file as the view, which is almost the same thing. What I'm looking for is being able to run UI tests against those previews, and I'm kind of surprised we don't have that yet.

4 Likes

Very excited about the new direction! The new #expect macro in particular is a huge improvement over XCTAssert and friends.

I have a question about the #expect macro and interop with XCTest (lmk if there's a better place to ask this sort of question):

Would it be possible for the #expect macro to integrate with XCTest such that it can be a drop-in replacement for XCTAssert calls in existing XCTestCase test implementations? For example:

import Testing
import XCTest

// An existing XCTest implementation:
final class MyExistingXCTestCase: XCTestCase {

  func testMyExistingCode() {
    // ...
    let someBoolean = ...
    #expect(someBoolean) // instead of XCTAssert(someBoolean)
  }

}

It would be really nice to be able to take advantage of #expect immediately in existing tests without having to migrate the whole test structure itself. This would help reduce context-switching in codebases with a mix of XCTestCase tests and Swift Testing tests.

The swift-power-assert library provides this sort of functionality: its #assert macro integrates with XCTest as a drop-in replacement for XCTAssert in XCTestCase implementations. However, having a direct XCTAssert replacement in the toolchain would be a big improvement over needing to pull in a separate package just for XCTestCase assertions.

2 Likes

This is very exciting stuff! Thanks to all who worked on it.

There is an issue we brought up in the original proposal that was never addressed, and now that we have access to an Xcode beta that uses swift-testing we can see it is indeed a problem.

And so we wanted to bring it up again:

I just want to reiterate that being able to import Testing and execute Issue.record from app and library code can be incredibly powerful.

For example, if you like to control dependencies in your app (e.g. date initializers, API clients, location managers, etc.), then a powerful pattern to follow is to make the test version of those dependencies perform an XCTFail whenever it is accessed. This makes it so that while testing a feature, if the dependency is used it will cause a test failure. That forces you to override the dependency, and makes tests stronger by allowing you to prove which dependencies are and are not used. And catches you in the future if you start using a new dependency.

Now, one way to do this without using XCTFail in an app/library target is to move the test dependency to its own “test support” module that is only ever imported in tests. However, this is not actually practical in real world projects for a few reasons:

  • First of all, it leads to a proliferation of “test support” modules. Nearly every module you create ends up needing a “test support” version, and so that begs the question: why can’t I just include my test helpers directly in the main module?

  • Second, due to a confluence of bugs in Xcode and SPM, it’s not possible for a SPM package to ship a library and a test support library at the same time. This means you actually need to ship a whole separate SPM package (hence separate repo) if you want to provide these testing helpers.

  • And third, sometimes it is just not possible to have a separate “test support” library. This is true when you want to build a library that requires one to integrate it in the app target, but also provides defaults that cause test failures.

    This is how our swift-dependencies library works. You can register a dependency with the library to use in your features’ logic, but if you access that dependency in a test without overriding it, you get a test failure. This XCTFail cannot be provided in some kind of “test support” library because it is an integral part of how the library works.

This problem was so pervasive that we created a dedicated SPM package just for dynamically loading XCTFail so that it could be used in app/library targets. And according to GitHub, it is getting over 13,000 unique clones per 2 weeks, and so there are a good number of people who are also finding this useful.

A “new direction for testing in Swift” seems like the perfect opportunity to move beyond this limitation of XCTest, and would also help prepare swift-testing for future requested functionality, like that of writing tests alongside application code in the same file.

So, is there any reason to continue to prevent application code from linking to the Testing framework?

20 Likes

I think the lack of automated test doubles (fakes, stubs, spies) generation is a huge gap in swift testing.
We have amazing tools which can be now currently used: Sourcery and Macros.

I do not agree that using test doubles is last resort. While using DI it is important to know, if given injected object called specific method and how many times.

Also, by using automated stubs we could achieve convenient factory methods which could help to focus on the specific context, without creating tons of boilerplate code.

Can I suggest, instead of folks describing things as "huge gaps", everyone considers that this is just an initial release (pre-release, even) of a framework that has great potential for future directions (potential that the whole community will have an opportunity to shape through the steps outlined in this vision document).

It's a small thing, but not framing things unnecessarily negatively helps keep the Swift project a friendly and enjoyable place for everyone to contribute.

47 Likes

I would like to apologize all who worked on swift-testing and feels offended by my last post.
My feedback may sound a little blatant, or too critique, but it wasn't my intention.

I am truly grateful that swift team noticed a missing gap in swift language and focused on quite a big (and challanging) feature, allowing developers make their code more reliable. This is a really big step forward.

What I would like to point out are my worries about one opinion here from one of key developer:

Just speaking for myself, I consider mocking to be a last-resort method of testing.

which makes me think if autogenerated test doubles have even a chance to arrive in the future.

I agree, that using mocks from all test doubles suite is a last resort. Mocks contain specific logic, may manage a state or could be even an actual implementation with some methods overriden. However, there are other test doubles, like stubs or spies which are more suitable in most cases.

With SwiftUI you guys have given an amazing opportunity for developers to not stick with one architecture but leaving a freedom of choice, yet making things simple.

That would be really nice to see similar approach as future steps in swift testing. I saw many developers who are tentative to write unit tests due to writing a lot of boilerplate code. Autogenerating test doubles would not only convince that developers to write tests but also encourage newcomers to start their first steps in testing.

With XCTest it's possible to do async / throwing tearDown. Since in Swift Testing this logic is now in init / deinit I can't seem to figure out how to have async / throwing teardown logic (deinits can't be async) - what's the best way to do this with Swift Testing?

2 Likes

To be clear, I work on the Swift core language, not the swift-testing project. My involvement with this was as one person among many reviewing the principles in the vision document. I don’t expect to serve on the Testing Workgroup, and I don’t think my personal opinions about testing should carry any special weight here.

Regardless, my opinion is not that there shouldn’t be any support for mocking in the framework. It’s just that I have seen a lot of code that’s been made strikingly more complex, harder to understand, and bug-prone in an attempt to test it, and that usually arises from an overly-narrow understanding of how testing should be done. I have seen projects that attempt to abstract over their entire in-memory data model, because in the developer’s mind, the way you test a subsystem is to replace it with a mock and log all the calls to it. So I will confess to having a somewhat reflexive reaction to mocking at this point, because I do think it is wildly over-used. But there are appropriate uses of it, and it may be that there are ways for the framework to usefully support it.

17 Likes

I didn't mention mocking in the vision document, and perhaps I should have because I agree it can be useful in some testing scenarios and it's a topic many people mentioned in our initial forum post about swift-testing. I also feel it's a technique which can sometimes be overused and lead to a false sense of security or other problems if poorly applied. Nevertheless, in my opinion there should eventually be some better approach for mocking that feels idiomatic in Swift, whether it be via something built-in to the language/runtime, better tools or macros for generating mocking code, or some combination.

So far, those of us contributing to swift-testing have focused on what we consider the fundamentals of a modern testing library: declaring tests, validating expected conditions, enabling common behaviors (mostly via built-in traits), and integrating with tools. Over time, we hope to add more advanced features and functionality, some of which the vision document covers. I think mocking could potentially be on that roadmap too, but unlike the other topics the vision mentions, there isn't yet consensus about what the ideal solution for mocking in Swift should be, and it's a complicated problem to solve well. That's why I didn't mention mocking in the document: not that we don't think it's important, but that we don't have anything specific to propose on the topic right now.

9 Likes

This is an area we have intentionally not built out yet. We don't want to end up reinventing existing language features just for testing if we can avoid it. Our hope is to leverage async deinit and async throws defer for this sort of functionality (with the understanding that neither feature is actually part of the language yet.)

6 Likes