ST-0008: Exit Tests

Evolution Proposal Reviews
1

Hello Swift community,

The review of ST-0008 "Exit Tests" begins now and runs through Tuesday April 8, 2025. The proposal is available here:

https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0008-exit-tests.md

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager. When emailing the review manager directly, please keep the proposal link at the top of the message.

Trying it out

To try this feature out, add a dependency to the main branch of swift-testing to your project:

...
dependencies: [
   .package(url: "https://github.com/swiftlang/swift-testing.git", branch: "main"),
],
...

and to your test target:

.testTarget(...
    ...,
    dependencies: [
       ...,
       .product(name: "Testing", package: "swift-testing")
    ]

Finally, import Swift Testing using @_spi(Experimental)
So, instead of import Testing, use @_spi(Experimental) import Testing instead.
(here is an example repo with that has all the setup)

What goes into a review?

The goal of the review process is to improve the proposal under review
through constructive criticism and, eventually, determine the direction of
Swift. When writing your review, here are some questions you might want to
answer in your review:

  • What is your evaluation of the proposal?
  • Is the problem being addressed significant enough to warrant a
    change to Swift?
  • Does this proposal fit well with the feel and direction of Swift?
  • If you have used other languages or libraries with a similar
    feature, how do you feel that this proposal compares to those?
  • How much effort did you put into your review? A glance, a quick
    reading, or an in-depth study?

More information about the Swift evolution process is available at

https://github.com/swiftlang/swift-evolution/blob/main/process.md

Thank you for contributing to Swift!

Maarten Engels

Review Manager

14 Likes
2

Small piece of feedback:

From an API-naming perspective, #expect(exit: .failure)—or #expect(exitWith: .failure)—reads more fluently (IMO) without loss of clarity as compared to #expect(exitsWith: .failure).

In particular, “expect” naturally takes a noun as its object, and “with” here (as is often the case for arguments) is implied and generally omissible as vacuous.

But otherwise, this is a hugely useful facility for testing. Excited.

1 Like
3

Thanks for the feedback! My thinking here is that there's an elided "this code" in the name of the macro, so that in full it would read "expect [this code] exits with failure". I hope that makes sense!

Edit: Or maybe it's more helpful to imagine "the following closure" instead, so "expect [the following closure] exits with failure"?

4

Sure. If this information (“this code” or “this closure”) is necessary for clarity, though, adding an “s” probably isn’t enough for that purpose?

However, it doesn’t seem particularly necessary to clarify that the expectation is of the closure—what else could it be?

5

I should also mention that this spelling is consistent with the existing #expect(throws:).

4 Likes
6

throws matches the keyword, and from that perspective, so would exit.

7

This is true, but the matching API is terrible and needs work.

Given

func noThrow() throws { }
enum SpecificError: Error { case `case` }
func `throw`() throws(SpecificError) { throw .case }

This is good in isolation:

#expect(throws: SpecificError.case, performing: `throw`)

Having to use SpecificError.case instead of .case is not good, but that's only tangentially relevant.

But the pattern is not good for Never or catch-alls:

#expect(throws: Never.self, performing: noThrow)
#expect(throws: (any Error).self, performing: `throw`)

What it should look like:

#expectNoError(performing: noThrow)
#expectError(performing: `throw`)
#expect(error: .case, performing: `throw`) // While it's not important to change this, `throws` was not a good choice. It doesn't create sentences like `error` does.

What exit tests should look like:

#expectNormalExit(performing: exitNormally)
#expectExitFailure(performing: exitAbnormally)
#expect(exitFailure: someExitFailure, performing: exitAbnormally)

The proposed ExitTest.Condition only exists to get around just having another overload spelled differently than another naked #expect. Please do continue with this approach anymore. It looks clever, but it's worse to read, and more to have to know about.

8

Hi @Danny

Thank you for your feedback.

Is your feedback about the API surface (i.e. using the new exit tests in your tests) or about the specific implementation of it as it currently exists?

As I collect the feedback, it's important for me to understand if feedback is about the API, the implementation or perhaps both.

9

+1 from my side. I also like the API form as it has been proposed.

The inability to test for assertions has been my main gripe with testing in Swift since coming from Objective-C a few years ago. Testing only happy paths has always felt incomplete and insufficient (what if the assertion is broken or maybe gone?). So I'm really happing to see this finally coming.

Two questions for clarity:

As I understand it, for now, there's really no way to pass any arguments or state to the closure? So we won't be able to use shared setup code or test arguments like so for now:

@Test(arguments: ["a", "b", "c"])
func exiting(_ arg: String) async {
    await #expect(exitsWith: .failure) {
         fatalError("I don't like \(arg)")
    }
}

To me it's not a problem that we cannot test for exits on iOS yet – better somewhere than nowhere! :wink: But how would we go about writing an exit test inside a universal test class? My understanding is that #expect(exitsWith:…) is just unavailable on unsupported platforms. So would this work and test for the exit on non-iOS platforms as expected?

@available(iOS, unavailable)
@Test func exciting() async {
    await #expect(exitsWith: .failure) {
         fatalError("only on macOS")
    }
}
10

Correct. This is obviously something we want to implement, and I know exactly how we'd do it (I dream in macro expansion code now… that's healthy, right? :sweat_smile:) but we likely need a new compiler or language feature to do it correctly. See the support for passing state section for more details.

Basically yes, but Swift itself will require you to spell it a little differently:

@available(iOS, unavailable) // technically optional at this point
@Test func exciting() async {
#if os(macOS) || os(Linux) || /* ... */
    await #expect(exitsWith: .failure) {
         fatalError("only on macOS")
    }
#endif
}

This is true of other kinds of unavailable symbol too, and isn't specific to exit tests, Swift Testing, or macros.

2 Likes