One of the first enhancement requests we received for Swift Testing was the ability to test for precondition failures and other critical failures that terminate the current process when they occur. This feature is also frequently requested for XCTest. With Swift Testing, we have the opportunity to build such a feature in an ergonomic way.
Read the full pitch here.
Edit: I have moved the proposal document to a pull request against Swift Evolution here.
My body tests are ready! Big +1 from me.
Super excited for this! We can definitely use this in Foundation. A few questions after my first read-through:
First, can other expectations be validated during an exit test (for example if I wanted to validated via an #expect or #require that something happened in the separate process as part of a setup step before triggering the crash)?
Additionally, is there room for a convenience that allows explicitly testing precondition/assert failures? Rather than needing to configure stdout yourself and write the correct signal/exit code in your test, I wonder if we could enable writing:
await #expect(exitsWith: .preconditionFailure("Invalid input")) {
// do something
}
which would validate that A) the exit code is the proper exit code that would be returned by precondition and B) that the precondition failure message (usually in stdout or the crashlog) is the expected failure message.
Yup, in general issue recording "just works".
This is something we did look at doing, but decided against (at least in the initial feature implementation) because Swift itself doesn't really define the effects of a precondition failure beyond "abnormal process termination". As implemented, swift::fatalError() calls abort() indiscriminately, which means there's no practical way to distinguish fatalError() from preconditionFailure() without scraping stderr.
There was some discussion on this topic in an earlier thread.
Gotcha ok thanks for the explanation. Yeah looking at that thread I see some comments around likely wanting a helper function that can easily pattern match against the stderr output which is probably what I'd reach for since precondition vs. fatalError doesn't matter as much to me as whether or not I hit the right failure, however it was produced.
This specific interface is beyond the scope of this proposal but is easy enough to write something that does 90% of the work:
let result = try await #require(
exitsWith: .failure,
observing: [\.standardErrorContent]
) {
preconditionFailure("splat")
}
#expect(result.standardErrorContent.contains("splat".utf8))
This looks outstanding! 
I love that the design is intentionally more broad than something like expectCrashLater – being able to evaluate independent chunks of code that terminate (possibly successfully!) will be a great advantage for certain kinds of libraries.
It would be great to know more about the context in which the tested closure is executed. I see that the documentation mentions that variables can't be captured in the closure, but of course any code called from within it can have mutating effects on global or shared state. What parts, if any, of the shared state from the testing module/file are inherited by the test?
I also have a couple notes regarding ExitCondition:
struct instead of an enum. You can still have static functions for nice usage in the #expect(exitsWith:) call site, and then provide var exitCode: Int? and var signal: CInt? properties that people can query, without the cumbersomeness of an enum.Equatable behavior seems like it needs justification, as it will be confusing for users. Is there something that can't be handled other ways, like by providing an isFailure Boolean?In the Swift-side documentation, we wrote:
Meanwhile, in the child process,
expressionis called directly. To ensure a clean environment for execution, it is not called within the context of the original test.
i.e. it's a new process and there is no state shared with the parent process (ignoring platform-specific minutiae like open file descriptors—thanks, POSIX!) If that's not sufficiently clear, we can revise the documentation to make it more explicit. I'm open to ideas for how we can best phrase it.
ExitCondition is effectively a Swift representation of process termination states as used by UNIX-like systems for decades. If we gain the ability to distinguish new and interesting states such as "exited due to preconditionFailure()", that can be represented via a different type and an overload of the macro.
We went back and forth about this part of the API surface. The logic goes:
It should be possible to write exitsWith: .success and exitsWith: .failure and not have to worry about the precise exit code. .success is defined, rather concretely, as "process exited with exit code EXIT_SUCCESS", however failure is any other value (ignoring hypothetical future directions.) So naïvely, yes, isFailure would capture that distinction.
But then, how do you compare two values? The ergonomic way to compare two values in Swift is with ==, not with some function such as isEqual(). If we allow == to be used to compare two values, this naïve code is bound to crop up:
#expect(exitCondition == .failure)
Which, reading it, seems perfectly sensible, but it'll fail if == is transitive because .exitCode(123) == .failure, .exitCode(456) == .failure, but .exitCode(123) != .exitCode(456). The likely intent here would be that the expectation pass, because .exitCode(123) is a failure condition.
== does behave exactly as you'd expect when passed two exit codes or two signals. Of course, sometimes you really do want to compare two exit conditions exactly and not special-case .failure, in which case === exists (by analogy to ===(lhs: AnyObject, rhs: AnyObject)).
My expectation (no pun intended) is that the lack of conformance to Equatable and this edge case where the operator is not transitive will not have a practical impact on developers. (But I can be convinced otherwise!)
Thanks so much – that added context is so helpful for understanding the design. What I see from that is that a user has two requirements:
I need to provide the exit condition that I am expecting, with the ability to be alternately general (e.g. .success/.failure) or specific (e.g. .exitCode(42)).
I need to be able to examine the result of the expectation, to determine specifically how the execution ended, for various reasons.
I don't think there's a need for those two requirements to be handled by the same type. At the #expect(exitsWith:) call site, the general .failure case is probably the most common thing to be used by far, but if I understand correctly, you never have a general failure after the expectation – the result is always something specific.
So a purely outward-only ExitRequirement could be passed into expect(exitsWith:), while ExitCondition would live only on ExitTestArtifacts, and could both provide that .isFailure property and be Equatable without compromises.
What if I have something like:
// Taco.swift
struct Taco {
static var cilantroIsDelicious = true
var hasCilantro = true
var isDelicious: Bool {
hasCilantro == Self.cilantroIsDelicious
}
// etc
}
// TacoTests.swift
import TacoTruck
Taco.cilantroIsDelicious = false
@Test
func eatATaco() {
await #expect(exitsWith: .failure) {
var taco = Taco()
eat(taco) // what happens here?
}
}
What if there's a compilation condition that sets cilantroIsDelicious, etc? It totally makes sense that the code inside an expectation might not be able to have the same behavior / semantics as code immediately outside it, but we should try to really cover the cases where that will happen.
So how about:
/// A type representing the various ways an exit test should exit.
public struct ExitCondition {
/// A type representing the actual/raw status reported by the system when the child terminated.
public enum Status: Equatable {
case exitCode(CInt)
case signal(CInt)
/// Standard equality operator, equivalent to current proposal's `===`
public static func ==(lhs: Self, rhs: Self) -> Bool
}
/// Different kinds of exit condition.
internal enum Kind {
case specificStatus(Status)
case anyFailure
}
internal var kind: Kind
/// Conveniences to get requirements matching specific exit conditions.
public static func exitCode(_: CInt) -> Self
public static func signal(_: CInt) -> Self
/// General-case conveniences.
public static var success: Self { .exitCode(EXIT_SUCCESS) }
public static var failure: Self { get }
/// Compare any two such values, equivalent to current proposal's `==`
/// Needed internally, but doesn't need to be exposed.
internal func approximatelyEqual(to other: Self) -> Bool
}
struct ExitTestArtifacts {
// ...
/// The actual exit condition reported for the child by the system.
public var status: ExitCondition.Status
}
Then you can compare two exit conditions directly, and internally we'll have mechanisms to compare .failure against exit conditions, but you can't accidentally write == with the wrong semantics.
(Naming is hard—happy to bikeshed the type names here.)
I don't love that this duplicates the .exitCode and .signal symbols, but that's probably a minor gripe.
The child process is a "clean slate" (as much as is possible) and is not going to inherit any such state, nor is it technically feasible to inherit it (Swift Testing doesn't even know cilantroIsDelicious exists, let alone that it might need to be preserved.)
I see this as a "just how the universe works"—type problem. We can't overcome it, but we can do our best to document how things work.
That design looks great! I would add an isFailure or isSuccess to ExitCondition.Status, but otherwise it's right on.
That all makes sense — maybe we can explain it as something like, "the closure is executed as if it's top-level code, along with the modules imported in this file" (if that's actually accurate). Are the compilation flags passed when running tests preserved for the exit test? That is, can tests expect that a directive like this will build the same way for the exit test as for the outer context?
// Taco.swift
struct Taco {
static var cilantroIsDelicious: Bool {
#if DELICIOUS_CILANTRO
true
#else
false
#endif
}
// etc
}
Edit to add: As a follow-up, maybe a future direction to be able to pass alternative/override compiler flags to #expect(exitsWith:).
I thought I had something to the effect of "an exit test is executed as if it were the main function", but it seems like I have all the words, arranged in the wrong order:
Meanwhile, in the child process,
expressionis called directly. To ensure a clean environment for execution, it is not called within the context of the original test. Ifexpressiondoes not terminate the child process, the process is terminated automatically as if the main function of the child process were allowed to return naturally. If an error is thrown fromexpression, it is handed as if the error were thrown frommain()and the process is terminated.
I can work on rephrasing that blob to be more explicit about the exit test body taking the approximate role of main().
Exit tests are compiled at the same time as the rest of the module, so yes, compiler conditions will be respected. (Also, cilantro is not delicious.)
Exit tests do not have this sort of control over their own compilation and do not dynamically recompile themselves or produce separate binaries or anything along those lines. We have the ability to rewrite the closure body at macro expansion time, but we don't have the ability to affect any code called by the closure, so saying "set/unset DELICIOUS_CILANTRO" could only affect the literal AST inside the macro, which is unlikely to be useful.
I've taken @nnnnnnnn's input and have opened a PR to adjust the shape of ExitCondition here. If/when that lands, I'll update the proposal document to reflect the changes.
The proposal makes several references to String.init(cString:), e.g. here:
public struct ExitTestArtifacts: Sendable {
/// All bytes written to the standard output stream of the exit test before
/// it exited.
///
/// The value of this property may contain any arbitrary sequence of bytes,
/// including sequences that are not valid UTF-8 and cannot be decoded by
/// [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s).
/// Consider using [`String.init(validatingCString:)`](https://developer.apple.com/documentation/swift/string/init(validatingcstring:)-992vo)
/// instead.
///
/// […]
public var standardOutputContent: [UInt8] { get set }
…
}
This seems wrong/misleading to me because init(cString:) expects a null-terminated input. Is the standardOutputContent actually (expected to be) null-terminated? I'd guess no.
I would point people to String.init(decoding:as:), which deals with invalid bytes sequences by replacing them with U+FFFD (sadly, this is not documented). Or String.init?(validating:as:) if you want the decoding to be failable.
(Also, the concrete overload of init(cString:) the proposal links to is the one taking an UnsafePointer, which isn't appropriate if your input is an Array<UInt8>. There is another overload that takes an array, but that one's deprecated and rightfully directs people to init(decoding:as:).)
I'll update the comments. I wrote them before some of the relevant API was added. They're specifically warning developers not to use API that assumes valid UTF-8, so init(validating:as:) is ironically the wrong thing to mention there.
Unless you want to make sure that the returned data really is valid UTF-8, right?
The point of the given prose is that you should not assume the arrays represent UTF-8 (since anything can write to them including code you don't control), so you should be careful when using API that does assume. 
Finally I’ll be able to achieve 100% code coverage for my unit tests! Huge +1 from me!
First off, obviously not a surprise, +1 from me as well (just for those who keep track of the count).
The API design looks slick as well.
About Equatable conformance. What if you renamed the .failure case to .anyFailure? Then the behaviour of == could be described as:
ExitCondition.signal(SIGINT) == .anyFailure // true, because .anyFailure implies SIGINT
ExitCondition.signal(SIGINT) == .signal(SIGABORT) // false, because we express intend to compare two signals
would make sense.
The basic API case:
await #expect(exitsWith: .anyFailure) {
also reads well.
And we wouldn't need to introduce the non-standard === operator.
The type could still not satisfy the informal requirements of Equatable, because .signal(SIGINT) == .anyFailure and .signal(SIGABORT) == .anyFailure but .signal(SIGINT) != .signal(SIGABORT).
Take a look at the PR I've opened to update this part of the interface: it should address the concerns Nate raised. 