I've just merged a PR that adds an experimental feature to swift-testing enabling what we're calling "exit tests." You might know them as "death tests", "death assertions", or a number of other aliases: they're tests that verify a process terminates under certain conditions.
Using swift-testing, you can now specify such a test on macOS, Linux, and Windows:
There are a number of future enhancements possible to this feature such as support for passing Codable arguments or inspecting the contents of stdout. Again, this is an experimental feature, so not all possible functionality is implemented.
While we're not prepared to promote exit tests to a supported feature just yet, I have a draft API proposal here that goes into more detail about the design of the feature, how to use it, and constraints that apply.
This is great news! And it has long been a requested feature for testing in Swift, so I'm really excited about this (it will help me achieve 100% code coverage in Sargon ).
Question: I did not find any example that continues to execute the test after an #expect(exitsWith:), is that supported? e.g.
@Test func fatalErrorWorks() async {
await #expect(exitsWith: .failure) {
fatalError("🧨 Kablooey!")
}
// Can I continue the test case here?
#expect(1 == 1) // continue with important expectations.
}
I guess that works, right? Since neither expect nor require returns Never and since the exitsWith spawns a new process (which might exit..), but that should not exit the 'parent' test process, right?
Yes, your example should work as intended. The closure passed to #expect(exitsWith:) is the only part of the test that executes in another process, and the parent process is not terminated as part of exit testing (if it were, that'd kind of defeat the purpose of spawning another process.)
Another question:
(I did not see it under "Future directions") Any plans to support "simulating" if tests are running in DEBUG mode or not: sometimes we wanna run unit tests with optimisations flag, where DEBUG is not set, but we might wanna be able to unit tests that the DEBUG only "exits" are triggered - all assert/assertionFailure ones. So would be great if we could allow #expect(exitsWith behave for both DEBUG and not.
Debug mode is a completely different compilation context and, to simulate it, would require recompiling your binary. If you want to run your tests in debug and release modes, and you'd need to compile and run them twice in order to do so, just call swift test and pass --configuration as appropriate. I hope that makes sense!
Right, I naively thought it would be somehow possible for swift-testing to affect the rules under which assertionFailure is evaluated or not, but on second thought that was a stretch and not something a package can do, how silly of me.
@Test
func test_frobnicate() {
#if DEBUG
#expect(exitsWith: .failure) { frobnicate(42, 3) } // works for DEBUG only
#endif
#expect(exitsWith: .failure) { frobnicate(5, 0) } // works for DEBUG and RELEASE
}
That we exhaustivly test frobnicate as much as possible when test run both under DEBUG and under RELEASE configurations.
I wonder what llvm-cov tool will think of the assert(a != 42) line when run under release config? Might regard it as an untested line, preventing me from 100% coverage... because AFAIK unfortunately there is no way to ignore a code line of code block.
I guess code coverate is too low level an area for swift-testing to work with, as in, requires low level tools like llvm-cov?
As far as I'm aware (and I may be wrong here!), llvm-cov is able to combine code coverage statistics across multiple processes. While Swift Package Manager does not currently have the ability to run your code under multiple configurations in one go (and thus doesn't know how to tell llvm-cov to combine statistics from both runs), that's conceivably something that could be added in the future.
I would suggest opening a GitHub issue against Swift Package Manager.
I know I'm a little late, but curious if the exit could be a little more specialized to Swift specifically. For example, catching that it failed due to various actual Swift failures:
fatalError vs precondition vs assert
concurrency checks (like dispatchPrecondition or assumeIsolated)
failed optional-unwrap
Basically, it'd be nice to test that it fails correctly and not just crashing for some random reason -- and be able to hook into very specific kinds of Swift runtime failures.
That's a future direction for exit tests (which, to be clear, are still an experimental feature.) As of right now, all those failure modes are programmatically indistinguishable—they all call through to swift::fatalErrorv() which may or may not log something, then calls abort().
We'd need to refactor those functions in the stdlib to pass some "who am I?" value down and provide a hook in the runtime to extract that value before it's called. That's technically feasible, but beyond the scope of the feature right now.
I can update the draft pitch to include this as a a future direction.
I’m not sure I agree with the premise here. Wanting to know that something failed for a specific reason is good, but trying to distinguish “optional unwrap” from precondition from fatalError from “concurrency violation” seems like it is neither extensible nor really the thing you want to test. I’d rather see “and stderr matches this regex” or “and the stack trace includes this location [if debug info is available]”. (The latter is kind of tricky to implement.) It’s not perfect, but it’s way more flexible, and well-precedented in other testing frameworks.
EDIT: …and the stderr capturing is included in the proposal, albeit in a form I might want to write a helper around for the common case.
The ability to distinguish specific language-level fatal errors would be a future direction which (as you know) does not guarantee future implementation. I would expect test authors to use #expect(exitsWith: .failure) currently. But it's worth documenting in the proposal either way.
As for your wrapper function: I recently proposed changing how we gather error information from #expect(throws:) to make it more ergonomic following developer feedback, and my instincts here are to keep the syntax similar. This proposal is still a draft (and that one isn't accepted yet), so if you've got some ideas how we can improve the syntax, I'm all ears.
It really would just be #expect(exitsWith: .failure, outputChannel: .stderr, matching: /error: too small/) { … }. I think returning a set of captured channels makes sense for the most generic interface, but the common case is just checking one thing from one channel. This is different from errors because (a) an error can carry structured information, and (b) pattern-matching an error case can’t be done purely with values today (without the CasePaths library, anyway).
I think we agree on the fundamentals here and it's just a question of spelling. One thing to keep in mind is that the standard streams often contain cruft, sometimes unpredictable, from library components outside the test author's control, so regexes may need to account for that.
We've also seen our fair share of processes that emit non-text output to either stream (and there are plenty of tools in the POSIXverse that do this intentionally!) which is why the streams in the proposal and experimental implementation are stored as UInt8 buffers, not as strings. So even if we provide shorthand regex-based interfaces, we do need our interfaces to work when String can't be used.
I wonder if helper functions on the output type would be a fair compromise? Something like:
extension ExitTestArtifacts.StreamContent {
/// Fails if not decodable as UTF-8.
func contains(_ regex: Regex) -> Bool
}
let output = try await #require(exitsWith: .failure) { ... }
#expect(output.standardOutputContent.contains(/error: too small/))
Ideally it wouldn’t even fail on non-UTF-8, it would skip over it (equivalent to “insert replacement character, but hopefully not actually reallocating the string). Don’t know if that’s possible with the APIs the stdlib provides today, though.
Since the proposal made a point of having stream capture be optional, I think I’d end up defining a helper anyway. There’s not really a better way to test fatalError, but at the same time you don’t need anything more than that. But not all helpers have to be in the library!
Distinguishing arbitrary binary data from valid UTF-8 in a single stream is possible, but non-trivial to the point that it probably falls beyond the scope of Swift Testing and into Standard Library Land (or maybe Foundation or a dedicated package.) It's also not guaranteed that text output is encoded as UTF-8, although it's very common these days obviously.
I'm still actively maintaining this branch/PR. We are not planning to formally propose this feature until the testing workgroup is set up, at the earliest.
).
because AFAIK unfortunately there is no way to ignore a code line of code block.