Testing assertions is outstandingly difficult in Swift today. A very informal poll suggests that some other people think so, too. There was a thread on the matter back in 2015, but it didn't get much traction. I would like to start a discussion on:
- What's the guidance on how to do it today?
- What can we do to improve this area?
For the sake of easier communication, let's take the DelayedImmutable property wrapper example (from SE-0258) as an example:
@propertyWrapper
struct DelayedImmutable<Value> {
private var _value: Value? = nil
var wrappedValue: Value {
get {
guard let value = _value else {
fatalError("property accessed before being initialized")
}
return value
}
// Perform an initialization, trapping if the
// value is already initialized.
set {
if _value != nil {
fatalError("property initialized twice")
}
_value = newValue
}
}
}
In this thread, by "assertion", I'm referring to any of assert, assertionFailure, precondition, preconditionFailure, fatalError, etc., such as those fatalErrors which prevent reading before intiailization and double initialization.
(Non)-Idea -1: Don't write assertions
I've seen several suggestions only document preconditions in API documentation, but not bother to verify them at runtime. The thinking goes that if a user calls your APIs in a way that doesn't meet your contractually-stipulated prerequisite, an "all bets are off" mode applies. Slap an "undefined behaviour" label on the problem, leave the callers to pick up the pieces, and call it a day.
I think this is a real "you're holding it wrong" vibe. It might fit in C, where the priority to to squeeze out every last cycle from the hardware, but it certainly doesn't fit the Swift ethos, which highly values safety and developer experience.
There's clearly a benefit to testing pre/post-conditions, look no further than the Swift standard library itself, which tests preconditions extensively (example). Some languages like Ada take its importance even further, and go out of their way to make precondition and postcondition contracts an explicit syntax in their language.
In the DelayedImmutable example, I'm not even sure if it's possible to remove the fatalError. The get must return a Value, or call a Never-returning function like fatalError. What Value could we return, if we don't have one? The only alternative I see to try to return some undefined garbage memory with unsafeBitCast.
(Non-)Idea 0: Use assertions, but don't bother testing them
Q: Do I have to test ALL my code?
A: Of course not, only the parts you want to work correctly.
– Uncle Bob or something, I don't remember :')
Idea 1: Use throw instead
This would make catching possible, thus testing really easy. Superficially, this appears to match how it's done in Java, Python and the like. The big difference is that those languages have exceptions, where raising an exception doesn't require changes to every caller up the call stack.
In practice, Swift's error-throwing does not at all work as an assertion mechanism:
-
throwsis exposed at the type system level (and it needs to be, because the caller needs to be an active participant in the error-handling process). This means that if you have some old non-throwsfunction, you can't just add a newif !precondition { throw PreconditionFailed() }. You'd need to mark itthrowswhich:- In the best case, you control all the callers and can just put
try!before all of them. - In the worst case, you're writing a public API of a library, and you simply can't make this change without breaking your API.
Compare this with Java, where checked exceptions are the norm, but unchecked exceptions are still possible. Any callee function can choose to raise an error, without any participation needed from its caller. (At the expense of a larger binary, and a much slower error path, which needs to do stack unwinding. I'm not arguing exception-throwing is better than Swift error-throwing.)
- In the best case, you control all the callers and can just put
-
There are many places that you simply can't mark
throws, even if you wanted:-
Property getters(Fixed in Swift 5.5 via SE-0310 Effectful Read-only Properties
)DelayedImmutable working example
@propertyWrapper struct DelayedImmutable<Value> { private var _value: Value? = nil struct NotYetInitialized: Error {} var wrappedValue: Value { get throws { // âś… Allowed to be marked `throws` guard let value = _value else { // fatalError("property accessed before being initialized") throw NotYetInitialized () // âś… } return value } // Perform an initialization, trapping if the // value is already initialized. set { if _value != nil { fatalError("property initialized twice") } _value = newValue } } } -
Property setters
DelayedImmutable comiler error
@propertyWrapper struct DelayedImmutable<Value> { private var _value: Value? = nil struct InitializedTwice: Error {} var wrappedValue: Value { get { guard let value = _value else { fatalError("property accessed before being initialized") } return value } // Perform an initialization, throwing InitializedTwice if the // value is already initialized. set throws { // ❌ 'set' accessor cannot have specifier 'throws' if _value != nil { throw InitializedTwice() } _value = newValue } } } -
@convention(c)functions (meaning you wouldn't be able to use assertions in callbacks from C code, which are arguably one of the places where careful validation is most necessary!) -
subscriptsetters (e.g. you can't assert that a given index is in range!) -
Any members which satisfy a protocol that doesn't allow it.
- E.g. you could never put an assertion in the
description: Stringproperty ofCustomStringConvertible, because that protocol declares that thedescriptionproperty is non-throwing.
- E.g. you could never put an assertion in the
-
-
On a more cosmetic, less show-stopping level, you now need
throwsandtryabsolutely everywhere in your program. If the Standard Library had taken this appraoch, everyarray[index]operation would need atryin front of it.
Idea 2: Run tests in a separate process, and detect crashes
As far as I can understand it, I think this is how the Standard Library's expectCrashLater works.
- Tests are run in a new process
- If a test called
expectCrashLaterto assert that a crash should happen, it prints a message to the stdout/stderr - The test runner detects that message, and knows to later check that the child test process terminated with an error
This is outstandingly complicated, and I don't think a reasonable expectation of "end users" of the compiler (Swift devs) to be able to build a system like this for themselves. If testing assertions is hard, nobody will do it.
Idea 3: Use platform-specific workarounds
I.e. let the trap happen, but use a platform-specific API to try to recover from it. Matt Gallagher has a fantastic article about this: Partial functions in Swift, Part 2: Catching precondition failures.
He wrote the CwlPreconditionTesting package, which also underpins the throwAssertion matcher in the Nimble test assertions package.
This has significant downsides:
- It's specific to the OS
- E.g., unlike macOS and iOS which use the Mach-based APIs, tvOS and Linux need the POSIX implementation from
CwlPosixPreconditionTesting
- E.g., unlike macOS and iOS which use the Mach-based APIs, tvOS and Linux need the POSIX implementation from
- It's specific to the CPU architecture
- E.g. It took almost a year, and many contributors to figure out how to port this to ARM64.
Are there any other ideas I've missed?