Testing fatalError and friends

Testing assertions is outstandingly difficult in Swift today. A very informal poll suggests that some other people think so, too. There was a thread on the matter back in 2015, but it didn't get much traction. I would like to start a discussion on:

  1. What's the guidance on how to do it today?
  2. What can we do to improve this area?

For the sake of easier communication, let's take the DelayedImmutable property wrapper example (from SE-0258) as an example:

@propertyWrapper
struct DelayedImmutable<Value> {
  private var _value: Value? = nil

  var wrappedValue: Value {
    get {
      guard let value = _value else {
        fatalError("property accessed before being initialized")
      }
      return value
    }

    // Perform an initialization, trapping if the
    // value is already initialized.
    set {
      if _value != nil {
        fatalError("property initialized twice")
      }
      _value = newValue
    }
  }
}

In this thread, by "assertion", I'm referring to any of assert, assertionFailure, precondition, preconditionFailure, fatalError, etc., such as those fatalErrors which prevent reading before intiailization and double initialization.

(Non)-Idea -1: Don't write assertions

I've seen several suggestions only document preconditions in API documentation, but not bother to verify them at runtime. The thinking goes that if a user calls your APIs in a way that doesn't meet your contractually-stipulated prerequisite, an "all bets are off" mode applies. Slap an "undefined behaviour" label on the problem, leave the callers to pick up the pieces, and call it a day.

I think this is a real "you're holding it wrong" vibe. It might fit in C, where the priority to to squeeze out every last cycle from the hardware, but it certainly doesn't fit the Swift ethos, which highly values safety and developer experience.

There's clearly a benefit to testing pre/post-conditions, look no further than the Swift standard library itself, which tests preconditions extensively (example). Some languages like Ada take its importance even further, and go out of their way to make precondition and postcondition contracts an explicit syntax in their language.

In the DelayedImmutable example, I'm not even sure if it's possible to remove the fatalError. The get must return a Value, or call a Never-returning function like fatalError. What Value could we return, if we don't have one? The only alternative I see to try to return some undefined garbage memory with unsafeBitCast.

(Non-)Idea 0: Use assertions, but don't bother testing them

Q: Do I have to test ALL my code?

A: Of course not, only the parts you want to work correctly.
– Uncle Bob or something, I don't remember :')

Idea 1: Use throw instead

This would make catching possible, thus testing really easy. Superficially, this appears to match how it's done in Java, Python and the like. The big difference is that those languages have exceptions, where raising an exception doesn't require changes to every caller up the call stack.

In practice, Swift's error-throwing does not at all work as an assertion mechanism:

  1. throws is exposed at the type system level (and it needs to be, because the caller needs to be an active participant in the error-handling process). This means that if you have some old non-throws function, you can't just add a new if !precondition { throw PreconditionFailed() }. You'd need to mark it throws which:

    1. In the best case, you control all the callers and can just put try! before all of them.
    2. In the worst case, you're writing a public API of a library, and you simply can't make this change without breaking your API.

    Compare this with Java, where checked exceptions are the norm, but unchecked exceptions are still possible. Any callee function can choose to raise an error, without any participation needed from its caller. (At the expense of a larger binary, and a much slower error path, which needs to do stack unwinding. I'm not arguing exception-throwing is better than Swift error-throwing.)

  2. There are many places that you simply can't mark throws, even if you wanted:

    1. Property getters (Fixed in Swift 5.5 via SE-0310 Effectful Read-only Properties :partying_face:)

      DelayedImmutable working example
      @propertyWrapper
      struct DelayedImmutable<Value> {
        private var _value: Value? = nil
      
        struct NotYetInitialized: Error {}
      
        var wrappedValue: Value {
          get throws { // ✅ Allowed to be marked `throws`
            guard let value = _value else {
              // fatalError("property accessed before being initialized")
              throw NotYetInitialized () // ✅
            }
            return value
          }
      
          // Perform an initialization, trapping if the
          // value is already initialized.
          set {
            if _value != nil {
              fatalError("property initialized twice")
            }
            _value = newValue
          }
        }
      }
      
    2. Property setters

      DelayedImmutable comiler error
      @propertyWrapper
      struct DelayedImmutable<Value> {
        private var _value: Value? = nil
      
        struct InitializedTwice: Error {}
      
        var wrappedValue: Value {
          get {
            guard let value = _value else {
              fatalError("property accessed before being initialized")
            }
            return value
          }
      
          // Perform an initialization, throwing InitializedTwice if the
          // value is already initialized.
          set throws { // ❌ 'set' accessor cannot have specifier 'throws'
            if _value != nil {
              throw InitializedTwice() 
            }
            _value = newValue
          }
        }
      }
      
    3. @convention(c) functions (meaning you wouldn't be able to use assertions in callbacks from C code, which are arguably one of the places where careful validation is most necessary!)

    4. subscript setters (e.g. you can't assert that a given index is in range!)

    5. Any members which satisfy a protocol that doesn't allow it.

      • E.g. you could never put an assertion in the description: String property of CustomStringConvertible, because that protocol declares that the description property is non-throwing.
  3. On a more cosmetic, less show-stopping level, you now need throws and try absolutely everywhere in your program. If the Standard Library had taken this appraoch, every array[index] operation would need a try in front of it.

Idea 2: Run tests in a separate process, and detect crashes

As far as I can understand it, I think this is how the Standard Library's expectCrashLater works.

This is outstandingly complicated, and I don't think a reasonable expectation of "end users" of the compiler (Swift devs) to be able to build a system like this for themselves. If testing assertions is hard, nobody will do it.

Idea 3: Use platform-specific workarounds

I.e. let the trap happen, but use a platform-specific API to try to recover from it. Matt Gallagher has a fantastic article about this: Partial functions in Swift, Part 2: Catching precondition failures.

He wrote the CwlPreconditionTesting package, which also underpins the throwAssertion matcher in the Nimble test assertions package.

This has significant downsides:

  1. It's specific to the OS
    • E.g., unlike macOS and iOS which use the Mach-based APIs, tvOS and Linux need the POSIX implementation from CwlPosixPreconditionTesting
  2. It's specific to the CPU architecture
    • E.g. It took almost a year, and many contributors to figure out how to port this to ARM64.

Are there any other ideas I've missed?

3 Likes

To be fair, this is how all test runners should work. A crash within a test case shouldn't prevent all the test suite to be tested. XCTest does it.
It's quite unfortunate XCTest doesn't support expected crashes, there is a big room for improvement here.
Also, it's a bad idea to try to restore a program after one of the threads(or coroutines) has entered an unrecoverable state. Swift (as well as most of other languages) doesn't provide instruments to isolate one part of the program from the others. So, while you can easily trap a faulty thread, you'll end up with leaked resources(heap objects, file handles, locks, etc) that thread was responsible for releasing (for more info on the subject search for reasons why Thread.stop() was deprecated in Java for example). To do it properly you should let the program die and delegate all the cleaning to the OS.

9 Likes

For assert, assertionFailure, precondition, and fatalErrors that aren't being used to satisfy the return type of a function, you can wrap the system functions in functions of your own which set a static variable instead of crashing when they're called. The test case can then read this to determine if an assertion would have occurred. Obviously execution of the function will continue, but for the assert family of functions, that will be the case in production anyway, and you might like to know what the behavior is in that case.

This can also be a useful hook to add things like logging so you can find out about asserts in production.

Perhaps I'm misunderstanding you, but precondition and fatalError are not omitted in release builds, only the assert family are.

Poorly worded on my part; I'll update it. I did note that it's the assert family of functions that doesn't stop executing in the next sentence.

I swift concurrency might help isolate crashes to specific async tasks.

A solution I've seen to this is to wrap fatalError or similar in functions that, by default, just do the default (hard terminate), but can be swapped out during tests with something else.

The trick is what the replacement does after receiving an error. The replacement has to return Never, but how do you do that without terminating the process? You certainly can't return to the caller!

See here for a clever solution to this: the alternative "never return" to terminating is to literally never return, as in an infinite loop. But you need to make sure this doesn't stall the run loop you might be in, so you need to pump it in the meantime.

I've never actually tried this though.

2 Likes

This doesn't help with the immediate problem, but I will say I've wished for a while that Swift let you "catch" the things we today call "fatal errors". I've never really agreed with the argument that a function can know, absolutely for all possible invocations of it, that the program calling it needs to terminate if something happens. A program might be able to respond to it by shutting down an entire thread or sub-process without terminating everything, or it might have more granular retry or backup options. Tests catching them is just a specific example.

I would love to be able to catch force unwrapped nils.

To me this would look like a language feature where you can "throw" but not have to declare yourself as throws. This could, perhaps, be a different keyword applied to an Error-conforming object, like fail MyError(). If you do ... catch around this, you'll catch it, but since the method isn't throws, you don't have to catch (or declare yourself throws), and then they cause termination. If this existed you'd probably rewrite fatalError and family to do this internally.

I can think of many many reasons why this either can't work or would run contrary to the language design. Maybe this would make Obj-C interop impossible, would destroy optimizations related to not building exception-unwinding mechanisms everywhere, what do you do with warnings that you're do... catching something with no try's (or trying something that isn't throws), etc. etc..

I still dream about it though...

1 Like

My rule of thumb is

  • If code is a dependency (being imported by clients or other dependencies) than throw Errors, only use fatalError as the last resort. I see fatalError as the better version of force unwrap because you can provide a crash message.
  • If code is an end app target, then in some cases where there are no meaningful fallbacks after some errors are thrown, do fatalError.

I never test fatalErrors myself as it is not an expected outcome, it should be invoked in only the unexpected cases. Its a great tool to catch problems during development.

1 Like

It could be interesting to see, how Rust handles #[should_panic] on test cases.

2 Likes

In our app we are using assertions which don’t terminate the process. Basically it is just logging. We have an extensible architecture which allows registering different assertion handlers. Default configuration is the following:

  • In production, assertions are logged to Firebase
  • In debug builds, if debugger is connected, hardware breakpoint is triggered (with an option to silence assertion for file-line pair)
  • If debugger is not connected, alert is shown.
  • If running inside the test, failure in current test is reported.

Also it is possible to override handlers for certain scope, and install handler for expected assertion - it will fail the test if no assertion is produced when leaving the scope.

But this applies only for assertions that don’t terminate the process. If terminating the process is the expected behavior to be tested, I really don’t see other alternatives to running SUT in a separate process.

2 Likes

Hey Dima,

XCTest does it.

I didn't know this, but it makes sense!

Perhaps a good improvement would be to build "Idea 2" directly into XCTest, so people don't have to roll it themselves.

Indeed, this is my issue with Idea 3. At a minimum, I know it's leaking memory during my test runs, but it might also leak other resources that cause more of an issue. It works on my machine™, but IDK if it'll come to bite my later.

Hey I cite this is an example all the time, myself! So yeah, it's pretty clear that any solution that stays within the same process might have these correctness issues. I'm not sure yet how important they happen to be in test suites (e.g. do any test suites use so many file handles, that leaking them would be an issue before the suite is done? And that kind of reasoning would have to be done about every other kind of resource).

I'm afraid that won't really help. You could isolate crashes to threads today, but that has all the issues Dima outlined in his post. Isolating at the task level instead won't solve that underlying problem.

1 Like

This is what I've been doing in my own code (I forgot to list it in the OP), but I find it breaks down with third party libraries. There's no way to hook into their calls to the standard assert and have them call my wrapped alternative.

This is an interesting idea. Does this mean that each fatalError effectively leaks a thread, by leaving it a needs-to-infinite-loop-forever state?

As I alluded in my original post, I don't think this is a viable option. Its applicability is just far too limited.

E.g. My implementation of NSTableViewDelegate.tableView(_:viewFor:row:) isn't allowed to throw, so I can only make assertions using fatalError.

I only don't test them because it's hard, but I do think they should be tested. If I have an array implementation that does bounds checks, I want to actually verify the bounds checks work, otherwise what's the point? Perhaps this is being too zealous about tests, but as far as I'm concerned, if I can't test a code branch and its correctness, it doesn't exist to me.

2 Likes

This is where Distributed Actors can come in (spiritual successor to Distributed Objects), since they can actually be process-isolated. Swift isn't Erlang, but you can get pretty close.

Not helpful to the original challenge posed in this thread, of course, but tangentially for anyone interested in a high level of robustness to runtime problems, Distributed Actors is worth exploring.

2 Likes

I, too, treat fatalError as basically a "hey, you forgot to do this" hook. It's useful for test-driven development where you drop one to remind yourself what the next step in the development process is. Having a test terminate with the stack pointing exactly where it stopped is very helpful.

This, of course, means they should all be gone once development is complete.

I've mostly accepted that any "expected" errors, and even programmer errors that haven't been designed out, should just throw. I've written wrappers for stuff like "force" unwrapping optionals that throw instead of hard failing (and also allow tailoring the error message). This means I have to deal with all the problems the OP mentioned about throwing (i.e. if it's a getter, the setter has to be separated out into a function), but I consider this the least bad option.

The biggest problem, as OP also pointed out, is third party code. The best you can do is wrap calls inside your own that anticipate what would fatalError and turn it into a throw before really calling into the library. That's what I think is problematic about fatalError: when you write that in a library, you're making every possible client of that library terminate, even when the client has a recovery plan.

2 Likes

And worse yet, even if your client does have a recovery plan, there's no standardized API for it, so libraries can't make use of it, even if they were willing.

It's basically the same issue that libraries had with logging, prior to Swift standardizing the logging APIs.

2 Likes

This is an interesting example. The reason it can't throw is because the framework code calling the delegate doesn't handle errors. The contract it makes you follow is "you need to give me an answer for this when I ask for it". If you could throw, what would the framework do? It would have to be a generic strategy that works with all apps. It seems to me it could only terminate itself, or provide some hook back into your application, like another delegate call (tableView(_:viewForRowError:) or something), but then it would still have to draw a row, and would have to pick a default.

Perhaps this just isn't the right place for an error to be triggered. Whatever work is being done to prepare the rows that might fail, this might need to move somewhere that is triggered by your code, not the framework, and just saves the result somewhere then notifies the framework it's ready to be read through the delegate (reloadData()). Then since it's your code starting this work, you can design it to throw, and decide what should happen. The prod code might catch the error and terminate, but the tests can catch and assert.

Indeed, I don't think the framework can do anything productive in that case, but that's kind of my point. fatalError is like an escape hatch of last resort, and error condition that was unexpected and unrecoverable. Except we want it to be sort of recoverable, for detecting that condition in a test fixture.

Kind of. You can do preflight checks that minimize the chance of failure in the delegate method itself, but the use of a force unwrap or fatalError remains basically unavoidable. Let me illustrate with an example:

Pretty much every implementation of tableView(_:viewFor:row:) will call NSTableView.makeView(withIdentifier:owner:) to reuse an existing view, or create a new one if none are available. It returns an optional, since of course there's a chance that at the time of look-up, there's no registered view's with the given identifier.

We can minimize the risk by doing a preflight check earlier in the code that constructs our NSTableView, to validate that every identifier we're going to eventually use is registered, but that has some caveats/limitations:

  1. Notice that the force unwrap continues to be necessary in the tableView(_:viewFor:row:) method. We can be more confident it'll never trip, but it'll still have to be ther, to satisfy the type system.
  2. You have to be careful to keep your preflight checks and tableView(_:viewFor:row:) implementation in sync. If you start using a new cell identifier in the delegate method, but forget to pre-check it, then you're susceptible to nil errors.
  3. "Move the error check elsewhere" doesn't scale well. Imagine if this use of NSTableViewDelegate was part of a UI framework that wraps AppKit. It would be rather tricky to design your API in a way that allows for you to the do the correct pre-checking on your client's behalf, and then accessing the makeView function without a risk of erroring.

At bottom, it still seems to me like a preferable solution be to have a fatalError right in the delegate method, with a friendly/informative message, and a test which validates that attempts to make views with unknown identifiers leads to those fatalError messages.

1 Like

I've dealt with this same issue on UIKit (UITableView and UICollectionView). I think part of the pain is not so much about assertions but Obj-C interop, plus the fact the API you're using just isn't that safe. As you said, the registration and dequeuing of cells is separated in the AppKit/UIKit API, so you can't compile-time eliminate errors about mismatches between the two...

...or can you?

What you can do (and I'm in the habit of doing this whenever I have to work with UIKit in Swift apps these days) is confront the trickiness of point 3 head-on and stuff this inherent lack of safety inside a Swift API that is compile-time safe:

protocol TableCell: UIView {
    associatedtype ViewModel

    var viewModel: ViewModel? { get set }
}

struct TableData<ViewModel> {
    let rows = CurrentValueSubject<[ViewModel], Never>([]) // Change to a double-layered array to support sections
}

final class TableView<Cell: TableCell>: UIView, UITableViewDataSource {
    typealias Data = TableData<Cell.ViewModel>
    
    init(data: Data) {
        self.data = data
        
        super.init(frame: .zero)
        
        addSubview(table)
        constrainSubviewToSelf(table) // Some helper function
        
        table.register(Cell.self, forCellReuseIdentifier: reuseId)
        table.dataSource = self
        
        dataSubscription = data.rows
            .sink { [table] _ in table.reloadData() }
    }
    
    required init?(coder: NSCoder) {
        fatalError("Newp")
    }
    
    func numberOfSections(in tableView: UITableView) -> Int {
        1
    }
    
    func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
        data.rows.value.count
    }
    
    func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
        let cell = tableView.dequeueReusableCell(withIdentifier: reuseId, for: indexPath) as! Cell // Not optional, but will crash if you didn't register the right cell.  Also the downcast is guaranteed to succeed
        cell.viewModel = data.rows.value[indexPath.row] // This is guaranteed to be a valid index
        return cell
    }
    
    private let table = UITableView()
    private let data: Data
    
    private let reuseId = "Cell"
    private var dataSubscription: AnyCancellable?
}

This approach restricts each table view to one type of cell. You can support variation within the cell, but that won't be optimal from a cell reuse perspective. If you want to support multiple cell types that are reused separately, you can add an inverse type relation from a cell's view model back to the cell:

protocol TableCellViewModel {
    associatedtype Cell: TableCell where Cell.ViewModel == Self
}

extension TableCellViewModel {
    // The Cell associatedtype is lost after type erasure, so we use an extension to open the existential back up and access that Cell type.
    func dequeueCell(in tableView: UITableView, at indexPath: IndexPath) -> UITableViewCell {
        let reuseId = String(reflecting: Cell.self)
        tableView.register(Cell.self, forCellReuseIdentifier: reuseId) // Registering every time is OK
        let cell = tableView.dequeueReusableCell(withIdentifier: reuseId, for: indexPath) as! Cell // Downcast will of course always succeed
        cell.viewModel = self
        return cell
    }
}

struct FlexibleTableData {
    let rows = CurrentValueSubject<[any TableCellViewModel], Never>([]) // Change to a double-layered array to support sections
}

final class FlexibleTableView: UIView, UITableViewDataSource {
    init(data: FlexibleTableData) {
        self.data = data
        
        super.init(frame: .zero)
        
        addSubview(table)
        constrainSubviewToSelf(table) // Some helper function
        
        table.dataSource = self
        
        dataSubscription = data.rows
            .sink { [table] _ in table.reloadData() }
    }
    
    required init?(coder: NSCoder) {
        fatalError("Newp")
    }
    
    func numberOfSections(in tableView: UITableView) -> Int {
        1
    }
    
    func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
        data.rows.value.count
    }
    
    func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
        let viewModel = data.rows.value[indexPath.row] // This is guaranteed to be a valid index
        return viewModel.dequeueCell(in: tableView, at: indexPath)
    }
    
    private let table = UITableView()
    private let data: FlexibleTableData
    
    private var dataSubscription: AnyCancellable?
}

This feels off-topic, but then again I get the impression the language designers intend any error that's not so unexpected it really should immediately alert the developer to its existence to be a throw, so the difficulty in "trapping" such unexpected failures is intentional and puts pressure on us to improve our design. This could either mean adjusting to a design that throws, or better yet, eliminating the possibility of failure through Swift's very powerful static type system. So maybe it's right on topic, and most "I really want to test for this assertion" discussions really should end with "why not make it a compile time failure instead?"

2 Likes

I've implemented a really similar set of abstractions for myself, but while it does sort-of address this example, it's not broadly generalizable.

I get the impression the language designers intend any error that's not so unexpected it really should immediately alert the developer to its existence to be a throw

I don't, because of how limited the contexts are in which you're actually allowed to throw. Non-throwing method requirements on protocols is the perfect example, where you're literally forced to not throw, and there's nothing you can do about it.

I think assertions are semantically different from thrown errors:

  1. Assertions shouldn't fail, so we don't want to burden callers with worrying about how to recovery from generally unrecoverable nonsense scenarios (e.g. the view controller NIB you wanted is missing. How do you proceed with an arbitrary part of your app being completely unavailable?).
  2. Errors might reasonably happen in a final production build, so we do want callers to consider carefully how to handle them.

I think even if throw could be used everywhere, it would still be a sub-optimal solution, because it blurs the lines between these two distinct kinds of error conditions. Consider how many try! statements you'd have in your code, and how hard it would be to discern:

Did this dev try! because they were just being lazy, or because they're dealing with an assertion-style error that shouldn't happen, and which they couldn't reasonably recover from?

For example, the bounds-check on Array's subscript is an assertion. To use throws would be inappropriate, because there wouldn't be much the caller could reasonably do to recover. It would make it easier to catch the error and detect it in a test fixture, but it would just muddy the real caller code. Imagine just how many try! a[i] you'd see all over. Nonetheless, it's better to have the assertion+trap than to have out-of-bounds reads and UB.

I generally agree, but there are fundamental times where the type system pass off the torch, and it's runtime values from then onwards. Again, subscripts on Arrays are a perfect example. Without dependent types and a significantly more sophisticated type system, there's no way to make indexing arrays with arbitrary integers into a compile-time problem. A lot of the convincing of the type system involves adding runtime assertions, to buy static guarentees (casting an object to get access to it as a more narrowed type, force unwrapping optionals to get a non-optional value from them, fatalError to avoid having to return a value from a function where a value isn't available, etc.). I don't see how that could be entirely avoided :/

1 Like