Testing fatalError and friends

While I totally agree that fatalError() should be testable, I'm sure that provided example with DelayedImmutable is irrelevant and DelayedImmutable should be:

  • implemented in another way
  • should have another api design
  • should be replaced by alternatives

The way DelayedImmutable is implemented here is a poor design from my point of view and one should not use it in this form. There are better approaches for solving such kind of tasks without introducing fatalError().

Described problems can be solved in several ways.

  • introduction of lazy let in addition to lazy var
  • log errors instead of crashing app when:
    • value is set twice
    • return fallBackValue value if value is not set
  • carefully use implicitly unwrapped optionals
  • use another approaches of implementing the whole component

Without taking into account this specific example, sometimes fatalError() is the only pereferred variant, for example checking index in Array subscript.

In this cases where fatalError() is unavoidable, we should have the ability to cover such pieces of code with tests.
The good example is URLSession – we can see fatalErrors() all over its source code, but it is tested and one almost never meet crashes.
But we need to test not only positive cases, we should be able to simulate conditions where it will crash and cover it with tests.

Example:

private func objectFromJSON<T>(at path: String) -> T {
  let bundle = Bundle(for: BundleToken.self)
  guard let url: URL = bundle.url(forResource: path, withExtension: nil),
    let json: Any = try? JSONSerialization.jsonObject(with: Data(contentsOf: url), options: []),
    let result: T = json as? T else {
    fatalError("Unable to load JSON at path: \(path)")
  }
  return result
}

This code is generated by Sourcery and should never fail. We expect no crashes because everything checked at compile time. But in reality it can crash.
So it is good to make a test that expects crash in certain conditions. If no crash happens then test fails. Th test suite can expect lots of crashes for these rare conditions without interrupting of running the tests suite itself.

1 Like

I agree that fatalError() should be testable. Just wondering: how difficult would it be to have XCTest detect that fatalError() was called? (Xcode seems to notice because they start the debugger).

I wonder if, for the functions where it would be overly cumbersome to have to catch an error every time you called it, like array indexing or integer arithmetic, it could be something like this:

func + (lhs: Int, rhs: Int) throws(silent) -> Int

Where you could call it like this and it would just fatalError as usual:

let i = 5 + Int.max

But you could add a try statement to catch the error.

do {
  let i = try 5 + Int.max
} catch {
 ...
}

Basically creating an overload of the function that throws instead of fatalErrors.

I think you can achieve the same already by just adding throwing overloads.

The limitation with this sort of approach is that the compiler provides no assistance. If you happen to use try when the operation can't actually throw, the compiler will not only let you but also still use the throwing version with any additional overhead that incurs.

Not using overloads would perhaps mitigate the type resolution impact, which apparently can be quite severe for binary operators, but then it requires language changes for new syntax and capabilities.

When this line is inside a function would that function in turn require being labeled as "throw(silent)" or not?

It would not be required to be labeled as such; the function can choose whether to send the error up to the caller or just crash.

I gave your idea a try. For this simple number type:

enum NumError: Error { case overflow }

struct Num8 {
    var value: Int8

    init(_ value: Int8) { self.value = value }

    static func + (lhs: Self, rhs: Self) -> Self {
        let (result, overflow) = lhs.value.addingReportingOverflow(rhs.value)
        if overflow {
            if silentTryState.tryLevel == 0 {
                fatalError("overflow")
            } else if silentTryState.error == nil {
                silentTryState.error = NumError.overflow
            }
        }
        return Num8(result)
    }
}

with this "silent try" implementation:

struct SilentTryState {
    var tryLevel = 0
    var error: Error! // or a bool flag
}

// global? bad!
// thread local storage? will it work with async/await?
var silentTryState = SilentTryState()

// a better name wanted here!
func silent_try<T>(_ execute: () -> T, catch: (Error) -> Void) -> T {
    silentTryState.tryLevel += 1
    let result = execute()
    silentTryState.tryLevel -= 1
    if let error = silentTryState.error {
        silentTryState.error = nil
        `catch`(error)
    }
    return result
}

in this test app:

func bar() -> Num8 {
    Num8(127) + Num8(1) // overflows
}

func foo() {
    let result = silent_try {
        bar()
    } catch: { error in
        print("error caught in foo: ", error) // prints error
    }
    bar() // traps
}

foo()

I am getting the expected output: caught errors are reported within the catch block and uncaught errors terminate the app. In this implementation:

  • a better name wanted for "silent"_try".

  • the difference with normal try/catch is that when used from within the "silent_try { .... }" block the statements after the one that causes "exception" (in this case "Num8(127) + Num8(1)") are run "normally" (with an incorrect "wrapped over" result), and only eventually control gets to the "catch" block. To put differently - there is no "return as soon as possible on error" behaviour.

  • using global variable for "silentTryState" is very bad (thread unsafe, etc). Better would be using TLS but the might still not play well with async/await. †

† SilentTryState implementation using DispatchQueue local storage.

See a much better implementation in the next message.

struct SilentTryState {
    var tryLevel = 0
    var errorOccurred = false
    
    static let key = DispatchSpecificKey<Self>()
    
    static var value: Self {
        get {
            DispatchQueue.current.getSpecific(key: key) ?? Self()
        }
        set {
            DispatchQueue.current.setSpecific(key: key, value: newValue)
        }
    }
}

Is there a thing similar to thread/fiber local storage that's async/await compatible?

1 Like

Answering myself: @TaskLocal

[details=""Dynamic try/catch" implementation with usage example (minimally tested)"]

// ------------------
// MARK: main unchecked exceptions infrastructure

import Foundation

final class DynamicTry {
    private var errors: [Error] = []
    private var tryLevel = 0
    @TaskLocal fileprivate static var shared = DynamicTry()
    
    fileprivate func dynamicTry<T>(_ execute: () -> T, catch: ([Error]) -> Void) -> T {
        tryLevel += 1
        let result = execute()
        tryLevel -= 1
        if !errors.isEmpty {
            let errors = self.errors
            self.errors = []
            `catch`(errors)
        }
        return result
    }
    fileprivate func dynamicThrow(_ error: Error) {
        if tryLevel == 0 {
            fatalError("error caught \(error)")
        }
        self.errors.append(error)
    }
}

func dynamicTry<T>(_ execute: () -> T, catch: ([Error]) -> Void) -> T {
    DynamicTry.shared.dynamicTry(execute, catch: `catch`)
}

func dynamicThrow(_ error: Error) {
    DynamicTry.shared.dynamicThrow(error)
}

typealias SourceLocation = (/*file:*/ String, /*line:*/ Int)
enum DynamicPreconditionError: Error { case preconditionFailed(SourceLocation, String) }

@discardableResult
func dynamicPrecondition(_ expression: Bool, _ message: String? = nil, error: @autoclosure () -> Error? = {nil}(), file: String = #fileID, line: Int = #line) -> Bool {
    guard expression else {
        dynamicThrow(error() ?? DynamicPreconditionError.preconditionFailed((file, line), message ?? "precondition failed"))
        return false
    }
    return true
}

extension [Error] {
    var lines: String {
        var lines = map { String(" - \($0)") }
        lines.insert("errors caught: ", at: 0)
        return lines.joined(separator: "\n")
    }
}

// ------------------
// MARK: precondtion

func preconditionTest() {
    print("preconditionTest started")
    dynamicTry {
        _ = dynamicPrecondition(false)
    } catch: { errors in
        print(errors.lines)
    }
    print("preconditionTest finished")
    
    #if TEST_TRAPS
    dynamicPrecondition(false) // traps
    fatalError("unreachable")
    #endif
}

// ------------------
// MARK: integer overflow

enum Op {
    case add(Int, Int)
    case div(Int, Int)
}
enum IntegerOverflowError: Error { case integerOverflow(SourceLocation, Op) }

extension Int {
    static func + (_ lhs: Self, _ rhs: Self) -> Self {
        lhs.add(rhs)
    }
    static func / (_ lhs: Self, _ rhs: Self) -> Self {
        lhs.div(rhs)
    }
    func add(_ other: Self, file: String = #fileID, line: Int = #line) -> Self {
        let result = addingReportingOverflow(other)
        dynamicPrecondition(!result.overflow, error: IntegerOverflowError.integerOverflow((file, line), .add(self, other)))
        return result.partialValue
    }
    func div(_ other: Self, file: String = #fileID, line: Int = #line) -> Self {
        let result = dividedReportingOverflow(by: other)
        dynamicPrecondition(!result.overflow, error: IntegerOverflowError.integerOverflow((file, line), .div(self, other)))
        return result.partialValue
    }
}

func integerOverflowTest() {
    print("integerOverflowTest started")
    var x: Int = 0
    x = Int.max
    let result = dynamicTry {
        x.add(1)
        // x + 1 as well
    } catch: { errors in
        print(errors.lines)
    }
    print("integerOverflowTest finished, result: \(result)")
    
    #if TEST_TRAPS
    _ = x.add(1) // traps
    fatalError("unreachable")
    #endif
}

// ------------------
// MARK: optional unwrapping

protocol Inittable { init() }

postfix operator ^

enum OptionalUnwrapError: Error { case unwrappingNil(SourceLocation) }

extension Optional where Wrapped: Inittable {
    static postfix func^(v: Self) -> Wrapped {
        v.unwrap()
    }
    func unwrap(file: String = #fileID, line: Int = #line) -> Wrapped {
        switch self {
            case .none:
                dynamicPrecondition(false, error: OptionalUnwrapError.unwrappingNil((file, line)))
                return Wrapped()
            case .some(let wrapped):
                return wrapped
        }
    }
}

extension Int: Inittable {}

func optionalUnwrapTest() {
    print("optionalUnwrapTest started")
    let x: Int? = nil
    let result = dynamicTry {
        x.unwrap()
        // x^ // could be written like this
    } catch: { errors in
        print(errors.lines)
    }
    print("optionalUnwrapTest finished, result: \(result)")
    
    #if TEST_TRAPS
    _ = x.unwrap()
    fatalError("unreachable")
    #endif
}

// ------------------
// MARK: stack overflow

enum StackOverflowError: Error { case stackOverflow(SourceLocation, function: String) }

extension Thread {
    static let minGoodStackSize = 32*1024
    
    var stackSpace: Int {
        var approximateSP = 0
        let thread = pthread_self()
        let stackBase = pthread_get_stackaddr_np(thread)
        let stackSize = pthread_get_stacksize_np(thread)
        let stackLimit = stackBase - stackSize
        guard &approximateSP >= stackLimit && &approximateSP <= stackBase else {
            fatalError("TODO: SOMETHING ODD HERE")
        }
        let remSize = &approximateSP - stackLimit
        return remSize
    }
    func isEnoughStackSpace(for size: Int = minGoodStackSize) -> Bool {
        Thread.stackSpace >= size
    }
    func ensureEnoughStackSpace(for size: Int = minGoodStackSize, file: String = #fileID, line: Int = #line, function: String = #function) -> Bool {
        guard dynamicPrecondition(isEnoughStackSpace(for: size), error: StackOverflowError.stackOverflow((file, line), function: function)) else {
            return false
        }
        return true
    }
    static var stackSpace: Int {
        Thread.current.stackSpace
    }
    static func isEnoughStackSpace(for size: Int = minGoodStackSize) -> Bool {
        Thread.current.isEnoughStackSpace(for: size)
    }
    static func ensureEnoughStackSpace(for size: Int = minGoodStackSize) -> Bool {
        Thread.current.ensureEnoughStackSpace(for: size)
    }
}

@inline(never) func stackAbuser() -> Int {
    guard Thread.ensureEnoughStackSpace() else {
        // dynamicThrow is already thrown inside ensureEnoughStackSpace, no need to do it here
        return 1
    }
    return 1 + stackAbuser()
}

@inline(never) func stackOverflowTest() {
    print("stackOverflowTest started")
    var level = 0
    dynamicTry {
        level = stackAbuser()
    } catch: { errors in
        print("\(errors.lines), level reached: \(level), each frame ~\(Thread.stackSpace / level) bytes")
    }
    print("stackOverflowTest finished, level: \(level)")
}

// ------------------
// MARK: array bounds checks

protocol DynamicThrowable {}
enum IndexOutOfRange: Error { case indexOutOfRange(SourceLocation, Int, Range<Int>) }

extension Array where Element: Inittable {
    private subscript(normal index: Int) -> Element {
        get { self[index] }
        set { self[index] = newValue }
    }
    subscript(at index: Int, file: String = #fileID, line: Int = #line) -> Element {
        get {
            guard dynamicPrecondition(index >= 0 && index < count, error: IndexOutOfRange.indexOutOfRange((file, line), index, 0 ..< count)) else {
                return Element()
            }
            return self[normal: index]
        }
        set {
            guard dynamicPrecondition(index >= 0 && index < count, error: IndexOutOfRange.indexOutOfRange((file, line), index, 0 ..< count)) else {
                return
            }
            self[normal: index] = newValue
        }
    }
}

extension Array where Element: DynamicThrowable & Inittable {
    subscript(_ index: Int) -> Element {
        get { self[at: index] }
        set { self[at: index] = newValue }
    }
}

// add more later:
extension Int: DynamicThrowable {}

func arrayTest() {
    print("arrayTest started")
    var intArray = [1, 2, 3]
    let result = dynamicTry {
        intArray[at: 3]
        // intArray[3] // could be written like this
    } catch: { errors in
        print(errors.lines)
    }
    print("arrayTest finished, result: \(result)")
    
    #if TEST_TRAPS
    _ = intArray[at: 3] // traps
    fatalError("unreachable")
    #endif
}

// ------------------
// MARK: combined test

func combinedTestInternal(_ i: Int = .max) {
    let array = [1, 2, 3]
    let result = 100.div(array[at: i.add(1)])
    // let result = 100 / array[i + 1] // could be written like this
    dynamicPrecondition(result == 50)
}

func combinedTest() {
    print("combinedTest started")
    dynamicTry {
        combinedTestInternal()
    } catch: { errors in
        print(errors.lines)
    }
    print("combinedTest finished")
    
    #if TEST_TRAPS
    combinedTestInternal() // traps
    fatalError("unreachable")
    #endif
}

// ------------------
// MARK: tests
preconditionTest()
integerOverflowTest()
optionalUnwrapTest()
stackOverflowTest()
arrayTest()
combinedTest()

[/details]

Edit: implementation cleaned & fixed.

1 Like

The new swift-testing framework has an issue tracking on this.

Maybe we can thumb up to show our concern about this issue. Potentially make the team raise its priority.

1 Like

Just upvoted. Indeed, I believe this should be solved in the tooling as the tooling will be able to run a test in a separate process and observe the process (which seems to be the simplest fix).

More in general: I believe that we as users of the language should be able to trust that all language features are testable in a straightforward manner. And not have to add complexity to our code/tests just to make a language feature testable.

1 Like

I significantly cleaned the "dynamic try/catch" implementation above. It could be used in places where other languages use "unchecked exceptions", e.g. we could use it instead of trapping when unsafely unwrapping an optional, or on out of bounds errors during array access, or on arithmetic overflow, or in as! casts / try! blocks when we currently trap, etc.


Example implementation of unsafe unwrapping:

protocol Inittable { init() }

postfix operator ^
enum OptionalError: Error { case unwrappingNil }

extension Optional where Wrapped: Inittable {
    static postfix func ^ (_ v: Self) -> Wrapped {
        guard let v else {
            dynamicThrow(OptionalError.unwrappingNil)
            return Wrapped()
        }
        return v
    }
}

extension Int: Inittable {}

To get to:

var x: Int?
dynamicTry {
    let y = x^
    print(y) // prints 0
} catch: { error in
    print("caught: \(error)") // caught: unwrappingNil
}

Note that without dynamicTry/catch the unwrap is failing right away (similar to how it currently works). When used within dynamicTry/catch brackets the code continues with a "default" value (so in this example "0" is being printed), and then the error is eventually getting caught in the catch block.

Similarly array subscript could be implemented, with the out-of-bounds getter returning "default" value (and then getting caught) and the out-of-bounds setter being ignored (and then getting caught).

Arithmetic overflow could be reworked similar to how Num8 is doing it in the example implementation linked above.


They're likely to be cases when we'd want to skip some "expensive" calculations in case of errors. Compared to built-in try/catch this would be a manual check:

// without error short-circuiting:
func foo() -> T {
    let barValue = bar() // this could throw dynamically
    return expensiveCalculation(barValue)
}

// with error short-circuiting:
func foo() -> T {
    let barValue = bar() // this could throw dynamically
    if DynamicTry.errorOccurred { return someDefaultValue }
    return expensiveCalculation(barValue)
}
1 Like

Stack overflow checks integrate nicely with dynamic throw.

@inline(never) func stackAbuser() -> Int {
    // †
    guard Thread.ensureEnoughStackSpace() else {
        // dynamicThrow is already thrown inside ensureEnoughStackSpace, no need to do it here
        return Int()
    }
    return 1 + stackAbuser()
}

func stackOverflowTest() {
    let level = dynamicTry {
        let level = stackAbuser()
        print("after stack Abuser, level: \(level)")
        return level
    } catch: { level, error in
        print("\(error), level reached: \(level), each frame ~\(Thread.stackSpace / level) bytes")
    }
    print("after dynamicTry, level: \(level)")
}

stackOverflowTest()

Example outputs:

stackOverflow, level reached: 173655, each frame ~48 bytes

Edit:
† - for functions returning Void or returning "Inittable" type (or equally a type that has a notion of "default value") the guard stack space check could be autogenerated.

func userFunction() -> T {
    // THE NEXT LINE COULD BE AUTOGENERATED:
    guard Thread.ensureEnoughStackSpace() else { return T() } // or T.defaultValue

    return 1 + stackAbuser()
}

There's an increased code size and runtime implications, other than that the change is additive:

  • if there's enough stack space the code works as before (aside from increased size and added runtime check)
  • if there's not enough space and the function is called outside "dynamicTry" brackets - it will reliably crash on a precondition before stack is actually overflown (without this change the code would semi-reliably crash because of the actual stack overflow).
  • if there's not enough space and the function is called within "dynamicTry" brackets the out of stack error will be thrown and reliably caught within the catch block.

The whole thing above.

1 Like

This one is seriously cool: checking array out of bounds errors using unmodified array subscript syntax!

func foo() {
    var intArray = [1, 2, 3]
    intArray[3] = 42 // normally traps!
}

func arrayTest() {
    dynamicTry {
        foo()
    } catch: { _, error in
        print("error caught: \(error)")
    }
    print("afterwards")
}

arrayTest()

Outputs:

error caught: indexOutOfRange(3, Range(0..<3))
afterwards

One of the design considerations is whether catch catches the first or the last thrown error. Or, perhaps all of them? Let's give it a try:

func combinedTestInternal(_ i: Int = .max) {
    let array = [1, 2, 3]
    // let result = 100.div(array[at: i.add(1)])
    let result = 100 / array[i + 1] // could be written like this
    dynamicPrecondition(result == 50)
}

This example would trigger several fatal errors in a row if run normally: addition overflow, index out of bounds, division by zero, precondition failure. Here's the output when those errors are caught by dynamicTry:

errors caught: 
 - integerOverflow(("dynamicThrow/main.swift", 283), dynamicThrow.Op.add(9223372036854775807, 1))
 - indexOutOfRange(("dynamicThrow/main.swift", 283), -9223372036854775808, Range(0..<3))
 - integerOverflow(("dynamicThrow/main.swift", 283), dynamicThrow.Op.div(100, 0))
 - preconditionFailed(("dynamicThrow/main.swift", 284), "precondition failed")

Why "100.div(array[at: i.add(1)])" instead of normal "100 / array[i + 1]" you may ask? Traditional form also works, but unfortunately I can't add file+line information to the operators (and if I add it to a subscript override my override is no longer preferred).

static func + (_ lhs: Self, _ rhs: Self, file: String = #file, line: Int = #line) -> Self {
// 🛑 Operators must have one or two arguments
1 Like

My quick ⌘+F didn't find a reference to the swift-testing framework but it seems like the #expect overload referenced by this PR does exactly what's being asked about.

I unfortunately don't see it in the public docs for expectations so I suspect that it's unreleased :frowning_face:. Hopefully it will be released soon though!

EDIT: There's a draft PR containing a pitch for this feature here

It's still under development and has not been approved as a feature yet. :slight_smile: I plan to pitch this feature formally once the Testing Workgroup has been established.

There's some additional discussion here.

2 Likes