tl;dr: Testing preconditions is important. We should be able to do it in Swift without contorting our tests to fit the language.
Strongly agree. I'm bummed to not see much else in this thread about testing preconditions/fatalErrors/assertions, etc.
It's an important use case and certainly fits with the "flexible and capable of accommodating many needs" principle laid out in the vision document, as well as the "validate expected behaviors or outcomes" bullet under "Approachability."
Consider making a custom collection, which is a use case where being able to test preconditions is incredibly important. For example:
precondition(i >= startIndex && i <= endIndex, "index out of bounds")
Even in a precondition that simple, it's easy to mess up the comparison operators (did you see the bug?).
Another common pattern is indices that are tied to a specific value, e.g. only being able to use a String.Index in the string it was born from, or an index into a tree that stores weak or unowned references to specific nodes for O(1) subscripting or navigation (index(after:) and friends). In these cases you need something like
precondition(i.isValidFor(self))
This could have some pretty complicated logic. You can test isValidFor in isolation, which helps, but not being able to test subscripting a collection with an invalid index from end to end leaves me with a distinct lack of confidence in my test suite.
In Testing fatalError and friends it seems that the consensus is "run the test in a child process and detect the crash," which is what the Swift standard library tests do. While I wish there was room for a larger rethink of how Swift deals with traps (perhaps something closer to Go's panic/recover mechanism), it would be a very heavy lift for such a mature language, and might not be possible at all due to decisions that have been locked down by ABI stability. There were some other suggestions in that thread that could be interesting – using distributed actors, or even having an in-process distributed actor with an isolated heap that could panic in a way that could be detectable or even catchable.
Assuming the path forward is run the test in a child process and detect the crash, there's one particular usecase that's worth considering: multiple assertPreconditionViolation expectations in a single test. Consider the following
struct MyArray: Collection, ExpressibleByArrayLiteral {
}
func testSubscripting() {
var a: MyArray = ["a", "b", "c"]
assertPreconditionViolation(a[-1], with: "index out of bounds") // assume the first argument is @autoclosure
assertEqual(a[0], "a")
assertEqual(a[1], "b")
assertEqual(a[2], "c")
assertPreconditionViolation(a[3], with: "index out of bounds")
a = []
assertPreconditionViolation(a[a.startIndex], with: "index out of bounds")
assertPreconditionViolation(a[a.endIndex], with: "index out of bounds")
}
This is how I'd like to write this test (YMMV of course). I want to have both edges of each edge case (e.g. a[-1] and a[0]) next to each other. It makes it easier to see at a glance that I'm testing exhaustively.
In the naieve implementation of "detect a crash in the child," the child would crash during the very first assertion. That assertion would pass, but none of the others would run at all. Which in practice means you can only have one "expect it to crash" assertion per test, and it must be the last assertion.
A more sophisticated implementation might run the test N times, once per assertPreconditionViolation, and only run a single assertPreconditionViolation per test run. But that implies that each non-crashing assertEqual in the above example would be run M times, where M is the number of calls to assertPreconditionViolation that appear after the assertEqual in question. If subscripting MyArray is slow for some reason, or if allocating new instances of MyArray is expensive, the extra calls can really slow things down. I think it's something resembling a factor of O(n^2).
You can imagine a version where fork(2) is called once per assertPreconditionViolation, which would be equivalent to the idealized functionality of being able to recover from traps – the test runs top to bottom with each line only run once – but that comes with all of the problems of using fork, and I'm sure isn't workable.
You can make a reasonable argument that getting the above test running in a straight line top to bottom is too big a lift, or perhaps not worth the effort. And even something like Rust's #[should_panic] which seems to essentially make any #[should_panic] test a single assertion, would be a big improvement to what we have now.
The big picture though, is that if we're going to allow for testing precondition violations (which I think we should!), the less Swift's implementation constrains what you can express in your test, the better. And the test above is seems to push pretty far up against what the language allows.