Proposal: XCTest Support for Swift Error Handling


(Chris Hanson) #1

We’d like feedback on a proposed design for adding support for Swift error handling to XCTest, attached below. I’ll mostly let the proposal speak for itself, but there are three components to it: Allowing test methods to throw errors, allowing the expressions evaluated by assertions to throw errors, and adding an assertion for checking error handling.

We’d love to hear your feedback. We’re particularly interested in some feedback on the idea of allowing the expressions evaluated by assertions to throw errors; it’s generated some debate because it results in writing test code slightly differently than other code that makes use of Swift error handling, so any thoughts on it would be particularly appreciated.

  -- Chris Hanson (chanson@apple.com <mailto:chanson@apple.com>)

XCTest Support for Swift Error Handling
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
Author(s): Chris Hanson <https://github.com/eschaton>
Status: Review
Review manager: TBD
Introduction
Swift 2 introduced a new error handling mechanism that, for completeness, needs to be accommodated by our testing frameworks. Right now, to write tests that involve methods that may throw an error, a developer needs to incorporate significant boilerplate into their test. We should move this into the framework in several ways, so tests of code that interacts with Swift error handling is concise and intention-revealing.

Motivation
Currently, if a developer wants to use a call that may throw an error in a test, they need to use Swift's do..catch construct in their test because tests are not themselves allowed to throw errors.

As an example, a vending machine object that has had insufficient funds deposited may throw an error if asked to vend an item. A test for that situation could reasonably use the do..catchconstruct to check that this occurs as expected. However, that means all other tests also need to use either a do..catch or try! construct — and the failure of a try! is catastrophic, so do..catch would be preferred simply for better reporting within tests.

func testVendingOneItem() {
    do {
        vendingMachine.deposit(5)
        let item = try vendingMachine.vend(row: 1, column: 1)
        XCTAssertEqual(item, "Candy Bar")
    } catch {
        XCTFail("Unexpected failure: \(error)")
    }
}
If the implementation of VendingMachine.vend(row:column:) changes during development such that it throws an error in this situation, the test will fail as it should.

One other downside of the above is that a failure caught this way will be reported as an expected failure, which would normally be a failure for which XCTest is explicitly testing via an assertion. This failure should ideally be treated as an unexpected failure, as it's not one that's anticipated in the execution of the test.

In addition, tests do not currently support throwing an error from within an assertion, requiring any code that throws an error to be invoked outside the assertion itself using the same techniques described above.

Finally, since Swift error handling is a general mechanism that developers should be implementing in their own applications and frameworks, we need to make it straightforward to write tests that ensure code that implements error handling does so correctly.

Proposed solution
I propose several related solutions to this issue:

Allow test methods to throw errors.
Allow test assertion expressions to throw errors.
Add an assertion for checking errors.
These solutions combine to make writing tests that involve thrown errors much more succinct.

Allowing Test Methods to Throw Errors

First, we can allow test methods to throw errors if desired, thus allowing the do..catch construct to be omitted when the test isn't directly checking error handling. This makes the code a developer writes when they're not explicitly trying to test error handling much cleaner.

Moving the handling of errors thrown by tests into XCTest itself also ensures they can be treated as unexpected failures, since the mechanism to do so is currently private to the framework.

With this, the test from the previous section can become:

func testVendingOneItem() throws {
    vendingMachine.deposit(5)
    let item = try vendingMachine.vend(row: 1, column: 1)
    XCTAssertEqual(item, "Candy Bar")
}
This shows much more directly that the test is intended to check a specific non-error case, and that the developer is relying on the framework to handle unexpected errors.

Allowing Test Assertions to Throw Errors

We can also allow the @autoclosure expression that is passed into an assertion to throw an error, and treat that error as an unexpected failure (since the code is being invoked in an assertion that isn't directly related to error handling). For example:

func testVendingMultipleItemsWithSufficientFunds() {
    vendingMachine.deposit(10)
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")
}
This can eliminate otherwise-dangerous uses of try! and streamline code that needs to make multiple assertions in a row.

Adding a "Throws Error" Assertion

In order to test code that throws an error, it would be useful to have an assertion that expects an error to be thrown in a particular case. Right now a developer writing code to test that an error is thrown has to test that error themselves:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        var vendingFailed = false
        do {
            _ = try vendingMachine.vend(row: 1, column: 1))
        } catch {
            vendingFailed = true
        }
        XCTAssert(vendingFailed)
    }
If we add an assertion that specifically checks whether an error was thrown, this code will be significantly streamlined:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
    }
Of course, some code may want to just detect that an error was thrown, but other code may need to check that the details of the thrown error are correct. We can take advantage of Swift's trailing closure syntax to enable this, by passing the thrown error (if any) to a closure that can itself contain assertions:

    XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
        guard let vendingError = error as? VendingMachineError else {
            XCTFail("Unexpected type of error thrown: \(error)")
            return
        }
        
        XCTAssertEquals(vendingError.item, "Candy Bar")
        XCTAssertEquals(vendingError.price, 5)
        XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
    }
This lets a developer very concisely describe an error condition for their code, in whatever level of detail they desire.

Detailed design
The design of each of the above components is slightly different, based on the functionality provided.

Tests That Throw

In order to enable test methods to throw an error, we will need to update XCTest to support test methods with a () throws -> Void signature in addition to test methods with a () -> Voidsignature as it already supports.

We will need to ensure tests that do throw an error have that error caught, and that it registers an unexpected failure.

Assertions That Throw

In order to allow assertions to throw an exception, we will need to enhance our existing assertions' @autoclosure expression parameters to add throws to their signature.

Because Swift defines a closure that can throw an error to be a proper supertype of a closure that does not, this will not result in a combinatorial explosion of assertion overrides, and will let developers naturally write code that may throw an error within an assertion.

We will treat any error thrown from within an assertion expression as an unexpected failure because while all assertions represent a test for some form of failure, they're not specifically checking for a thrown error.

The "Throws Error" Assertion

To write tests for code that throws error, we will add a new assertion function to XCTest with the following prototype:

public func XCTAssertThrowsError(
    @autoclosure expression: () throws -> Void,
                  _ message: String = "",
                       file: StaticString = __FILE__,
                       line: UInt = __LINE__,
             _ errorHandler: (error: ErrorType) -> Void = { _ in })
Rather than treat an error thrown from its expression as a failure, this will treat the lack of an error thrown from its expression as an expected failure.

Furthermore, so long as an error is thrown, the error will be passed to the errorHandler block passed as a trailing closure, where the developer may make further assertions against it.

In both cases, the new assertion function is generic on an ErrorType in order to ensure that little to no casting will be required in the trailing closure.

Impact on existing code
There should be little impact on existing test code because we are only adding features and API, not changing existing features or API.

All existing tests should continue to work as implemented, and can easily adopt the new conventions we're making available to become more concise and intention-revealing with respect to their error handling as shown above.

Alternatives considered
We considered asking developers continue using XCTest as-is, and encouraging them to use Swift's native error handling to both suppress and check the validity of errors. We also considered adding additional ways of registering failures when doing this, so that developers could register unexpected failures themselves.

While this would result in developers using the language the same way in their tests as in their functional code, this would also result in much more verbose tests. We rejected this approach because such verbosity can be a significant obstacle to testing.

Making it quick and clean to write tests for error handling could also encourage developers to implement error handling in their code as they need it, rather than to try to work around the feature because of any perceived difficulty in testing.

We considered adding the ability to check that a specific error was thrown in XCTAssertThrowsError, but this would require the ErrorType passed to also conform to Equatable, which is also unnecessary given that this can easily be checked in a trailing closure if desired. (In some cases a developer may just want to ensure an error is thrown rather than a specific error is thrown.)

We explicitly chose not to offer a comprehensive suite of DoesNotThrowError assertions for XCTest in Swift, though we do offer such DoesNotThrow assertions for XCTest in Objective-C. We feel these are of limited utility given that our plan is for all assertions (except XCTAssertThrowsError) to treat any thrown error as a failure.

We explicitly chose not to offer any additional support for Objective-C exceptions beyond what we already provide: In the Xcode implementation of XCTest, an Objective-C exception that occurs within one of our existing assertions or tests will result in a test failure; doing more than this is not practical given that it's possible to neither catch and handle nor generate an Objective-C exception in Swift.


(Drew Crawford) #2

I have on the order of ~700 tests in XCTest in Swift-language projects. I'm considering migrating away from XCTest, although not over this issue.

This proposal IMO addresses an important problem, but I am not convinced it actually solves it. #2 & #3 are basically sound API designs. It is a mystery to me why #3 "generated some debate" as this is a feature I already implement manually, but I can't address unknown concerns. I can tell you I implement this, and nothing terrible has happened to me so far.

#1 I would not use. The rest of this comment explains why.

Currently I write tests about like this:

try! hopefullyNothingBad()

Now this is "bad" because it "crashes" but that's (big sigh) actually "good" because the debugger stops and/or I get a crash report that identifies at least "some" line where something bad definitely happened.

Now that is not everything I want to know–I want someone to tell me a story of how this error was created deep down in the bowels of my application, how it spent its kindergarten years in the models layer before being passed into a controller where it was rethrown onto a different dispatch queue and finally ended up in my unit test–but we can't have everything. So I settle for collecting a line number from the test case and then going hunting by hand.

When the test function throws we no longer even find out a line number in the test case anymore, because the error is passed into XCTest and the information is lost. We have just the name of the test case (I assume; the proposal is silent on this issue, but that's the only way I can think of to implement it), and some of my tests are pretty long. So, that makes it even harder to track down.

This sounds like a small thing but my test coverage is so thorough on mature projects that mostly what I turn up are heisenbugs that reproduce with 2% probability. So getting the report from the CI that has the most possible detail is critical, because if the report is not thorough enough for you to guess the bug, too bad, because that's all the information you get and the bug is not reproducible.

For that reason, I won't use #1. I hesitate about whether to call it bad idea altogether, or whether it's just not to my taste. My sense is it's probably somewhere in the middle of those two poles.

I would use #2 and #3, assuming that I don't first migrate out to a non-XCTest framework.

···

On Jan 9, 2016, at 8:58 PM, Chris Hanson via swift-evolution <swift-evolution@swift.org> wrote:

We’d like feedback on a proposed design for adding support for Swift error handling to XCTest, attached below. I’ll mostly let the proposal speak for itself, but there are three components to it: Allowing test methods to throw errors, allowing the expressions evaluated by assertions to throw errors, and adding an assertion for checking error handling.

We’d love to hear your feedback. We’re particularly interested in some feedback on the idea of allowing the expressions evaluated by assertions to throw errors; it’s generated some debate because it results in writing test code slightly differently than other code that makes use of Swift error handling, so any thoughts on it would be particularly appreciated.

  -- Chris Hanson (chanson@apple.com <mailto:chanson@apple.com>)

XCTest Support for Swift Error Handling
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
Author(s): Chris Hanson <https://github.com/eschaton>
Status: Review
Review manager: TBD
Introduction
Swift 2 introduced a new error handling mechanism that, for completeness, needs to be accommodated by our testing frameworks. Right now, to write tests that involve methods that may throw an error, a developer needs to incorporate significant boilerplate into their test. We should move this into the framework in several ways, so tests of code that interacts with Swift error handling is concise and intention-revealing.

Motivation
Currently, if a developer wants to use a call that may throw an error in a test, they need to use Swift's do..catch construct in their test because tests are not themselves allowed to throw errors.

As an example, a vending machine object that has had insufficient funds deposited may throw an error if asked to vend an item. A test for that situation could reasonably use the do..catchconstruct to check that this occurs as expected. However, that means all other tests also need to use either a do..catch or try! construct — and the failure of a try! is catastrophic, so do..catch would be preferred simply for better reporting within tests.

func testVendingOneItem() {
    do {
        vendingMachine.deposit(5)
        let item = try vendingMachine.vend(row: 1, column: 1)
        XCTAssertEqual(item, "Candy Bar")
    } catch {
        XCTFail("Unexpected failure: \(error)")
    }
}
If the implementation of VendingMachine.vend(row:column:) changes during development such that it throws an error in this situation, the test will fail as it should.

One other downside of the above is that a failure caught this way will be reported as an expected failure, which would normally be a failure for which XCTest is explicitly testing via an assertion. This failure should ideally be treated as an unexpected failure, as it's not one that's anticipated in the execution of the test.

In addition, tests do not currently support throwing an error from within an assertion, requiring any code that throws an error to be invoked outside the assertion itself using the same techniques described above.

Finally, since Swift error handling is a general mechanism that developers should be implementing in their own applications and frameworks, we need to make it straightforward to write tests that ensure code that implements error handling does so correctly.

Proposed solution
I propose several related solutions to this issue:

Allow test methods to throw errors.
Allow test assertion expressions to throw errors.
Add an assertion for checking errors.
These solutions combine to make writing tests that involve thrown errors much more succinct.

Allowing Test Methods to Throw Errors

First, we can allow test methods to throw errors if desired, thus allowing the do..catch construct to be omitted when the test isn't directly checking error handling. This makes the code a developer writes when they're not explicitly trying to test error handling much cleaner.

Moving the handling of errors thrown by tests into XCTest itself also ensures they can be treated as unexpected failures, since the mechanism to do so is currently private to the framework.

With this, the test from the previous section can become:

func testVendingOneItem() throws {
    vendingMachine.deposit(5)
    let item = try vendingMachine.vend(row: 1, column: 1)
    XCTAssertEqual(item, "Candy Bar")
}
This shows much more directly that the test is intended to check a specific non-error case, and that the developer is relying on the framework to handle unexpected errors.

Allowing Test Assertions to Throw Errors

We can also allow the @autoclosure expression that is passed into an assertion to throw an error, and treat that error as an unexpected failure (since the code is being invoked in an assertion that isn't directly related to error handling). For example:

func testVendingMultipleItemsWithSufficientFunds() {
    vendingMachine.deposit(10)
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")
}
This can eliminate otherwise-dangerous uses of try! and streamline code that needs to make multiple assertions in a row.

Adding a "Throws Error" Assertion

In order to test code that throws an error, it would be useful to have an assertion that expects an error to be thrown in a particular case. Right now a developer writing code to test that an error is thrown has to test that error themselves:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        var vendingFailed = false
        do {
            _ = try vendingMachine.vend(row: 1, column: 1))
        } catch {
            vendingFailed = true
        }
        XCTAssert(vendingFailed)
    }
If we add an assertion that specifically checks whether an error was thrown, this code will be significantly streamlined:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
    }
Of course, some code may want to just detect that an error was thrown, but other code may need to check that the details of the thrown error are correct. We can take advantage of Swift's trailing closure syntax to enable this, by passing the thrown error (if any) to a closure that can itself contain assertions:

    XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
        guard let vendingError = error as? VendingMachineError else {
            XCTFail("Unexpected type of error thrown: \(error)")
            return
        }
        
        XCTAssertEquals(vendingError.item, "Candy Bar")
        XCTAssertEquals(vendingError.price, 5)
        XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
    }
This lets a developer very concisely describe an error condition for their code, in whatever level of detail they desire.

Detailed design
The design of each of the above components is slightly different, based on the functionality provided.

Tests That Throw

In order to enable test methods to throw an error, we will need to update XCTest to support test methods with a () throws -> Void signature in addition to test methods with a () -> Voidsignature as it already supports.

We will need to ensure tests that do throw an error have that error caught, and that it registers an unexpected failure.

Assertions That Throw

In order to allow assertions to throw an exception, we will need to enhance our existing assertions' @autoclosure expression parameters to add throws to their signature.

Because Swift defines a closure that can throw an error to be a proper supertype of a closure that does not, this will not result in a combinatorial explosion of assertion overrides, and will let developers naturally write code that may throw an error within an assertion.

We will treat any error thrown from within an assertion expression as an unexpected failure because while all assertions represent a test for some form of failure, they're not specifically checking for a thrown error.

The "Throws Error" Assertion

To write tests for code that throws error, we will add a new assertion function to XCTest with the following prototype:

public func XCTAssertThrowsError(
    @autoclosure expression: () throws -> Void,
                  _ message: String = "",
                       file: StaticString = __FILE__,
                       line: UInt = __LINE__,
             _ errorHandler: (error: ErrorType) -> Void = { _ in })
Rather than treat an error thrown from its expression as a failure, this will treat the lack of an error thrown from its expression as an expected failure.

Furthermore, so long as an error is thrown, the error will be passed to the errorHandler block passed as a trailing closure, where the developer may make further assertions against it.

In both cases, the new assertion function is generic on an ErrorType in order to ensure that little to no casting will be required in the trailing closure.

Impact on existing code
There should be little impact on existing test code because we are only adding features and API, not changing existing features or API.

All existing tests should continue to work as implemented, and can easily adopt the new conventions we're making available to become more concise and intention-revealing with respect to their error handling as shown above.

Alternatives considered
We considered asking developers continue using XCTest as-is, and encouraging them to use Swift's native error handling to both suppress and check the validity of errors. We also considered adding additional ways of registering failures when doing this, so that developers could register unexpected failures themselves.

While this would result in developers using the language the same way in their tests as in their functional code, this would also result in much more verbose tests. We rejected this approach because such verbosity can be a significant obstacle to testing.

Making it quick and clean to write tests for error handling could also encourage developers to implement error handling in their code as they need it, rather than to try to work around the feature because of any perceived difficulty in testing.

We considered adding the ability to check that a specific error was thrown in XCTAssertThrowsError, but this would require the ErrorType passed to also conform to Equatable, which is also unnecessary given that this can easily be checked in a trailing closure if desired. (In some cases a developer may just want to ensure an error is thrown rather than a specific error is thrown.)

We explicitly chose not to offer a comprehensive suite of DoesNotThrowError assertions for XCTest in Swift, though we do offer such DoesNotThrow assertions for XCTest in Objective-C. We feel these are of limited utility given that our plan is for all assertions (except XCTAssertThrowsError) to treat any thrown error as a failure.

We explicitly chose not to offer any additional support for Objective-C exceptions beyond what we already provide: In the Xcode implementation of XCTest, an Objective-C exception that occurs within one of our existing assertions or tests will result in a test failure; doing more than this is not practical given that it's possible to neither catch and handle nor generate an Objective-C exception in Swift.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Dmitri Gribenko) #3

I have significant concerns about doing this. Consider the following code:

var foo = Foo()
do {
    XCTAssertEqual(try foo.doSomething(), 42)
} catch {
    XCTAssertEqual(foo.success, false)
}

Adding ‘throws’ to the autoclosures will:

(1) change meaning of existing tests (the example above used to proceed to
the catch block in case of errors, and will start failing with this change),

(2) hijacks ‘try’ and defeats its purpose — being able to understand the
control flow. Developers know that if they see a ‘try’, it should match
with either a do-catch, or be used in a throwing function. Adding this API
makes the code misleading.

Note that although (2) applies to the XCTAssertThrowsError() API, because
the name of that API makes it clear it is doing something special with
error handling, I’m not concerned about it. But adding error handling to
regular assertion functions has the potential to create misleading code.

Changing the way control flow works breaks one of the basic language
features — substitutability:

let bar1 = try bar()
foo(bar1)

should be equivalent to:

foo(try bar())

Dmitri

···

On Sat, Jan 9, 2016 at 6:58 PM, Chris Hanson via swift-evolution < swift-evolution@swift.org> wrote:

Allowing Test Assertions to Throw Errors

We can also allow the @autoclosure expression that is passed into an
assertion to throw an error, and treat that error as an unexpected failure
(since the code is being invoked in an assertion that isn't directly
related to error handling). For example:

func testVendingMultipleItemsWithSufficientFunds() {
    vendingMachine.deposit(10)
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")}

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/


(Lily Ballard) #4

Very strong +1 to the entire proposal. I've been meaning to submit something like this for a while, though not as comprehensive as what you have here. In particular, making all the @autoclosure-accepting functions take a throwing closure and reporting errors as failures solves my biggest annoyance with XCTest. Making the test functions themselves also be able to throw is a neat idea too. And the trailing closure argument to XCTAssertThrowsError is a good idea.

The only change I'd like to make to this proposal is to XCTAssertThrowsError. Instead of having it take an autoclosure that returns Void, it should just be generic so it can take a closure that returns any type. That gets rid of the need for `_ =` in the argument, and XCTest could even report the actual return value in the event that the closure does not, in fact, throw an error.

I also wonder if maybe there's some utility in adding an overload to XCTAssertThrowsError that accepts any Equatable ErrorType as the second parameter, to simplify the cases where you do want to test against some specific error that happens to be Equatable. Although I'm not sure how likely it is for any given ErrorType to also be Equatable. The overload might look like

public func XCTAssertThrowsError<T, E: ErrorType where E: Equatable>(
    @autoclosure expression: () throws -> T,
    _ message: String = "",
    file: StaticString = __FILE__,
    line: UInt = __LINE__,
    _ expectedError: E
) {
    XCTAssertThrowsError(expression, message, file: file, line: line) { error in
        if let error = error as? E {
            if error != expectedError {
                XCTFail("Expected error \(expectedError), found error \(error)")
            }
        } else {
            XCTFail("Unexpected type of error thrown: \(error)")
        }
    }
}

-Kevin Ballard

···

On Sat, Jan 9, 2016, at 06:58 PM, Chris Hanson via swift-evolution wrote:

We’d like feedback on a proposed design for adding support for Swift error handling to XCTest, attached below. I’ll mostly let the proposal speak for itself, but there are three components to it: Allowing test methods to throw errors, allowing the expressions evaluated by assertions to throw errors, and adding an assertion for checking error handling.

We’d love to hear your feedback. We’re particularly interested in some feedback on the idea of allowing the expressions evaluated by assertions to throw errors; it’s generated some debate because it results in writing test code slightly differently than other code that makes use of Swift error handling, so any thoughts on it would be particularly appreciated.


(Jerome Duquennoy) #5

Hi everyone,

I did encounter this problem too : I find the error handling model of swift pretty nice, so I use it a lot. And of course, I need to unit test those error cases behaviours of my code.

I don’t think test methods that throws does address the problem : if we do so, we will have only one information, an error was thrown by some line of code.
But in the test itself there are very different types of code :
- the setup
- the execution (the call to the method that is being unit-tested)
- the validation code

I need to be able to handle errors differently depending on whether it is thrown in the setup or in the execution.
In the first case, the problem is that the test cannot run correctly. It will fail, but not because the tested behaviour failed. I treat those errors with a try!, and if it fails, the test is reported in “error” state by my CI system (crash of the test)
In the second case, the tested behaviour is invalide. I need it to fail with an assertion failure, so that the test is reported in “failed” state by my CI system

So on my side, I go for solution 3, but I would also add an assert that no error is thrown. Consider for exemple a case where you have an integer input parameter with a range, but the contract of your API is that if the range is exceeded, the value will be restricted but no error will be thrown. That behaviour should be unit tested, and the assertNoThrow can be useful.

Here is the code I have for those two :
extension XCTestCase {
  
  func XCTAssertThrows(file: String = __FILE__, line: UInt = __LINE__, _ closure:() throws -> Void) -> ErrorType? {
    do {
      try closure()
      XCTFail("Closure did not throw an error", file: file, line: line)
    } catch {
      return error
    }
    return nil
  }
  
  func XCTAssertNoThrow<T>(file: String = __FILE__, line: UInt = __LINE__, _ closure:() throws -> T) -> T? {
    do {
      return try closure()
    } catch let error {
      XCTFail("Closure throw unexpected error \(error)", file: file, line: line)
    }
    return nil;
  }
  
}

This is very close to what you have. Note that the XCTAssertThrows does not return a return value of the block : as the test expects it will throw, there is not reason for it to use a return value, that would only be returned if no error is thrown.
It does return the received error, that can later be checked, in a classical setup -> execute -> validate flow.

The test code looks like this :
  func testSample() {
    // Setup
    let test = TestClass()
    let error: ErrorType?

    // Execute
    error = XCTAssertThrows{
      try test.codeThatThrowAnError()
    }
    
    // Validation
    XCTAssertEqual(error.someProperty, "someValue")
  }

Jerome

···

On 10 Jan 2016, at 03:58, Chris Hanson via swift-evolution <swift-evolution@swift.org> wrote:

We’d like feedback on a proposed design for adding support for Swift error handling to XCTest, attached below. I’ll mostly let the proposal speak for itself, but there are three components to it: Allowing test methods to throw errors, allowing the expressions evaluated by assertions to throw errors, and adding an assertion for checking error handling.

We’d love to hear your feedback. We’re particularly interested in some feedback on the idea of allowing the expressions evaluated by assertions to throw errors; it’s generated some debate because it results in writing test code slightly differently than other code that makes use of Swift error handling, so any thoughts on it would be particularly appreciated.

  -- Chris Hanson (chanson@apple.com <mailto:chanson@apple.com>)

XCTest Support for Swift Error Handling
Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
Author(s): Chris Hanson <https://github.com/eschaton>
Status: Review
Review manager: TBD
Introduction
Swift 2 introduced a new error handling mechanism that, for completeness, needs to be accommodated by our testing frameworks. Right now, to write tests that involve methods that may throw an error, a developer needs to incorporate significant boilerplate into their test. We should move this into the framework in several ways, so tests of code that interacts with Swift error handling is concise and intention-revealing.

Motivation
Currently, if a developer wants to use a call that may throw an error in a test, they need to use Swift's do..catch construct in their test because tests are not themselves allowed to throw errors.

As an example, a vending machine object that has had insufficient funds deposited may throw an error if asked to vend an item. A test for that situation could reasonably use the do..catchconstruct to check that this occurs as expected. However, that means all other tests also need to use either a do..catch or try! construct — and the failure of a try! is catastrophic, so do..catch would be preferred simply for better reporting within tests.

func testVendingOneItem() {
    do {
        vendingMachine.deposit(5)
        let item = try vendingMachine.vend(row: 1, column: 1)
        XCTAssertEqual(item, "Candy Bar")
    } catch {
        XCTFail("Unexpected failure: \(error)")
    }
}
If the implementation of VendingMachine.vend(row:column:) changes during development such that it throws an error in this situation, the test will fail as it should.

One other downside of the above is that a failure caught this way will be reported as an expected failure, which would normally be a failure for which XCTest is explicitly testing via an assertion. This failure should ideally be treated as an unexpected failure, as it's not one that's anticipated in the execution of the test.

In addition, tests do not currently support throwing an error from within an assertion, requiring any code that throws an error to be invoked outside the assertion itself using the same techniques described above.

Finally, since Swift error handling is a general mechanism that developers should be implementing in their own applications and frameworks, we need to make it straightforward to write tests that ensure code that implements error handling does so correctly.

Proposed solution
I propose several related solutions to this issue:

Allow test methods to throw errors.
Allow test assertion expressions to throw errors.
Add an assertion for checking errors.
These solutions combine to make writing tests that involve thrown errors much more succinct.

Allowing Test Methods to Throw Errors

First, we can allow test methods to throw errors if desired, thus allowing the do..catch construct to be omitted when the test isn't directly checking error handling. This makes the code a developer writes when they're not explicitly trying to test error handling much cleaner.

Moving the handling of errors thrown by tests into XCTest itself also ensures they can be treated as unexpected failures, since the mechanism to do so is currently private to the framework.

With this, the test from the previous section can become:

func testVendingOneItem() throws {
    vendingMachine.deposit(5)
    let item = try vendingMachine.vend(row: 1, column: 1)
    XCTAssertEqual(item, "Candy Bar")
}
This shows much more directly that the test is intended to check a specific non-error case, and that the developer is relying on the framework to handle unexpected errors.

Allowing Test Assertions to Throw Errors

We can also allow the @autoclosure expression that is passed into an assertion to throw an error, and treat that error as an unexpected failure (since the code is being invoked in an assertion that isn't directly related to error handling). For example:

func testVendingMultipleItemsWithSufficientFunds() {
    vendingMachine.deposit(10)
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")
}
This can eliminate otherwise-dangerous uses of try! and streamline code that needs to make multiple assertions in a row.

Adding a "Throws Error" Assertion

In order to test code that throws an error, it would be useful to have an assertion that expects an error to be thrown in a particular case. Right now a developer writing code to test that an error is thrown has to test that error themselves:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        var vendingFailed = false
        do {
            _ = try vendingMachine.vend(row: 1, column: 1))
        } catch {
            vendingFailed = true
        }
        XCTAssert(vendingFailed)
    }
If we add an assertion that specifically checks whether an error was thrown, this code will be significantly streamlined:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
    }
Of course, some code may want to just detect that an error was thrown, but other code may need to check that the details of the thrown error are correct. We can take advantage of Swift's trailing closure syntax to enable this, by passing the thrown error (if any) to a closure that can itself contain assertions:

    XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
        guard let vendingError = error as? VendingMachineError else {
            XCTFail("Unexpected type of error thrown: \(error)")
            return
        }
        
        XCTAssertEquals(vendingError.item, "Candy Bar")
        XCTAssertEquals(vendingError.price, 5)
        XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
    }
This lets a developer very concisely describe an error condition for their code, in whatever level of detail they desire.

Detailed design
The design of each of the above components is slightly different, based on the functionality provided.

Tests That Throw

In order to enable test methods to throw an error, we will need to update XCTest to support test methods with a () throws -> Void signature in addition to test methods with a () -> Voidsignature as it already supports.

We will need to ensure tests that do throw an error have that error caught, and that it registers an unexpected failure.

Assertions That Throw

In order to allow assertions to throw an exception, we will need to enhance our existing assertions' @autoclosure expression parameters to add throws to their signature.

Because Swift defines a closure that can throw an error to be a proper supertype of a closure that does not, this will not result in a combinatorial explosion of assertion overrides, and will let developers naturally write code that may throw an error within an assertion.

We will treat any error thrown from within an assertion expression as an unexpected failure because while all assertions represent a test for some form of failure, they're not specifically checking for a thrown error.

The "Throws Error" Assertion

To write tests for code that throws error, we will add a new assertion function to XCTest with the following prototype:

public func XCTAssertThrowsError(
    @autoclosure expression: () throws -> Void,
                  _ message: String = "",
                       file: StaticString = __FILE__,
                       line: UInt = __LINE__,
             _ errorHandler: (error: ErrorType) -> Void = { _ in })
Rather than treat an error thrown from its expression as a failure, this will treat the lack of an error thrown from its expression as an expected failure.

Furthermore, so long as an error is thrown, the error will be passed to the errorHandler block passed as a trailing closure, where the developer may make further assertions against it.

In both cases, the new assertion function is generic on an ErrorType in order to ensure that little to no casting will be required in the trailing closure.

Impact on existing code
There should be little impact on existing test code because we are only adding features and API, not changing existing features or API.

All existing tests should continue to work as implemented, and can easily adopt the new conventions we're making available to become more concise and intention-revealing with respect to their error handling as shown above.

Alternatives considered
We considered asking developers continue using XCTest as-is, and encouraging them to use Swift's native error handling to both suppress and check the validity of errors. We also considered adding additional ways of registering failures when doing this, so that developers could register unexpected failures themselves.

While this would result in developers using the language the same way in their tests as in their functional code, this would also result in much more verbose tests. We rejected this approach because such verbosity can be a significant obstacle to testing.

Making it quick and clean to write tests for error handling could also encourage developers to implement error handling in their code as they need it, rather than to try to work around the feature because of any perceived difficulty in testing.

We considered adding the ability to check that a specific error was thrown in XCTAssertThrowsError, but this would require the ErrorType passed to also conform to Equatable, which is also unnecessary given that this can easily be checked in a trailing closure if desired. (In some cases a developer may just want to ensure an error is thrown rather than a specific error is thrown.)

We explicitly chose not to offer a comprehensive suite of DoesNotThrowError assertions for XCTest in Swift, though we do offer such DoesNotThrow assertions for XCTest in Objective-C. We feel these are of limited utility given that our plan is for all assertions (except XCTAssertThrowsError) to treat any thrown error as a failure.

We explicitly chose not to offer any additional support for Objective-C exceptions beyond what we already provide: In the Xcode implementation of XCTest, an Objective-C exception that occurs within one of our existing assertions or tests will result in a test failure; doing more than this is not practical given that it's possible to neither catch and handle nor generate an Objective-C exception in Swift.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Chris Hanson) #6

Thanks, everyone, for all of the feedback.

I sent this proposal out as more of a “heads up” that we were planning to make these changes and get feedback in advance of that, so we won’t be taking this through the full proposal process. That said, here’s what we’re going to be doing:

1. We’re going to let test methods throw, and treat those as unexpected failures, as planned.

2. We’re going to let test assertion expressions throw, and treat those as unexpected failures, as planned. We recognize that this will result in slightly different style in tests versus in regular code, but the convenience and reduction in boilerplate makes this worthwhile.

3. We’re going to add the XCTAssertThrowsError assertion, incorporating Kevin Ballard’s suggestion for how to avoid the “_ = blah()” in its expression.

4. We’re not going to add the ability for XCTAssertThrowsError to check that a specific error was thrown. This could definitely be useful but would require the function to take an ErrorType instance that also conforms to Equatable. We need to think about that some more before making such a change. Fortunately this sort of addition will be straightforward to put in via an overload or default argument, so if we choose to do it later, it shouldn’t break any code that starts to adopt XCTAssertThrowsError.

We’re going to start landing these changes today, so thanks again for all of your attention and feedback!

  -- Chris


(James Campbell) #7

I would love it if we could do a full review of XCtest in general. As there are other things it could help with I.r mocking or allowing us to express tests in a BDD way

···

Sent from my iPhone

On 10 Jan 2016, at 10:29, Drew Crawford via swift-evolution <swift-evolution@swift.org> wrote:

I have on the order of ~700 tests in XCTest in Swift-language projects. I'm considering migrating away from XCTest, although not over this issue.

This proposal IMO addresses an important problem, but I am not convinced it actually solves it. #2 & #3 are basically sound API designs. It is a mystery to me why #3 "generated some debate" as this is a feature I already implement manually, but I can't address unknown concerns. I can tell you I implement this, and nothing terrible has happened to me so far.

#1 I would not use. The rest of this comment explains why.

Currently I write tests about like this:

try! hopefullyNothingBad()

Now this is "bad" because it "crashes" but that's (big sigh) actually "good" because the debugger stops and/or I get a crash report that identifies at least "some" line where something bad definitely happened.

Now that is not everything I want to know–I want someone to tell me a story of how this error was created deep down in the bowels of my application, how it spent its kindergarten years in the models layer before being passed into a controller where it was rethrown onto a different dispatch queue and finally ended up in my unit test–but we can't have everything. So I settle for collecting a line number from the test case and then going hunting by hand.

When the test function throws we no longer even find out a line number in the test case anymore, because the error is passed into XCTest and the information is lost. We have just the name of the test case (I assume; the proposal is silent on this issue, but that's the only way I can think of to implement it), and some of my tests are pretty long. So, that makes it even harder to track down.

This sounds like a small thing but my test coverage is so thorough on mature projects that mostly what I turn up are heisenbugs that reproduce with 2% probability. So getting the report from the CI that has the most possible detail is critical, because if the report is not thorough enough for you to guess the bug, too bad, because that's all the information you get and the bug is not reproducible.

For that reason, I won't use #1. I hesitate about whether to call it bad idea altogether, or whether it's just not to my taste. My sense is it's probably somewhere in the middle of those two poles.

I would use #2 and #3, assuming that I don't first migrate out to a non-XCTest framework.

On Jan 9, 2016, at 8:58 PM, Chris Hanson via swift-evolution <swift-evolution@swift.org> wrote:

We’d like feedback on a proposed design for adding support for Swift error handling to XCTest, attached below. I’ll mostly let the proposal speak for itself, but there are three components to it: Allowing test methods to throw errors, allowing the expressions evaluated by assertions to throw errors, and adding an assertion for checking error handling.

We’d love to hear your feedback. We’re particularly interested in some feedback on the idea of allowing the expressions evaluated by assertions to throw errors; it’s generated some debate because it results in writing test code slightly differently than other code that makes use of Swift error handling, so any thoughts on it would be particularly appreciated.

  -- Chris Hanson (chanson@apple.com)

XCTest Support for Swift Error Handling
Proposal: SE-NNNN
Author(s): Chris Hanson
Status: Review
Review manager: TBD
Introduction
Swift 2 introduced a new error handling mechanism that, for completeness, needs to be accommodated by our testing frameworks. Right now, to write tests that involve methods that may throw an error, a developer needs to incorporate significant boilerplate into their test. We should move this into the framework in several ways, so tests of code that interacts with Swift error handling is concise and intention-revealing.

Motivation
Currently, if a developer wants to use a call that may throw an error in a test, they need to use Swift's do..catch construct in their test because tests are not themselves allowed to throw errors.

As an example, a vending machine object that has had insufficient funds deposited may throw an error if asked to vend an item. A test for that situation could reasonably use the do..catchconstruct to check that this occurs as expected. However, that means all other tests also need to use either a do..catch or try! construct — and the failure of a try! is catastrophic, so do..catch would be preferred simply for better reporting within tests.

func testVendingOneItem() {
    do {
        vendingMachine.deposit(5)
        let item = try vendingMachine.vend(row: 1, column: 1)
        XCTAssertEqual(item, "Candy Bar")
    } catch {
        XCTFail("Unexpected failure: \(error)")
    }
}
If the implementation of VendingMachine.vend(row:column:) changes during development such that it throws an error in this situation, the test will fail as it should.

One other downside of the above is that a failure caught this way will be reported as an expected failure, which would normally be a failure for which XCTest is explicitly testing via an assertion. This failure should ideally be treated as an unexpected failure, as it's not one that's anticipated in the execution of the test.

In addition, tests do not currently support throwing an error from within an assertion, requiring any code that throws an error to be invoked outside the assertion itself using the same techniques described above.

Finally, since Swift error handling is a general mechanism that developers should be implementing in their own applications and frameworks, we need to make it straightforward to write tests that ensure code that implements error handling does so correctly.

Proposed solution
I propose several related solutions to this issue:

Allow test methods to throw errors.
Allow test assertion expressions to throw errors.
Add an assertion for checking errors.
These solutions combine to make writing tests that involve thrown errors much more succinct.

Allowing Test Methods to Throw Errors

First, we can allow test methods to throw errors if desired, thus allowing the do..catch construct to be omitted when the test isn't directly checking error handling. This makes the code a developer writes when they're not explicitly trying to test error handling much cleaner.

Moving the handling of errors thrown by tests into XCTest itself also ensures they can be treated as unexpected failures, since the mechanism to do so is currently private to the framework.

With this, the test from the previous section can become:

func testVendingOneItem() throws {
    vendingMachine.deposit(5)
    let item = try vendingMachine.vend(row: 1, column: 1)
    XCTAssertEqual(item, "Candy Bar")
}
This shows much more directly that the test is intended to check a specific non-error case, and that the developer is relying on the framework to handle unexpected errors.

Allowing Test Assertions to Throw Errors

We can also allow the @autoclosure expression that is passed into an assertion to throw an error, and treat that error as an unexpected failure (since the code is being invoked in an assertion that isn't directly related to error handling). For example:

func testVendingMultipleItemsWithSufficientFunds() {
    vendingMachine.deposit(10)
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")
}
This can eliminate otherwise-dangerous uses of try! and streamline code that needs to make multiple assertions in a row.

Adding a "Throws Error" Assertion

In order to test code that throws an error, it would be useful to have an assertion that expects an error to be thrown in a particular case. Right now a developer writing code to test that an error is thrown has to test that error themselves:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        var vendingFailed = false
        do {
            _ = try vendingMachine.vend(row: 1, column: 1))
        } catch {
            vendingFailed = true
        }
        XCTAssert(vendingFailed)
    }
If we add an assertion that specifically checks whether an error was thrown, this code will be significantly streamlined:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
    }
Of course, some code may want to just detect that an error was thrown, but other code may need to check that the details of the thrown error are correct. We can take advantage of Swift's trailing closure syntax to enable this, by passing the thrown error (if any) to a closure that can itself contain assertions:

    XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
        guard let vendingError = error as? VendingMachineError else {
            XCTFail("Unexpected type of error thrown: \(error)")
            return
        }
        
        XCTAssertEquals(vendingError.item, "Candy Bar")
        XCTAssertEquals(vendingError.price, 5)
        XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
    }
This lets a developer very concisely describe an error condition for their code, in whatever level of detail they desire.

Detailed design
The design of each of the above components is slightly different, based on the functionality provided.

Tests That Throw

In order to enable test methods to throw an error, we will need to update XCTest to support test methods with a () throws -> Void signature in addition to test methods with a () -> Voidsignature as it already supports.

We will need to ensure tests that do throw an error have that error caught, and that it registers an unexpected failure.

Assertions That Throw

In order to allow assertions to throw an exception, we will need to enhance our existing assertions' @autoclosure expression parameters to add throws to their signature.

Because Swift defines a closure that can throw an error to be a proper supertype of a closure that does not, this will not result in a combinatorial explosion of assertion overrides, and will let developers naturally write code that may throw an error within an assertion.

We will treat any error thrown from within an assertion expression as an unexpected failure because while all assertions represent a test for some form of failure, they're not specifically checking for a thrown error.

The "Throws Error" Assertion

To write tests for code that throws error, we will add a new assertion function to XCTest with the following prototype:

public func XCTAssertThrowsError(
    @autoclosure expression: () throws -> Void,
                  _ message: String = "",
                       file: StaticString = __FILE__,
                       line: UInt = __LINE__,
             _ errorHandler: (error: ErrorType) -> Void = { _ in })
Rather than treat an error thrown from its expression as a failure, this will treat the lack of an error thrown from its expression as an expected failure.

Furthermore, so long as an error is thrown, the error will be passed to the errorHandler block passed as a trailing closure, where the developer may make further assertions against it.

In both cases, the new assertion function is generic on an ErrorType in order to ensure that little to no casting will be required in the trailing closure.

Impact on existing code
There should be little impact on existing test code because we are only adding features and API, not changing existing features or API.

All existing tests should continue to work as implemented, and can easily adopt the new conventions we're making available to become more concise and intention-revealing with respect to their error handling as shown above.

Alternatives considered
We considered asking developers continue using XCTest as-is, and encouraging them to use Swift's native error handling to both suppress and check the validity of errors. We also considered adding additional ways of registering failures when doing this, so that developers could register unexpected failures themselves.

While this would result in developers using the language the same way in their tests as in their functional code, this would also result in much more verbose tests. We rejected this approach because such verbosity can be a significant obstacle to testing.

Making it quick and clean to write tests for error handling could also encourage developers to implement error handling in their code as they need it, rather than to try to work around the feature because of any perceived difficulty in testing.

We considered adding the ability to check that a specific error was thrown in XCTAssertThrowsError, but this would require the ErrorType passed to also conform to Equatable, which is also unnecessary given that this can easily be checked in a trailing closure if desired. (In some cases a developer may just want to ensure an error is thrown rather than a specific error is thrown.)

We explicitly chose not to offer a comprehensive suite of DoesNotThrowError assertions for XCTest in Swift, though we do offer such DoesNotThrow assertions for XCTest in Objective-C. We feel these are of limited utility given that our plan is for all assertions (except XCTAssertThrowsError) to treat any thrown error as a failure.

We explicitly chose not to offer any additional support for Objective-C exceptions beyond what we already provide: In the Xcode implementation of XCTest, an Objective-C exception that occurs within one of our existing assertions or tests will result in a test failure; doing more than this is not practical given that it's possible to neither catch and handle nor generate an Objective-C exception in Swift.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Lily Ballard) #8

On Sun, Jan 10, 2016, at 11:18 AM, Dmitri Gribenko via swift-evolution wrote:0

I have significant concerns about doing this. Consider the following code:

var foo = Foo() do { XCTAssertEqual(try foo.doSomething(), 42) }
catch { XCTAssertEqual(foo.success, false) }

Adding ‘throws’ to the autoclosures will:

(1) change meaning of existing tests (the example above used to
    proceed to the catch block in case of errors, and will start
    failing with this change),

Did you actually test this code? Because this code *does not compile
today*. You cannot pass a throwing expression to an @autoclosure that
takes a non-throwing function. The specific error you get is

error: call can throw, but it is executed in a non-throwing autoclosure

(2) hijacks ‘try’ and defeats its purpose — being able to understand
    the control flow. Developers know that if they see a ‘try’, it
    should match with either a do-catch, or be used in a throwing
    function. Adding this API makes the code misleading.

Or use it in a throwing @autoclosure.

Note that although (2) applies to the XCTAssertThrowsError() API,
because the name of that API makes it clear it is doing something
special with error handling, I’m not concerned about it. But adding
error handling to regular assertion functions has the potential to
create misleading code.

Changing the way control flow works breaks one of the basic language
features — substitutability:

let bar1 = try bar() foo(bar1)

should be equivalent to:

foo(try bar())

@autoclosure already breaks that. Adding error handling to the
@autoclosure doesn't change anything. In fact, breaking that
substitutability is basically the whole reason for @autoclosure to
exist.

-Kevin Ballard


(Brian Gesiak) #9

Chris: Thank you for also sending this to the swift-corelibs-dev mailing list!

I am in favor of the proposed changes; I think they are a great step
forward for testing throwing functions in Swift. That being said, I've
rarely used Swift error handling in my own applications--until now I
have preferred Result<T, U> (see:
http://www.sunsetlakesoftware.com/2015/06/12/swift-2-error-handling-practice).

I agree with Kevin Ballard's proposed amendment to the
`XCTAssertThrowsError` parameter type. As-is, I imagine many users of
the proposed API would be confused as to why they need to type
`XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column:
1))`.

Because Swift defines a closure that can throw an error to be a proper supertype of a closure that does not

Chris: How fortuitous! Glad to hear that we won't have to append a
`throws` to all of our test functions.

I'm considering migrating away from XCTest, although not over this issue.

Drew: I'd be very interested to hear why. If it's something you feel
can be addressed in swift-corelibs-xctest, please email the
swift-corelibs-dev mailing list with your concerns. Otherwise I would
be glad if you could send me an email.

- Brian Gesiak

···

On Sun, Jan 10, 2016 at 6:08 PM, Kevin Ballard via swift-evolution <swift-evolution@swift.org> wrote:

On Sat, Jan 9, 2016, at 06:58 PM, Chris Hanson via swift-evolution wrote:

We’d like feedback on a proposed design for adding support for Swift error
handling to XCTest, attached below. I’ll mostly let the proposal speak for
itself, but there are three components to it: Allowing test methods to throw
errors, allowing the expressions evaluated by assertions to throw errors,
and adding an assertion for checking error handling.

We’d love to hear your feedback. We’re particularly interested in some
feedback on the idea of allowing the expressions evaluated by assertions to
throw errors; it’s generated some debate because it results in writing test
code slightly differently than other code that makes use of Swift error
handling, so any thoughts on it would be particularly appreciated.

Very strong +1 to the entire proposal. I've been meaning to submit something
like this for a while, though not as comprehensive as what you have here. In
particular, making all the @autoclosure-accepting functions take a throwing
closure and reporting errors as failures solves my biggest annoyance with
XCTest. Making the test functions themselves also be able to throw is a neat
idea too. And the trailing closure argument to XCTAssertThrowsError is a
good idea.

The only change I'd like to make to this proposal is to
XCTAssertThrowsError. Instead of having it take an autoclosure that returns
Void, it should just be generic so it can take a closure that returns any
type. That gets rid of the need for `_ =` in the argument, and XCTest could
even report the actual return value in the event that the closure does not,
in fact, throw an error.

I also wonder if maybe there's some utility in adding an overload to
XCTAssertThrowsError that accepts any Equatable ErrorType as the second
parameter, to simplify the cases where you do want to test against some
specific error that happens to be Equatable. Although I'm not sure how
likely it is for any given ErrorType to also be Equatable. The overload
might look like

public func XCTAssertThrowsError<T, E: ErrorType where E: Equatable>(
    @autoclosure expression: () throws -> T,
    _ message: String = "",
    file: StaticString = __FILE__,
    line: UInt = __LINE__,
    _ expectedError: E
) {
    XCTAssertThrowsError(expression, message, file: file, line: line) {
error in
        if let error = error as? E {
            if error != expectedError {
                XCTFail("Expected error \(expectedError), found error
\(error)")
            }
        } else {
            XCTFail("Unexpected type of error thrown: \(error)")
        }
    }
}

-Kevin Ballard

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Philippe Hausler) #10

I think I am in full agreement with Dmitri on this: The XCTAssertThrowsError should be the only one that interacts with errors and that others should not try to hijack the try.

But super-big +1 on the XCTAssertThrowsError being in place to verify error cases; swift-corelibs-foundation is missing a lot of testing of error cases and it would be nice to have an effort to start verifying those (and their structures)

···

On Jan 10, 2016, at 11:18 AM, Dmitri Gribenko via swift-evolution <swift-evolution@swift.org> wrote:

On Sat, Jan 9, 2016 at 6:58 PM, Chris Hanson via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
Allowing Test Assertions to Throw Errors

We can also allow the @autoclosure expression that is passed into an assertion to throw an error, and treat that error as an unexpected failure (since the code is being invoked in an assertion that isn't directly related to error handling). For example:

func testVendingMultipleItemsWithSufficientFunds() {
    vendingMachine.deposit(10)
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")
}

I have significant concerns about doing this. Consider the following code:

var foo = Foo()
do {
    XCTAssertEqual(try foo.doSomething(), 42)
} catch {
    XCTAssertEqual(foo.success, false)
}

Adding ‘throws’ to the autoclosures will:

(1) change meaning of existing tests (the example above used to proceed to the catch block in case of errors, and will start failing with this change),

(2) hijacks ‘try’ and defeats its purpose — being able to understand the control flow. Developers know that if they see a ‘try’, it should match with either a do-catch, or be used in a throwing function. Adding this API makes the code misleading.

Note that although (2) applies to the XCTAssertThrowsError() API, because the name of that API makes it clear it is doing something special with error handling, I’m not concerned about it. But adding error handling to regular assertion functions has the potential to create misleading code.

Changing the way control flow works breaks one of the basic language features — substitutability:

let bar1 = try bar()
foo(bar1)

should be equivalent to:

foo(try bar())

Dmitri

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com <mailto:gribozavr@gmail.com>>*/
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Brian Gesiak) #11

Loving those commits! Especially the corresponding tests :slight_smile:

Can you explain the decision to not go through the review process? I think
it's a prudent decision and I'm supportive of it, but I was under the
impression that additional APIs should be proposed and reviewed. Is the
rationale that swift-corelibs-xctest is too immature to justify going
through proposals?

- Brian Gesiak

···

On Wed, Jan 13, 2016 at 4:52 PM, Chris Hanson via swift-evolution < swift-evolution@swift.org> wrote:

Thanks, everyone, for all of the feedback.

I sent this proposal out as more of a “heads up” that we were planning to
make these changes and get feedback in advance of that, so we won’t be
taking this through the full proposal process. That said, here’s what we’re
going to be doing:

1. We’re going to let test methods throw, and treat those as unexpected
failures, as planned.

2. We’re going to let test assertion expressions throw, and treat those as
unexpected failures, as planned. We recognize that this will result in
slightly different style in tests versus in regular code, but the
convenience and reduction in boilerplate makes this worthwhile.

3. We’re going to add the XCTAssertThrowsError assertion, incorporating
Kevin Ballard’s suggestion for how to avoid the “_ = blah()” in its
expression.

4. We’re *not* going to add the ability for XCTAssertThrowsError to check
that a specific error was thrown. This could definitely be useful but would
require the function to take an ErrorType instance that also conforms to
Equatable. We need to think about that some more before making such a
change. Fortunately this sort of addition will be straightforward to put in
via an overload or default argument, so if we choose to do it later, it
shouldn’t break any code that starts to adopt XCTAssertThrowsError.

We’re going to start landing these changes today, so thanks again for all
of your attention and feedback!

  -- Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jeremy W. Sherman) #12

I’m excited to see these improvements arrive in XCTest. Up till now, I’ve been using helpers like:

expectingThrowsAnyError(“message goes here”) {
    // do some set-up for the call
    let result = try CallSomeThrowingMethod()
    XCTAssertEqual(result.foo, expectedFoo)
}

Do you see any way to address the testing gap left by not being able to catch failing preconditions? Preconditions are significant elements of the developer-facing API, and it is frustrating to be unable to drive these requirements out - and document them - using tests.

Concretely, this means I cannot write a test like func testProvidingNegativeCountViolatesPrecondition() { … } that actually tests this element of the API specification.

Thanks,

···

--
Jeremy W. Sherman
https://jeremywsherman.com/

El 13 ene 2016, a las 19:52, Chris Hanson via swift-evolution <swift-evolution@swift.org> escribió:

Thanks, everyone, for all of the feedback.

I sent this proposal out as more of a “heads up” that we were planning to make these changes and get feedback in advance of that, so we won’t be taking this through the full proposal process. That said, here’s what we’re going to be doing:

1. We’re going to let test methods throw, and treat those as unexpected failures, as planned.

2. We’re going to let test assertion expressions throw, and treat those as unexpected failures, as planned. We recognize that this will result in slightly different style in tests versus in regular code, but the convenience and reduction in boilerplate makes this worthwhile.

3. We’re going to add the XCTAssertThrowsError assertion, incorporating Kevin Ballard’s suggestion for how to avoid the “_ = blah()” in its expression.

4. We’re not going to add the ability for XCTAssertThrowsError to check that a specific error was thrown. This could definitely be useful but would require the function to take an ErrorType instance that also conforms to Equatable. We need to think about that some more before making such a change. Fortunately this sort of addition will be straightforward to put in via an overload or default argument, so if we choose to do it later, it shouldn’t break any code that starts to adopt XCTAssertThrowsError.

We’re going to start landing these changes today, so thanks again for all of your attention and feedback!

  -- Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jerome Duquennoy) #13

4. We’re not going to add the ability for XCTAssertThrowsError to check that a specific error was thrown. This could definitely be useful but would require the function to take an ErrorType instance that also conforms to Equatable. We need to think about that some more before making such a change. Fortunately this sort of addition will be straightforward to put in via an overload or default argument, so if we choose to do it later, it shouldn’t break any code that starts to adopt XCTAssertThrowsError.

Wouldn't it be possible to have XCTAssertThrowsError return the thrown error, so that the dev can do it's own test (validating an error code for exemple) ?
This would be fairly easy to implement (see the code I proposed on the 11th), and very flexible (it allows any dev to validate whatever he wants in its custom error type).

Jérôme


(Ross O'Brien) #14

I've been wondering for a while, while writing unit tests, why we still
start with "func test[behaviourOfThing]() {" and not "test behaviourOfThing
{". It's not just about less typing: parsing a function for the 'test'
prefix is an Objective C holdover, and doesn't feel Swift to me, and it has
tests behaving like functions when - as illustrated in this proposal - they
should have different behaviours. It should be clearer when reading code
whether a function is a test or a helper function to make tests easier to
write.

···

On Sun, Jan 10, 2016 at 12:34 PM, James Campbell via swift-evolution < swift-evolution@swift.org> wrote:

I would love it if we could do a full review of XCtest in general. As
there are other things it could help with I.r mocking or allowing us to
express tests in a BDD way

Sent from my iPhone

On 10 Jan 2016, at 10:29, Drew Crawford via swift-evolution < > swift-evolution@swift.org> wrote:

I have on the order of ~700 tests in XCTest in Swift-language projects.
I'm considering migrating away from XCTest, although not over this issue.

This proposal IMO addresses an important problem, but I am not convinced
it actually solves it. #2 & #3 are basically sound API designs. It is a
mystery to me why #3 "generated some debate" as this is a feature I already
implement manually, but I can't address unknown concerns. I can tell you I
implement this, and nothing terrible has happened to me so far.

#1 I would not use. The rest of this comment explains why.

Currently I write tests about like this:

try! hopefullyNothingBad()

Now this is "bad" because it "crashes" but that's (big sigh) actually
"good" because the debugger stops and/or I get a crash report that
identifies at least "some" line where something bad definitely happened.

Now that is not everything I *want* to know–I *want* someone to tell me a
story of how this error was created deep down in the bowels of my
application, how it spent its kindergarten years in the models layer before
being passed into a controller where it was rethrown onto a different
dispatch queue and finally ended up in my unit test–but we can't have
everything. So I settle for collecting a line number from the test case
and then going hunting by hand.

When the test function throws we no longer even find out a line number in
the test case anymore, because the error is passed into XCTest and the
information is lost. We have just the name of the test case (I assume; the
proposal is silent on this issue, but that's the only way I can think of to
implement it), and some of my tests are pretty long. So, that makes it
even harder to track down.

This sounds like a small thing but my test coverage is so thorough on
mature projects that mostly what I turn up are heisenbugs that reproduce
with 2% probability. So getting the report from the CI that has the most
possible detail is critical, because if the report is not thorough enough
for you to guess the bug, too bad, because that's all the information you
get and the bug is not reproducible.

For that reason, I won't use #1. I hesitate about whether to call it bad
idea altogether, or whether it's just not to my taste. My sense is it's
probably somewhere in the middle of those two poles.

I would use #2 and #3, assuming that I don't first migrate out to a
non-XCTest framework.

On Jan 9, 2016, at 8:58 PM, Chris Hanson via swift-evolution < > swift-evolution@swift.org> wrote:

We’d like feedback on a proposed design for adding support for Swift error
handling to XCTest, attached below. I’ll mostly let the proposal speak for
itself, but there are three components to it: Allowing test methods to
throw errors, allowing the expressions evaluated by assertions to throw
errors, and adding an assertion for checking error handling.

We’d love to hear your feedback. We’re particularly interested in some
feedback on the idea of allowing the expressions evaluated by assertions to
throw errors; it’s generated some debate because it results in writing test
code slightly differently than other code that makes use of Swift error
handling, so any thoughts on it would be particularly appreciated.

  -- Chris Hanson (chanson@apple.com)

XCTest Support for Swift Error Handling

   - Proposal: SE-NNNN
   <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
   - Author(s): Chris Hanson <https://github.com/eschaton>
   - Status: *Review*
   - Review manager: TBD

Introduction

Swift 2 introduced a new error handling mechanism that, for completeness,
needs to be accommodated by our testing frameworks. Right now, to write
tests that involve methods that may throw an error, a developer needs to
incorporate significant boilerplate into their test. We should move this
into the framework in several ways, so tests of code that interacts with
Swift error handling is concise and intention-revealing.
Motivation

Currently, if a developer wants to use a call that may throw an error in a
test, they need to use Swift's do..catch construct in their test because
tests are not themselves allowed to throw errors.

As an example, a vending machine object that has had insufficient funds
deposited may throw an error if asked to vend an item. A test for that
situation could reasonably use the do..catchconstruct to check that this
occurs as expected. However, that means all other tests *also* need to
use either a do..catch or try! construct — and the failure of a try! is
catastrophic, so do..catch would be preferred simply for better reporting
within tests.

func testVendingOneItem() {
    do {
        vendingMachine.deposit(5)
        let item = try vendingMachine.vend(row: 1, column: 1)
        XCTAssertEqual(item, "Candy Bar")
    } catch {
        XCTFail("Unexpected failure: \(error)")
    }}

If the implementation of VendingMachine.vend(row:column:) changes during
development such that it throws an error in this situation, the test will
fail as it should.

One other downside of the above is that a failure caught this way will be
reported as an *expected failure*, which would normally be a failure for
which XCTest is explicitly testing via an assertion. This failure should
ideally be treated as an *unexpected failure*, as it's not one that's
anticipated in the execution of the test.

In addition, tests do not currently support throwing an error from within
an assertion, requiring any code that throws an error to be invoked outside
the assertion itself using the same techniques described above.

Finally, since Swift error handling is a general mechanism that developers
should be implementing in their own applications and frameworks, we need to
make it straightforward to write tests that ensure code that implements
error handling does so correctly.
Proposed solution

I propose several related solutions to this issue:

   1. Allow test methods to throw errors.
   2. Allow test assertion expressions to throw errors.
   3. Add an assertion for checking errors.

These solutions combine to make writing tests that involve thrown errors
much more succinct.
Allowing Test Methods to Throw Errors

First, we can allow test methods to throw errors if desired, thus allowing
the do..catch construct to be omitted when the test isn't directly
checking error handling. This makes the code a developer writes when
they're not explicitly trying to test error handling much cleaner.

Moving the handling of errors thrown by tests into XCTest itself also
ensures they can be treated as unexpected failures, since the mechanism to
do so is currently private to the framework.

With this, the test from the previous section can become:

func testVendingOneItem() throws {
    vendingMachine.deposit(5)
    let item = try vendingMachine.vend(row: 1, column: 1)
    XCTAssertEqual(item, "Candy Bar")}

This shows much more directly that the test is intended to check a
specific non-error case, and that the developer is relying on the framework
to handle unexpected errors.
Allowing Test Assertions to Throw Errors

We can also allow the @autoclosure expression that is passed into an
assertion to throw an error, and treat that error as an unexpected failure
(since the code is being invoked in an assertion that isn't directly
related to error handling). For example:

func testVendingMultipleItemsWithSufficientFunds() {
    vendingMachine.deposit(10)
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")}

This can eliminate otherwise-dangerous uses of try! and streamline code
that needs to make multiple assertions in a row.
Adding a "Throws Error" Assertion

In order to test code that throws an error, it would be useful to have an
assertion that expects an error to be thrown in a particular case. Right
now a developer writing code to test that an error is thrown has to test
that error themselves:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        var vendingFailed = false
        do {
            _ = try vendingMachine.vend(row: 1, column: 1))
        } catch {
            vendingFailed = true
        }
        XCTAssert(vendingFailed)
    }

If we add an assertion that specifically checks whether an error was
thrown, this code will be significantly streamlined:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
    }

Of course, some code may want to just detect that an error was thrown, but
other code may need to check that the details of the thrown error are
correct. We can take advantage of Swift's trailing closure syntax to enable
this, by passing the thrown error (if any) to a closure that can itself
contain assertions:

    XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
        guard let vendingError = error as? VendingMachineError else {
            XCTFail("Unexpected type of error thrown: \(error)")
            return
        }

        XCTAssertEquals(vendingError.item, "Candy Bar")
        XCTAssertEquals(vendingError.price, 5)
        XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
    }

This lets a developer very concisely describe an error condition for their
code, in whatever level of detail they desire.
Detailed design

The design of each of the above components is slightly different, based on
the functionality provided.
Tests That Throw

In order to enable test methods to throw an error, we will need to update
XCTest to support test methods with a () throws -> Void signature in
addition to test methods with a () -> Voidsignature as it already
supports.

We will need to ensure tests that do throw an error have that error
caught, and that it registers an unexpected failure.
Assertions That Throw

In order to allow assertions to throw an exception, we will need to
enhance our existing assertions' @autoclosure expression parameters to
add throws to their signature.

Because Swift defines a closure that can throw an error to be a proper
supertype of a closure that does not, this *will not* result in a
combinatorial explosion of assertion overrides, and will let developers
naturally write code that may throw an error within an assertion.

We will treat any error thrown from within an assertion expression as an
*unexpected* failure because while all assertions represent a test for
some form of failure, they're not specifically checking for a thrown error.
The "Throws Error" Assertion

To write tests for code that throws error, we will add a new assertion
function to XCTest with the following prototype:

public func XCTAssertThrowsError(
    @autoclosure expression: () throws -> Void,
                  _ message: String = "",
                       file: StaticString = __FILE__,
                       line: UInt = __LINE__,
             _ errorHandler: (error: ErrorType) -> Void = { _ in })

Rather than treat an error thrown from its expression as a failure, this
will treat *the lack of* an error thrown from its expression as an
expected failure.

Furthermore, so long as an error is thrown, the error will be passed to
the errorHandler block passed as a trailing closure, where the developer
may make further assertions against it.

In both cases, the new assertion function is generic on an ErrorType in
order to ensure that little to no casting will be required in the trailing
closure.
Impact on existing code

There should be little impact on existing test code because we are only
adding features and API, not changing existing features or API.

All existing tests should continue to work as implemented, and can easily
adopt the new conventions we're making available to become more concise and
intention-revealing with respect to their error handling as shown above.
Alternatives considered

We considered asking developers continue using XCTest as-is, and
encouraging them to use Swift's native error handling to both suppress and
check the validity of errors. We also considered adding additional ways of
registering failures when doing this, so that developers could register
unexpected failures themselves.

While this would result in developers using the language the same way in
their tests as in their functional code, this would also result in much
more verbose tests. We rejected this approach because such verbosity can be
a significant obstacle to testing.

Making it quick and clean to write tests for error handling could also
encourage developers to implement error handling in their code as they need
it, rather than to try to work around the feature because of any perceived
difficulty in testing.

We considered adding the ability to check that a specific error was thrown
in XCTAssertThrowsError, but this would require the ErrorType passed to
also conform to Equatable, which is also unnecessary given that this can
easily be checked in a trailing closure if desired. (In some cases a
developer may just want to ensure *an error* is thrown rather than *a
specific error* is thrown.)

We explicitly chose *not* to offer a comprehensive suite of
DoesNotThrowError assertions for XCTest in Swift, though we do offer such
DoesNotThrow assertions for XCTest in Objective-C. We feel these are of
limited utility given that our plan is for all assertions (except
XCTAssertThrowsError) to treat any thrown error as a failure.

We explicitly chose not to offer any additional support for Objective-C
exceptions beyond what we already provide: In the Xcode implementation of
XCTest, an Objective-C exception that occurs within one of our existing
assertions or tests will result in a test failure; doing more than this is
not practical given that it's possible to neither catch and handle nor
generate an Objective-C exception in Swift.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Chris Hanson) #15

The degree of change to which XCTest is open via the Swift process would be best discussed as a separate thread on either swift-corelibs-xctest or swift-evolution. Please keep this thread to discussion of the error handling proposal. Thanks.

  -- Chris

···

Sent from my iPad

On Jan 10, 2016, at 4:34 AM, James Campbell <james@supmenow.com> wrote:

I would love it if we could do a full review of XCtest in general. As there are other things it could help with I.r mocking or allowing us to express tests in a BDD way

Sent from my iPhone

On 10 Jan 2016, at 10:29, Drew Crawford via swift-evolution <swift-evolution@swift.org> wrote:

I have on the order of ~700 tests in XCTest in Swift-language projects. I'm considering migrating away from XCTest, although not over this issue.

This proposal IMO addresses an important problem, but I am not convinced it actually solves it. #2 & #3 are basically sound API designs. It is a mystery to me why #3 "generated some debate" as this is a feature I already implement manually, but I can't address unknown concerns. I can tell you I implement this, and nothing terrible has happened to me so far.

#1 I would not use. The rest of this comment explains why.

Currently I write tests about like this:

try! hopefullyNothingBad()

Now this is "bad" because it "crashes" but that's (big sigh) actually "good" because the debugger stops and/or I get a crash report that identifies at least "some" line where something bad definitely happened.

Now that is not everything I want to know–I want someone to tell me a story of how this error was created deep down in the bowels of my application, how it spent its kindergarten years in the models layer before being passed into a controller where it was rethrown onto a different dispatch queue and finally ended up in my unit test–but we can't have everything. So I settle for collecting a line number from the test case and then going hunting by hand.

When the test function throws we no longer even find out a line number in the test case anymore, because the error is passed into XCTest and the information is lost. We have just the name of the test case (I assume; the proposal is silent on this issue, but that's the only way I can think of to implement it), and some of my tests are pretty long. So, that makes it even harder to track down.

This sounds like a small thing but my test coverage is so thorough on mature projects that mostly what I turn up are heisenbugs that reproduce with 2% probability. So getting the report from the CI that has the most possible detail is critical, because if the report is not thorough enough for you to guess the bug, too bad, because that's all the information you get and the bug is not reproducible.

For that reason, I won't use #1. I hesitate about whether to call it bad idea altogether, or whether it's just not to my taste. My sense is it's probably somewhere in the middle of those two poles.

I would use #2 and #3, assuming that I don't first migrate out to a non-XCTest framework.

On Jan 9, 2016, at 8:58 PM, Chris Hanson via swift-evolution <swift-evolution@swift.org> wrote:

We’d like feedback on a proposed design for adding support for Swift error handling to XCTest, attached below. I’ll mostly let the proposal speak for itself, but there are three components to it: Allowing test methods to throw errors, allowing the expressions evaluated by assertions to throw errors, and adding an assertion for checking error handling.

We’d love to hear your feedback. We’re particularly interested in some feedback on the idea of allowing the expressions evaluated by assertions to throw errors; it’s generated some debate because it results in writing test code slightly differently than other code that makes use of Swift error handling, so any thoughts on it would be particularly appreciated.

  -- Chris Hanson (chanson@apple.com)

XCTest Support for Swift Error Handling
Proposal: SE-NNNN
Author(s): Chris Hanson
Status: Review
Review manager: TBD
Introduction
Swift 2 introduced a new error handling mechanism that, for completeness, needs to be accommodated by our testing frameworks. Right now, to write tests that involve methods that may throw an error, a developer needs to incorporate significant boilerplate into their test. We should move this into the framework in several ways, so tests of code that interacts with Swift error handling is concise and intention-revealing.

Motivation
Currently, if a developer wants to use a call that may throw an error in a test, they need to use Swift's do..catch construct in their test because tests are not themselves allowed to throw errors.

As an example, a vending machine object that has had insufficient funds deposited may throw an error if asked to vend an item. A test for that situation could reasonably use the do..catchconstruct to check that this occurs as expected. However, that means all other tests also need to use either a do..catch or try! construct — and the failure of a try! is catastrophic, so do..catch would be preferred simply for better reporting within tests.

func testVendingOneItem() {
    do {
        vendingMachine.deposit(5)
        let item = try vendingMachine.vend(row: 1, column: 1)
        XCTAssertEqual(item, "Candy Bar")
    } catch {
        XCTFail("Unexpected failure: \(error)")
    }
}
If the implementation of VendingMachine.vend(row:column:) changes during development such that it throws an error in this situation, the test will fail as it should.

One other downside of the above is that a failure caught this way will be reported as an expected failure, which would normally be a failure for which XCTest is explicitly testing via an assertion. This failure should ideally be treated as an unexpected failure, as it's not one that's anticipated in the execution of the test.

In addition, tests do not currently support throwing an error from within an assertion, requiring any code that throws an error to be invoked outside the assertion itself using the same techniques described above.

Finally, since Swift error handling is a general mechanism that developers should be implementing in their own applications and frameworks, we need to make it straightforward to write tests that ensure code that implements error handling does so correctly.

Proposed solution
I propose several related solutions to this issue:

Allow test methods to throw errors.
Allow test assertion expressions to throw errors.
Add an assertion for checking errors.
These solutions combine to make writing tests that involve thrown errors much more succinct.

Allowing Test Methods to Throw Errors

First, we can allow test methods to throw errors if desired, thus allowing the do..catch construct to be omitted when the test isn't directly checking error handling. This makes the code a developer writes when they're not explicitly trying to test error handling much cleaner.

Moving the handling of errors thrown by tests into XCTest itself also ensures they can be treated as unexpected failures, since the mechanism to do so is currently private to the framework.

With this, the test from the previous section can become:

func testVendingOneItem() throws {
    vendingMachine.deposit(5)
    let item = try vendingMachine.vend(row: 1, column: 1)
    XCTAssertEqual(item, "Candy Bar")
}
This shows much more directly that the test is intended to check a specific non-error case, and that the developer is relying on the framework to handle unexpected errors.

Allowing Test Assertions to Throw Errors

We can also allow the @autoclosure expression that is passed into an assertion to throw an error, and treat that error as an unexpected failure (since the code is being invoked in an assertion that isn't directly related to error handling). For example:

func testVendingMultipleItemsWithSufficientFunds() {
    vendingMachine.deposit(10)
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")
}
This can eliminate otherwise-dangerous uses of try! and streamline code that needs to make multiple assertions in a row.

Adding a "Throws Error" Assertion

In order to test code that throws an error, it would be useful to have an assertion that expects an error to be thrown in a particular case. Right now a developer writing code to test that an error is thrown has to test that error themselves:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        var vendingFailed = false
        do {
            _ = try vendingMachine.vend(row: 1, column: 1))
        } catch {
            vendingFailed = true
        }
        XCTAssert(vendingFailed)
    }
If we add an assertion that specifically checks whether an error was thrown, this code will be significantly streamlined:

    func testVendingFailsWithInsufficientFunds() {
        vendingMachine.deposit(1)
        XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
    }
Of course, some code may want to just detect that an error was thrown, but other code may need to check that the details of the thrown error are correct. We can take advantage of Swift's trailing closure syntax to enable this, by passing the thrown error (if any) to a closure that can itself contain assertions:

    XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
        guard let vendingError = error as? VendingMachineError else {
            XCTFail("Unexpected type of error thrown: \(error)")
            return
        }
        
        XCTAssertEquals(vendingError.item, "Candy Bar")
        XCTAssertEquals(vendingError.price, 5)
        XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
    }
This lets a developer very concisely describe an error condition for their code, in whatever level of detail they desire.

Detailed design
The design of each of the above components is slightly different, based on the functionality provided.

Tests That Throw

In order to enable test methods to throw an error, we will need to update XCTest to support test methods with a () throws -> Void signature in addition to test methods with a () -> Voidsignature as it already supports.

We will need to ensure tests that do throw an error have that error caught, and that it registers an unexpected failure.

Assertions That Throw

In order to allow assertions to throw an exception, we will need to enhance our existing assertions' @autoclosure expression parameters to add throws to their signature.

Because Swift defines a closure that can throw an error to be a proper supertype of a closure that does not, this will not result in a combinatorial explosion of assertion overrides, and will let developers naturally write code that may throw an error within an assertion.

We will treat any error thrown from within an assertion expression as an unexpected failure because while all assertions represent a test for some form of failure, they're not specifically checking for a thrown error.

The "Throws Error" Assertion

To write tests for code that throws error, we will add a new assertion function to XCTest with the following prototype:

public func XCTAssertThrowsError(
    @autoclosure expression: () throws -> Void,
                  _ message: String = "",
                       file: StaticString = __FILE__,
                       line: UInt = __LINE__,
             _ errorHandler: (error: ErrorType) -> Void = { _ in })
Rather than treat an error thrown from its expression as a failure, this will treat the lack of an error thrown from its expression as an expected failure.

Furthermore, so long as an error is thrown, the error will be passed to the errorHandler block passed as a trailing closure, where the developer may make further assertions against it.

In both cases, the new assertion function is generic on an ErrorType in order to ensure that little to no casting will be required in the trailing closure.

Impact on existing code
There should be little impact on existing test code because we are only adding features and API, not changing existing features or API.

All existing tests should continue to work as implemented, and can easily adopt the new conventions we're making available to become more concise and intention-revealing with respect to their error handling as shown above.

Alternatives considered
We considered asking developers continue using XCTest as-is, and encouraging them to use Swift's native error handling to both suppress and check the validity of errors. We also considered adding additional ways of registering failures when doing this, so that developers could register unexpected failures themselves.

While this would result in developers using the language the same way in their tests as in their functional code, this would also result in much more verbose tests. We rejected this approach because such verbosity can be a significant obstacle to testing.

Making it quick and clean to write tests for error handling could also encourage developers to implement error handling in their code as they need it, rather than to try to work around the feature because of any perceived difficulty in testing.

We considered adding the ability to check that a specific error was thrown in XCTAssertThrowsError, but this would require the ErrorType passed to also conform to Equatable, which is also unnecessary given that this can easily be checked in a trailing closure if desired. (In some cases a developer may just want to ensure an error is thrown rather than a specific error is thrown.)

We explicitly chose not to offer a comprehensive suite of DoesNotThrowError assertions for XCTest in Swift, though we do offer such DoesNotThrow assertions for XCTest in Objective-C. We feel these are of limited utility given that our plan is for all assertions (except XCTAssertThrowsError) to treat any thrown error as a failure.

We explicitly chose not to offer any additional support for Objective-C exceptions beyond what we already provide: In the Xcode implementation of XCTest, an Objective-C exception that occurs within one of our existing assertions or tests will result in a test failure; doing more than this is not practical given that it's possible to neither catch and handle nor generate an Objective-C exception in Swift.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Chris Hanson) #16

As Kevin Ballard pointed out, the examples give won’t currently compile and give the error “Call can throw, but it is executed in a non-throwing autoclosure.” I also don’t think they match up to what most developers would write. If it did compile, I would expect a developer to either just tack “throws” onto the test method, or to write something more like this:

func testFoo() {
    var foo = Foo()
    do {
        XCTAssertEqual(try foo.doSomething(), 42)
    } catch(
        XCTFail("testFoo failed: threw error \(error)")
    }
}

Then if they turn out to need easy matching between unexpected failures and individual calls, they will wind up writing multiple do…try blocks in their tests, one per expression that can throw, until they have sufficient granularity that a test failure in a log can instantly pinpoint the code that failed.

I think that would wind up cluttering up tests that are mostly intended not to be about error handling. That’s why I think it’s better to just let all of the assertions catch thrown errors as well, regardless of the substitutability argument; the assertions are already “special” by doing delayed rather than immediate evaluation, so I don’t think we’re really changing them all that much by allowing the assertion expression to throw.

Ultimately, I think allowing assertion expressions to throw makes tests easier to write as well as to read & maintain over time by moving as much of the burden of error handling on the success path to the testing framework itself. In this model, the only error handling code a developer needs to write in their tests is actually testing their error handling.

  -- Chris

···

On Jan 11, 2016, at 6:15 PM, Philippe Hausler <phausler@apple.com> wrote:

I think I am in full agreement with Dmitri on this: The XCTAssertThrowsError should be the only one that interacts with errors and that others should not try to hijack the try.

But super-big +1 on the XCTAssertThrowsError being in place to verify error cases; swift-corelibs-foundation is missing a lot of testing of error cases and it would be nice to have an effort to start verifying those (and their structures)

On Jan 10, 2016, at 11:18 AM, Dmitri Gribenko via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Sat, Jan 9, 2016 at 6:58 PM, Chris Hanson via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
Allowing Test Assertions to Throw Errors

We can also allow the @autoclosure expression that is passed into an assertion to throw an error, and treat that error as an unexpected failure (since the code is being invoked in an assertion that isn't directly related to error handling). For example:

func testVendingMultipleItemsWithSufficientFunds() {
    vendingMachine.deposit(10)
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
    XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")
}

I have significant concerns about doing this. Consider the following code:

var foo = Foo()
do {
    XCTAssertEqual(try foo.doSomething(), 42)
} catch {
    XCTAssertEqual(foo.success, false)
}

Adding ‘throws’ to the autoclosures will:

(1) change meaning of existing tests (the example above used to proceed to the catch block in case of errors, and will start failing with this change),

(2) hijacks ‘try’ and defeats its purpose — being able to understand the control flow. Developers know that if they see a ‘try’, it should match with either a do-catch, or be used in a throwing function. Adding this API makes the code misleading.

Note that although (2) applies to the XCTAssertThrowsError() API, because the name of that API makes it clear it is doing something special with error handling, I’m not concerned about it. But adding error handling to regular assertion functions has the potential to create misleading code.

Changing the way control flow works breaks one of the basic language features — substitutability:

let bar1 = try bar()
foo(bar1)

should be equivalent to:

foo(try bar())

Dmitri

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com <mailto:gribozavr@gmail.com>>*/
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Joar Wingfors) #17

The Swift language doesn’t currently make it possible to catch these failures, they result in unconditional process termination. It has been suggested that this isn’t great for testing, and could be an area for improvement in the future.

Joar

···

On Jan 14, 2016, at 6:29 AM, Jeremy W. Sherman <jeremyw.sherman@gmail.com> wrote:

I’m excited to see these improvements arrive in XCTest. Up till now, I’ve been using helpers like:

expectingThrowsAnyError(“message goes here”) {
    // do some set-up for the call
    let result = try CallSomeThrowingMethod()
    XCTAssertEqual(result.foo, expectedFoo)
}

Do you see any way to address the testing gap left by not being able to catch failing preconditions? Preconditions are significant elements of the developer-facing API, and it is frustrating to be unable to drive these requirements out - and document them - using tests.

Concretely, this means I cannot write a test like func testProvidingNegativeCountViolatesPrecondition() { … } that actually tests this element of the API specification.

Thanks,
--
Jeremy W. Sherman
https://jeremywsherman.com/

El 13 ene 2016, a las 19:52, Chris Hanson via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> escribió:

Thanks, everyone, for all of the feedback.

I sent this proposal out as more of a “heads up” that we were planning to make these changes and get feedback in advance of that, so we won’t be taking this through the full proposal process. That said, here’s what we’re going to be doing:

1. We’re going to let test methods throw, and treat those as unexpected failures, as planned.

2. We’re going to let test assertion expressions throw, and treat those as unexpected failures, as planned. We recognize that this will result in slightly different style in tests versus in regular code, but the convenience and reduction in boilerplate makes this worthwhile.

3. We’re going to add the XCTAssertThrowsError assertion, incorporating Kevin Ballard’s suggestion for how to avoid the “_ = blah()” in its expression.

4. We’re not going to add the ability for XCTAssertThrowsError to check that a specific error was thrown. This could definitely be useful but would require the function to take an ErrorType instance that also conforms to Equatable. We need to think about that some more before making such a change. Fortunately this sort of addition will be straightforward to put in via an overload or default argument, so if we choose to do it later, it shouldn’t break any code that starts to adopt XCTAssertThrowsError.

We’re going to start landing these changes today, so thanks again for all of your attention and feedback!

  -- Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Mike Ferris) #18

Honestly, we’re still trying to figure out how best to handle the balance between work that we are planning and carrying out in the Xcode version of XCTest under our normal development process and engagement of the open source community that is developing around the corelibs version of XCTest.

We would like to do as much as we can in the open. There are several competing tensions we’re trying to balance though.

- We need to be cognizant of our own release schedules and targets for the work we’re doing to extend the features and IDE integration of XCTest. Unlike the Swift language itself, the things we may be adding to corelibs XCTest are not the whole story. We are adding these things to Xcode’s XCTest as well and that work also must be accommodated. The fact that the corelibs XCTest is not the same code base as Xcode’s XCTest is going to have implications for how development on this project goes that I think we’re all still trying to figure out.

- Especially when the changes we’re making are specific to Swift or even highly relevant to Swift, we want to be able to allow the corelibs XCTest to run a little ahead of Xcode’s XCTest, but only where we’re committed to closing the gap in a known and relatively short time frame. So, for the moment, we’ve actually added this API to corelibs XCTest even though the corresponding API has not yet showed up in Xcode’s XCTest (at least not in a shipping Xcode yet… since this particular change is implemented in the overlay for Xcode’s XCTest, it should be available ahead of official release to those using toolchains from the swift project.)

- And, at times there will be cases where we’re unable to do certain XCTest work in the open (not in this case, but it will come up in the future) and there are also times when we will decide to do work in Xcode’s XCTest that is not simultaneously brought to corelibs XCTest by Apple.

In this particular case, I’d say the main issue that caused us to take a more abbreviated path to getting this stuff in was scheduling (which is an area that I cannot go into specifics about). As well there’s an aspect of us still trying to get used to the idea of how this will all work. So, we decided to dip our toes in, at least, by pushing the proposal out and getting some feedback, but we did not treat it fully as a formal proposal and review process.

Some of this is perhaps a bit vague, I realize, but hopefully it gives at least a bit of insight into the decision…

Mike

···

On Jan 13, 2016, at 5:20 PM, Brian Gesiak via swift-corelibs-dev <swift-corelibs-dev@swift.org> wrote:

Loving those commits! Especially the corresponding tests :slight_smile:

Can you explain the decision to not go through the review process? I think it's a prudent decision and I'm supportive of it, but I was under the impression that additional APIs should be proposed and reviewed. Is the rationale that swift-corelibs-xctest is too immature to justify going through proposals?

- Brian Gesiak

On Wed, Jan 13, 2016 at 4:52 PM, Chris Hanson via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
Thanks, everyone, for all of the feedback.

I sent this proposal out as more of a “heads up” that we were planning to make these changes and get feedback in advance of that, so we won’t be taking this through the full proposal process. That said, here’s what we’re going to be doing:

1. We’re going to let test methods throw, and treat those as unexpected failures, as planned.

2. We’re going to let test assertion expressions throw, and treat those as unexpected failures, as planned. We recognize that this will result in slightly different style in tests versus in regular code, but the convenience and reduction in boilerplate makes this worthwhile.

3. We’re going to add the XCTAssertThrowsError assertion, incorporating Kevin Ballard’s suggestion for how to avoid the “_ = blah()” in its expression.

4. We’re not going to add the ability for XCTAssertThrowsError to check that a specific error was thrown. This could definitely be useful but would require the function to take an ErrorType instance that also conforms to Equatable. We need to think about that some more before making such a change. Fortunately this sort of addition will be straightforward to put in via an overload or default argument, so if we choose to do it later, it shouldn’t break any code that starts to adopt XCTAssertThrowsError.

We’re going to start landing these changes today, so thanks again for all of your attention and feedback!

  -- Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev


(Mike Ferris) #19

Do you think that approach has a specific advantage over the current approach where the thrown error is passed into the optional block parameter?

They seem more or less equal in what they allow you to accomplish and I kind of like the scoping of the code to check the error within the block associated with the assert that catches the throw…

Mike

···

On Jan 14, 2016, at 11:31 AM, Jérôme Duquennoy via swift-evolution <swift-evolution@swift.org> wrote:

4. We’re not going to add the ability for XCTAssertThrowsError to check that a specific error was thrown. This could definitely be useful but would require the function to take an ErrorType instance that also conforms to Equatable. We need to think about that some more before making such a change. Fortunately this sort of addition will be straightforward to put in via an overload or default argument, so if we choose to do it later, it shouldn’t break any code that starts to adopt XCTAssertThrowsError.

Wouldn't it be possible to have XCTAssertThrowsError return the thrown error, so that the dev can do it's own test (validating an error code for exemple) ?
This would be fairly easy to implement (see the code I proposed on the 11th), and very flexible (it allows any dev to validate whatever he wants in its custom error type).

Jérôme

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jerome Duquennoy) #20

This though comes from the rules applied in my dev team : one of those rules specifies a pretty strict formatting of unit tests, with the three different parts, separated by an empty line : setup, execute (most often one single line for unit tests, highlighted by an "// Execution" comment) and validation. The validation part regroups asserts, and can mix validation on a thrown error and on the status of an object.
This requires that the thrown error is stored in a variable at execution time, to be accessible at validation time.
This strict and consistent formatting across tests is very pleasant : it makes it easy to read tests, and analyse failures. It is a time gain when tests go red :-).

But I understand that this is a local development rule, and such rules are very different between teams.
Maybe having both possibilities (the error handling bloc and the return variable) would be ideal : it would fit all situations.

Jérôme

···

Le 14 janv. 2016 à 20:34, Mike Ferris <mferris@apple.com> a écrit :

Do you think that approach has a specific advantage over the current approach where the thrown error is passed into the optional block parameter?

They seem more or less equal in what they allow you to accomplish and I kind of like the scoping of the code to check the error within the block associated with the assert that catches the throw…

Mike

On Jan 14, 2016, at 11:31 AM, Jérôme Duquennoy via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

4. We’re not going to add the ability for XCTAssertThrowsError to check that a specific error was thrown. This could definitely be useful but would require the function to take an ErrorType instance that also conforms to Equatable. We need to think about that some more before making such a change. Fortunately this sort of addition will be straightforward to put in via an overload or default argument, so if we choose to do it later, it shouldn’t break any code that starts to adopt XCTAssertThrowsError.

Wouldn't it be possible to have XCTAssertThrowsError return the thrown error, so that the dev can do it's own test (validating an error code for exemple) ?
This would be fairly easy to implement (see the code I proposed on the 11th), and very flexible (it allows any dev to validate whatever he wants in its custom error type).

Jérôme

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution