I wouldn't mind if we could have something closer to this.
···
On Sun, Jan 10, 2016 at 3:01 PM, Ross O'Brien <narrativium+swift@gmail.com> wrote:
I've been wondering for a while, while writing unit tests, why we still
start with "func test[behaviourOfThing]() {" and not "test behaviourOfThing
{". It's not just about less typing: parsing a function for the 'test'
prefix is an Objective C holdover, and doesn't feel Swift to me, and it has
tests behaving like functions when - as illustrated in this proposal - they
should have different behaviours. It should be clearer when reading code
whether a function is a test or a helper function to make tests easier to
write.On Sun, Jan 10, 2016 at 12:34 PM, James Campbell via swift-evolution < > swift-evolution@swift.org> wrote:
I would love it if we could do a full review of XCtest in general. As
there are other things it could help with I.r mocking or allowing us to
express tests in a BDD waySent from my iPhone
On 10 Jan 2016, at 10:29, Drew Crawford via swift-evolution < >> swift-evolution@swift.org> wrote:
I have on the order of ~700 tests in XCTest in Swift-language projects.
I'm considering migrating away from XCTest, although not over this issue.This proposal IMO addresses an important problem, but I am not convinced
it actually solves it. #2 & #3 are basically sound API designs. It is a
mystery to me why #3 "generated some debate" as this is a feature I already
implement manually, but I can't address unknown concerns. I can tell you I
implement this, and nothing terrible has happened to me so far.#1 I would not use. The rest of this comment explains why.
Currently I write tests about like this:
try! hopefullyNothingBad()
Now this is "bad" because it "crashes" but that's (big sigh) actually
"good" because the debugger stops and/or I get a crash report that
identifies at least "some" line where something bad definitely happened.Now that is not everything I *want* to know–I *want* someone to tell me
a story of how this error was created deep down in the bowels of my
application, how it spent its kindergarten years in the models layer before
being passed into a controller where it was rethrown onto a different
dispatch queue and finally ended up in my unit test–but we can't have
everything. So I settle for collecting a line number from the test case
and then going hunting by hand.When the test function throws we no longer even find out a line number in
the test case anymore, because the error is passed into XCTest and the
information is lost. We have just the name of the test case (I assume; the
proposal is silent on this issue, but that's the only way I can think of to
implement it), and some of my tests are pretty long. So, that makes it
even harder to track down.This sounds like a small thing but my test coverage is so thorough on
mature projects that mostly what I turn up are heisenbugs that reproduce
with 2% probability. So getting the report from the CI that has the most
possible detail is critical, because if the report is not thorough enough
for you to guess the bug, too bad, because that's all the information you
get and the bug is not reproducible.For that reason, I won't use #1. I hesitate about whether to call it bad
idea altogether, or whether it's just not to my taste. My sense is it's
probably somewhere in the middle of those two poles.I would use #2 and #3, assuming that I don't first migrate out to a
non-XCTest framework.On Jan 9, 2016, at 8:58 PM, Chris Hanson via swift-evolution < >> swift-evolution@swift.org> wrote:
We’d like feedback on a proposed design for adding support for Swift
error handling to XCTest, attached below. I’ll mostly let the proposal
speak for itself, but there are three components to it: Allowing test
methods to throw errors, allowing the expressions evaluated by assertions
to throw errors, and adding an assertion for checking error handling.We’d love to hear your feedback. We’re particularly interested in some
feedback on the idea of allowing the expressions evaluated by assertions to
throw errors; it’s generated some debate because it results in writing test
code slightly differently than other code that makes use of Swift error
handling, so any thoughts on it would be particularly appreciated.-- Chris Hanson (chanson@apple.com)
XCTest Support for Swift Error Handling
- Proposal: SE-NNNN
<https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
- Author(s): Chris Hanson <https://github.com/eschaton>
- Status: *Review*
- Review manager: TBDIntroduction
Swift 2 introduced a new error handling mechanism that, for completeness,
needs to be accommodated by our testing frameworks. Right now, to write
tests that involve methods that may throw an error, a developer needs to
incorporate significant boilerplate into their test. We should move this
into the framework in several ways, so tests of code that interacts with
Swift error handling is concise and intention-revealing.
MotivationCurrently, if a developer wants to use a call that may throw an error in
a test, they need to use Swift's do..catch construct in their test
because tests are not themselves allowed to throw errors.As an example, a vending machine object that has had insufficient funds
deposited may throw an error if asked to vend an item. A test for that
situation could reasonably use the do..catchconstruct to check that this
occurs as expected. However, that means all other tests *also* need to
use either a do..catch or try! construct — and the failure of a try! is
catastrophic, so do..catch would be preferred simply for better
reporting within tests.func testVendingOneItem() {
do {
vendingMachine.deposit(5)
let item = try vendingMachine.vend(row: 1, column: 1)
XCTAssertEqual(item, "Candy Bar")
} catch {
XCTFail("Unexpected failure: \(error)")
}}If the implementation of VendingMachine.vend(row:column:) changes during
development such that it throws an error in this situation, the test will
fail as it should.One other downside of the above is that a failure caught this way will be
reported as an *expected failure*, which would normally be a failure for
which XCTest is explicitly testing via an assertion. This failure should
ideally be treated as an *unexpected failure*, as it's not one that's
anticipated in the execution of the test.In addition, tests do not currently support throwing an error from within
an assertion, requiring any code that throws an error to be invoked outside
the assertion itself using the same techniques described above.Finally, since Swift error handling is a general mechanism that
developers should be implementing in their own applications and frameworks,
we need to make it straightforward to write tests that ensure code that
implements error handling does so correctly.
Proposed solutionI propose several related solutions to this issue:
1. Allow test methods to throw errors.
2. Allow test assertion expressions to throw errors.
3. Add an assertion for checking errors.These solutions combine to make writing tests that involve thrown errors
much more succinct.
Allowing Test Methods to Throw ErrorsFirst, we can allow test methods to throw errors if desired, thus
allowing the do..catch construct to be omitted when the test isn't
directly checking error handling. This makes the code a developer writes
when they're not explicitly trying to test error handling much cleaner.Moving the handling of errors thrown by tests into XCTest itself also
ensures they can be treated as unexpected failures, since the mechanism to
do so is currently private to the framework.With this, the test from the previous section can become:
func testVendingOneItem() throws {
vendingMachine.deposit(5)
let item = try vendingMachine.vend(row: 1, column: 1)
XCTAssertEqual(item, "Candy Bar")}This shows much more directly that the test is intended to check a
specific non-error case, and that the developer is relying on the framework
to handle unexpected errors.
Allowing Test Assertions to Throw ErrorsWe can also allow the @autoclosure expression that is passed into an
assertion to throw an error, and treat that error as an unexpected failure
(since the code is being invoked in an assertion that isn't directly
related to error handling). For example:func testVendingMultipleItemsWithSufficientFunds() {
vendingMachine.deposit(10)
XCTAssertEqual(try vendingMachine.vend(row: 1, column: 1), "Candy Bar")
XCTAssertEqual(try vendingMachine.vend(row: 1, column: 2), "Chips")}This can eliminate otherwise-dangerous uses of try! and streamline code
that needs to make multiple assertions in a row.
Adding a "Throws Error" AssertionIn order to test code that throws an error, it would be useful to have an
assertion that expects an error to be thrown in a particular case. Right
now a developer writing code to test that an error is thrown has to test
that error themselves:func testVendingFailsWithInsufficientFunds() {
vendingMachine.deposit(1)
var vendingFailed = false
do {
_ = try vendingMachine.vend(row: 1, column: 1))
} catch {
vendingFailed = true
}
XCTAssert(vendingFailed)
}If we add an assertion that specifically checks whether an error was
thrown, this code will be significantly streamlined:func testVendingFailsWithInsufficientFunds() {
vendingMachine.deposit(1)
XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1))
}Of course, some code may want to just detect that an error was thrown,
but other code may need to check that the details of the thrown error are
correct. We can take advantage of Swift's trailing closure syntax to enable
this, by passing the thrown error (if any) to a closure that can itself
contain assertions:XCTAssertThrowsError(_ = try vendingMachine.vend(row: 1, column: 1)) { error in
guard let vendingError = error as? VendingMachineError else {
XCTFail("Unexpected type of error thrown: \(error)")
return
}XCTAssertEquals(vendingError.item, "Candy Bar")
XCTAssertEquals(vendingError.price, 5)
XCTAssertEquals(vendingError.message, "A Candy Bar costs 5 coins")
}This lets a developer very concisely describe an error condition for
their code, in whatever level of detail they desire.
Detailed designThe design of each of the above components is slightly different, based
on the functionality provided.
Tests That ThrowIn order to enable test methods to throw an error, we will need to update
XCTest to support test methods with a () throws -> Void signature in
addition to test methods with a () -> Voidsignature as it already
supports.We will need to ensure tests that do throw an error have that error
caught, and that it registers an unexpected failure.
Assertions That ThrowIn order to allow assertions to throw an exception, we will need to
enhance our existing assertions' @autoclosure expression parameters to
add throws to their signature.Because Swift defines a closure that can throw an error to be a proper
supertype of a closure that does not, this *will not* result in a
combinatorial explosion of assertion overrides, and will let developers
naturally write code that may throw an error within an assertion.We will treat any error thrown from within an assertion expression as an
*unexpected* failure because while all assertions represent a test for
some form of failure, they're not specifically checking for a thrown error.
The "Throws Error" AssertionTo write tests for code that throws error, we will add a new assertion
function to XCTest with the following prototype:public func XCTAssertThrowsError(
@autoclosure expression: () throws -> Void,
_ message: String = "",
file: StaticString = __FILE__,
line: UInt = __LINE__,
_ errorHandler: (error: ErrorType) -> Void = { _ in })Rather than treat an error thrown from its expression as a failure, this
will treat *the lack of* an error thrown from its expression as an
expected failure.Furthermore, so long as an error is thrown, the error will be passed to
the errorHandler block passed as a trailing closure, where the developer
may make further assertions against it.In both cases, the new assertion function is generic on an ErrorType in
order to ensure that little to no casting will be required in the trailing
closure.
Impact on existing codeThere should be little impact on existing test code because we are only
adding features and API, not changing existing features or API.All existing tests should continue to work as implemented, and can easily
adopt the new conventions we're making available to become more concise and
intention-revealing with respect to their error handling as shown above.
Alternatives consideredWe considered asking developers continue using XCTest as-is, and
encouraging them to use Swift's native error handling to both suppress and
check the validity of errors. We also considered adding additional ways of
registering failures when doing this, so that developers could register
unexpected failures themselves.While this would result in developers using the language the same way in
their tests as in their functional code, this would also result in much
more verbose tests. We rejected this approach because such verbosity can be
a significant obstacle to testing.Making it quick and clean to write tests for error handling could also
encourage developers to implement error handling in their code as they need
it, rather than to try to work around the feature because of any perceived
difficulty in testing.We considered adding the ability to check that a specific error was
thrown in XCTAssertThrowsError, but this would require the ErrorType passed
to also conform to Equatable, which is also unnecessary given that this
can easily be checked in a trailing closure if desired. (In some cases a
developer may just want to ensure *an error* is thrown rather than *a
specific error* is thrown.)We explicitly chose *not* to offer a comprehensive suite of
DoesNotThrowError assertions for XCTest in Swift, though we do offer
such DoesNotThrow assertions for XCTest in Objective-C. We feel these
are of limited utility given that our plan is for all assertions (except
XCTAssertThrowsError) to treat any thrown error as a failure.We explicitly chose not to offer any additional support for Objective-C
exceptions beyond what we already provide: In the Xcode implementation of
XCTest, an Objective-C exception that occurs within one of our existing
assertions or tests will result in a test failure; doing more than this is
not practical given that it's possible to neither catch and handle nor
generate an Objective-C exception in Swift.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution
--
Wizard
james@supmenow.com
+44 7523 279 698