We have received feedback from a number of developers indicating that their tests have constraints that can only be checked after a test has started, and they would like the ability to end a test early and see that state change reflected in their development tools.
To date, we have not provided an API for ending a test's execution early because we want to encourage developers to use the .enabled(if:)et al. trait. This trait can be evaluated early and lets Swift Testing plan a test run more efficiently. However, we recognize that these traits aren't sufficient. Some test constraints are dependent on data that isn't available until the test starts, while others only apply to specific test cases in a parameterized test function.
So… pitch time! What if you could… cancel a test after it started running?
@Test(arguments: Species.all(in: .dinosauria))
func `Are all dinosaurs extinct?`(_ species: Species) throws {
if species.in(.aves) {
try Test.Case.cancel("\(species) is birds!")
}
// ...
}
I strongly support this and have needed it on multiple occasions. So far I’ve used withKnownIssue to try to simulate cancellation.
One question I have which I think wasn’t explicitly spelled out in the proposal is whether cancellation is new kind of state. e.g. is a cancelled test distinguished from a test that was skipped by using an enabled(false) trait?
It's distinguishable in the event stream as a new kind of event (.testCancelled instead of .testSkipped), but you cannot query a test at runtime from within your test target and ask it "did you run yet?" (etc.) I'm not sure if that answers your question?
Mainly wondering whether it was a new state such that clients (like SwiftPM, Xcode, VSCode, etc.) could decide to represent skipped and cancelled distinctly if desired. Sounds like yes.
Consequently, for SwiftPM’s CLI output is the plan is to label the two states distinctly? I’d assume yes.
Console output details aren't part of the pitch, but we do emit different phrasing for cancellation after a test or test case starts vs. skipping a test during the planning stage.
This looks great. I like that it uses distinct event types that can be tracked, for observers that might want to handle these differently. (In our case we'd probably just fold it into "skipped", but the flexibility is nice.)
We have a convention for XCTest that we require someone throwing XCTSkip to start the message with an issue number in our internal tracker. Since swift-testing has the Bug trait, I wonder if it would be helpful to have a form of Test(.Case).cancel that takes a Bug argument:
@Test func `can process image`() {
let bundle = Bundle(for: ResourceFinder.self)
guard let imageURL = bundle.url(forResource: "image", withExtension: "png") else {
try Test.cancel(.bug(id: 12345), "image not bundled yet")
}
}
The .bug could be attached to the @Test, but that loses a bit of the context of the specific reason for cancelation, and makes it harder to force the presence of a .bug as a simple regex-based pre-submit check.
Of course, the same argument about wanting a structured Bug could be made for withKnownIssue as well, so maybe that's a different request that doesn't have to be tied to this one. Test(.Case).cancel as proposed here is consistent with the other swift-testing API signatures.
For the moment at least, traits are scoped to a test function or a test suite, not to something local. I would encourage you to place a .bug() trait and a .disabled() trait on a test that is not able to run due to some bug rather than starting the test and then cancelling it.
Xcode is able to pick up on .bug() traits applied to tests and show them in the test report. It would not know how to handle a .bug() trait applied at runtime. (I know Bazel isn't bound to Xcode. Just providing some more context.)
Applying traits in a scope narrower than a test function is not, at this time, a planned feature, although we have seen some interest in something I'm calling "locally-scoped traits" which might dovetail well with what you're thinking of.
In the future world I'm thinking of, withKnownIssue { ... } naturally becomes a synonym of something like withLocalTrait(.knownIssue) { ... } (naming is hard). This is beyond the scope of this proposal, but I can add it to the Future directions section.
I sort of want to quibble about the exception that Comment is also a trait, but I understand what you mean here.
We would likely just continue to enforce syntax like Test(\(\.Case\))?\.cancel\("TODO: \d+ - in the meantime. Future directions like the one you described are nice to take advantage of structured representations and make runtime reporting more detailed, but also are trickier to detect in text-based pre-submit hooks like this, so there are always trade-offs.
But again, nothing that would require changes to the feature as proposed.
The API seems useful and usable both for the test developer and the consuming tools.
The API could be clearer if the cancel functions returned Never, which would also permit the compiler to avoid returns in guard blocks.
Also it seems like a public typed error or protocol would help. Typed throws can be very useful, and I can imagine users catching and rethrowing these errors to do their own accounting, since the bare semantics of cancel could be used for a wide variety of runtime suite-managing features. But perhaps those concerns should be handled in the test issue recording?
That suggests some exploration of the reasons users want to cancel at runtime could inform the API discussion, e.g.,:
Precondition failure, e.g., remote DB went offline, disk space filled up
Fast unwind: some signal or internal condition warrants cancelling remaining tests
The majority are failing
It’s a fuzzing suite that runs unbounded until some failure is reproduced
Sufficient coverage: for generated test cases, some threshold met
Uncovered semantics: some generated combinations are not testable
(However, it’s not clear if any of these in particular drive towards any different API.)
There are some technical reasons we aren't exposing a type here, primarily that we'd really like to use _Concurrency.CancellationError for this purpose as it is the canonical error thrown by Swift concurrency API for this purpose. _Concurrency.CancellationError does not carry any state with it though, so it's insufficient for this purpose.
This is basically the new XCTSkip, right? If so, yes please. I’ve been missing it, particularly during development when I’m not quite ready to write a full suite but I want some indication that those tests are there and need my attention at some point in the future.
In your case, I'd recommend using the .disabled() trait instead:
@Test(.disabled("FIXME: needs a suite (or whatever)"))
func `Toad the wet sprocket`() { ... }
Test cancellation is useful when a test has dependencies that can't be resolved early and you need to bail out after it has started, but test failure is not appropriate.
I have amended the proposal draft to include discussion of XCTSkip. I have an experimental PR up to add support for this type as an alternative to Test.cancel(). While we'd prefer you use Test.cancel() and/or Test.Case.cancel() in Swift Testing code, if you're already using XCTSkip then it makes sense for us to recognize when you throw it (right?)
I’d actually prefer a clean break from XCTest to Swift Testing. I don’t think we need to accommodate XCTSkip if we had Test.cancel() and Test.Case.cancel(). Feels very off to allow random legacy tidbits for their own sake.
They are different features/functions, yes. Issue.record() records an issue (unsurprisingly) which causes a test failure (or warning if you explicitly specify .warning severity.) Test.cancel() and Test.Case.cancel()cancel the current test/case respectively, which does not cause the test to fail but simply to exit early.
For comparison, see XCTFail() and XCTSkip() in XCTest.
