[Pitch] Test cancellation

Evolution Pitches
,
1

We have received feedback from a number of developers indicating that their tests have constraints that can only be checked after a test has started, and they would like the ability to end a test early and see that state change reflected in their development tools.

To date, we have not provided an API for ending a test's execution early because we want to encourage developers to use the .enabled(if:) et al. trait. This trait can be evaluated early and lets Swift Testing plan a test run more efficiently. However, we recognize that these traits aren't sufficient. Some test constraints are dependent on data that isn't available until the test starts, while others only apply to specific test cases in a parameterized test function.

So… pitch time! What if you could… cancel a test after it started running? :exploding_head:

@Test(arguments: Species.all(in: .dinosauria))
func `Are all dinosaurs extinct?`(_ species: Species) throws {
  if species.in(.aves)  {
    try Test.Case.cancel("\(species) is birds!")
  }
  // ...
}

Read the full proposal here.

Trying it out

To try this feature out, add a dependency to the main branch of swift-testing to your project:

...
dependencies: [
  ...
  .package(url: "https://github.com/swiftlang/swift-testing.git", branch: "main"),
]

Then, add a target dependency to your test target:

.testTarget(
  ...
  dependencies: [
    ...
    .product(name: "Testing", package: "swift-testing"),
  ]
)

Finally, import Swift Testing using @_spi(Experimental) import Testing.

11 Likes
2

I strongly support this and have needed it on multiple occasions. So far I’ve used withKnownIssue to try to simulate cancellation.

One question I have which I think wasn’t explicitly spelled out in the proposal is whether cancellation is new kind of state. e.g. is a cancelled test distinguished from a test that was skipped by using an enabled(false) trait?

1 Like
3

It's distinguishable in the event stream as a new kind of event (.testCancelled instead of .testSkipped), but you cannot query a test at runtime from within your test target and ask it "did you run yet?" (etc.) I'm not sure if that answers your question?

4

Mainly wondering whether it was a new state such that clients (like SwiftPM, Xcode, VSCode, etc.) could decide to represent skipped and cancelled distinctly if desired. Sounds like yes.

Consequently, for SwiftPM’s CLI output is the plan is to label the two states distinctly? I’d assume yes.

5

Console output details aren't part of the pitch, but we do emit different phrasing for cancellation after a test or test case starts vs. skipping a test during the planning stage.

1 Like
6

This looks great. I like that it uses distinct event types that can be tracked, for observers that might want to handle these differently. (In our case we'd probably just fold it into "skipped", but the flexibility is nice.)

We have a convention for XCTest that we require someone throwing XCTSkip to start the message with an issue number in our internal tracker. Since swift-testing has the Bug trait, I wonder if it would be helpful to have a form of Test(.Case).cancel that takes a Bug argument:

@Test func `can process image`() {
  let bundle = Bundle(for: ResourceFinder.self)
  guard let imageURL = bundle.url(forResource: "image", withExtension: "png") else {
    try Test.cancel(.bug(id: 12345), "image not bundled yet")
  }
}

The .bug could be attached to the @Test, but that loses a bit of the context of the specific reason for cancelation, and makes it harder to force the presence of a .bug as a simple regex-based pre-submit check.

Of course, the same argument about wanting a structured Bug could be made for withKnownIssue as well, so maybe that's a different request that doesn't have to be tied to this one. Test(.Case).cancel as proposed here is consistent with the other swift-testing API signatures.

7

For the moment at least, traits are scoped to a test function or a test suite, not to something local. I would encourage you to place a .bug() trait and a .disabled() trait on a test that is not able to run due to some bug rather than starting the test and then cancelling it.

Xcode is able to pick up on .bug() traits applied to tests and show them in the test report. It would not know how to handle a .bug() trait applied at runtime. (I know Bazel isn't bound to Xcode. Just providing some more context.)

Applying traits in a scope narrower than a test function is not, at this time, a planned feature, although we have seen some interest in something I'm calling "locally-scoped traits" which might dovetail well with what you're thinking of.

In the future world I'm thinking of, withKnownIssue { ... } naturally becomes a synonym of something like withLocalTrait(.knownIssue) { ... } (naming is hard). This is beyond the scope of this proposal, but I can add it to the Future directions section.

8

I sort of want to quibble about the exception that Comment is also a trait, but I understand what you mean here. :grin:

We would likely just continue to enforce syntax like Test(\(\.Case\))?\.cancel\("TODO: \d+ - in the meantime. Future directions like the one you described are nice to take advantage of structured representations and make runtime reporting more detailed, but also are trickier to detect in text-based pre-submit hooks like this, so there are always trade-offs.

But again, nothing that would require changes to the feature as proposed.

9

The API seems useful and usable both for the test developer and the consuming tools.

The API could be clearer if the cancel functions returned Never, which would also permit the compiler to avoid returns in guard blocks.

Also it seems like a public typed error or protocol would help. Typed throws can be very useful, and I can imagine users catching and rethrowing these errors to do their own accounting, since the bare semantics of cancel could be used for a wide variety of runtime suite-managing features. But perhaps those concerns should be handled in the test issue recording?

That suggests some exploration of the reasons users want to cancel at runtime could inform the API discussion, e.g.,:

  1. Precondition failure, e.g., remote DB went offline, disk space filled up
  2. Fast unwind: some signal or internal condition warrants cancelling remaining tests
    1. The majority are failing
    2. It’s a fuzzing suite that runs unbounded until some failure is reproduced
    3. Sufficient coverage: for generated test cases, some threshold met
  3. Uncovered semantics: some generated combinations are not testable

(However, it’s not clear if any of these in particular drive towards any different API.)

10

They already do return Never, because they always throw. See the signatures in the Detailed Design.

11

There are some technical reasons we aren't exposing a type here, primarily that we'd really like to use _Concurrency.CancellationError for this purpose as it is the canonical error thrown by Swift concurrency API for this purpose. _Concurrency.CancellationError does not carry any state with it though, so it's insufficient for this purpose.

12

Sorry for the noise: Fail to scroll.

1 Like
13

Ship it this is sweet.