[Pitch] Capturing values in exit tests

In Swift 6.2, we introduced the concept of an exit test: a section of code in a test function that would run in an independent process and allow test authors to test code that terminates the process. For example:

enum Fruit: Sendable, Codable, Equatable {
  case apple, orange, olive, tomato
  var isSweet: Bool { get }

  consuming func feed(to bat: FruitBat) {
    precondition(self.isSweet, "Fruit bats don't like savory fruits!")
    ...
  }
}

@Test func `Fruit bats don't eat savory fruits`() async {
  await #expect(processExitsWith: .failure) {
    let fruit = Fruit.olive
    let bat = FruitBat(named: "Chauncey")
    fruit.feed(to: bat) // should trigger a precondition failure and process termination
  }
}

I propose extending exit tests to support capturing state from the enclosing context (subject to several practical constraints):

@Test(arguments: [Fruit.olive, .tomato])
func `Fruit bats don't eat savory fruits`(_ fruit: Fruit) async {
  await #expect(processExitsWith: .failure) { [fruit] in
    let bat = FruitBat(named: "Chauncey")
    fruit.feed(to: bat)
  }
}

Read the full proposal here.

Trying it out

To try this feature out, add a dependency to the main branch of swift-testing to your project and enable the ExperimentalExitTestValueCapture package trait[1]:

...
dependencies: [
  ...
  .package(
    url: "https://github.com/swiftlang/swift-testing.git",
    branch: "main",
    traits: [.defaults, "ExperimentalExitTestValueCapture",]
  ),
]

Then, add a target dependency to your test target:

.testTarget(
  ...
  dependencies: [
    ...
    .product(name: "Testing", package: "swift-testing"),
  ]

Finally, import Swift Testing using @_spi(Experimental) import Testing.


  1. Package traits require a package tools version of 6.2 or newer. You specify the package tools version with a comment at the top of Package.swift of the form // swift-tools-version: 6.2. ↩︎

10 Likes

Sounds like a good addition. Haven't come into contact with this specific need yet, but can imagine this popping up sooner or later.

Seems like a great addition to me. In Foundation I've had a few places where I wish I could have used this. Those cases are mainly where I have a test which exercises many different preconditions but all based on the same starting state, and it'd be nice to just capture that starting state in each expectation rather than creating it in each expectation. For example, swift-foundation/Tests/FoundationEssentialsTests/AttributedString/AttributedStringIndexTrackingTests.swift at 09a05243474b59949ed20c7601f6915e391feb62 · swiftlang/swift-foundation · GitHub.

My biggest concern when reading the proposal was the diagnostics and whether they would be clear and actionable (in cases where you have non-Codable captures or where the captures are of unknown types), but reading through your example diagnostics it sounds like all of those expected failure cases have well formed diagnostics.

Only one suggestion:

Hence, only values listed in the closure's capture list will be captured. Implicitly captured values will produce a compile-time diagnostic as they do today.

The capture syntax of [fruit = fruit as Fruit] is a little bit unintuitive at first (since I don't use the explicit capture syntax much, there's no autocompletion for it, and there's few cases where you'd need to spell out a dynamic cast here). Would it be possible for the diagnostic errors that specify that you have not explicitly listed a captured value in the captures list to provide a Fix-it that adds it to the list (and creates the syntax for the list if not present already)? That would aid in the experience when adopting this since even if you don't remember the explicit capture syntax as you start (since you don't need it in many other places and the type syntax is a little bit contrived) the compiler can substitute it for you with a single click.

I'd much prefer if you could just write [fruit] here, but that's what the "decltype()" future direction bullet is about. In this case, "the perfect is the enemy of the good" and, since [fruit = fruit as Fruit] is valid in this position regardless of what our macro does, it makes sense to me that we should support this syntax now and extend type inference support when the language allows. I hope that makes sense!

This specific diagnostic is generated by the compiler rather than our macro and is the same as the one you'd get if you tried to capture a value in an @convention(c) function:

:stop_sign: A C function pointer cannot be formed from a closure that captures context

Although in the release Swift 6.1.2 toolchain it crashes in a SIL pass (#80269).

Either way, we don't have control over this diagnostic in Swift Testing. If we had something like (hypothetically) @_explicitCapturesOnly that we could apply to a closure to indicate all captures needed to be explicit, then the compiler could provide a Fix-It like the one you describe. We'd probably need more use cases than just this one to justify that sort of compiler change.

This looks great, and the limitations are reasonable and understandable.

If we are able to lift this constraint in the future, we expect it will not require (no pun intended) a second Swift Evolution proposal.

Nit: I believe the pun not intended would be on "expect" rather than "require" :wink:

4 Likes

It's both!

1 Like

+1 from me overall. A couple specific suggestions on the proposal:

  • I think it may help to be a little more explicit about what the technical changes being proposed are. It might not be obvious to users that they cannot currently include a capture list in an exit test body closure, so that's one thing specific thing I would suggest clarifying by mentioning that capture lists are disallowed. The "Source compatibility" section mentions it but it just might be useful to say that more prominently earlier in the document.

  • The title of the proposal is "Capturing values in exit tests" — I wonder if it would be useful to prefix that with the word Explicitly to clarify that this feature does require an explicit capture list. Someday in the future, if we ever got a feature like decltype or typeof as you allude to, would we need another proposal to facilitate implicit captures? And if so, I wonder if the title of this earlier proposal might be confusing.

Stuart

2 Likes

I will adjust the text to make it clear that capture lists currently diagnose. If you have other specific suggestions, feel free to reply here or DM me.

If you feel strongly, I can rename the proposal document, but that level of nuance is probably not needed in its title…?

I do not anticipate that we will ever support implicit captures. They are not visible in the syntax tree and macros like #expect do not have the ability to see them, nor do I have any reason to think they will gain this ability in the future.