[Second review] ST-0008: Exit tests

Evolution Proposal Reviews
1

Hello Swift Community,

We recently concluded the review conducted from March 20 through April 8, 2025. The feedback on the proposed feature of Exit Tests was overwhelmingly positive, with a main focus on improving the naming of the parameter label for expected test outcomes, such as failure or exit codes. Based on this feedback, we have decided to initiate a second review specifically addressing the refinement of this label. Details of this upcoming review can be found below.

Additional Suggestions:

Skip tests on unsupported platforms There was a suggestion to explicitly mark tests as 'skipped' on platforms where Exit Tests cannot currently be supported, instead of failing silently or not compiling. After discussion within the workgroup, we decided to maintain the current behavior of not compiling on unsupported platforms for the following reasons:

  1. Swift Testing does not presently support skipping tests once they have started, and implementing such a mechanism is beyond the scope of this proposal. This is a valid feature worthy of its own proposal and SE process. If this capability becomes available in the future, we will revisit our decision.
  2. The existing options are failing silently or not compiling (failing loudly). We believe not compiling is beneficial as it prevents the false impression of passing tests when none are actually executed.
  3. Our intention is to extend Exit Tests to all platforms that support Swift in time, making interim solutions unnecessary.

Extending support to more platforms The proposal lacked clarity on whether future platform support would require a separate SE process. We agreed that Swift Testing features should be extended to as many Swift-supported platforms as possible without necessitating a new SE proposal. To clarify, the proposal will be updated to reflect this intention explicitly.

Second review for naming the parameter label
Feedback indicated significant interest in changing the name of the label currently known as #expect(exitsWith: ). Suggestions included:

  • Simplifying to exit or exits
  • Alternatives like exitsReturning
  • Substituting exits with terminating
  • Removing the label altogether
  • Modifying the API to #expectExitFailure
  • Other variations

After considering these suggestions, we've determined that:

  • The #expect( prefix will remain unchanged for consistency with other test forms, such as #expect(throws:)
  • Retaining a label is crucial to distinguishing between exit tests and tests expecting failures
  • Including process in the label is beneficial, highlighting the importance of test execution in a separate process, which affects test writing capabilities
  • The API should facilitate fluent reading. Therefore, a transition word like With or Reporting is necessary following #expect(<exits|terminates|...).

Our current preference is #expect(processExitsWith:). The second review will aim to finalize the naming of this parameter. We welcome further ideas and suggestions within the review thread.

The second review runs from April 10 - April 21.

Thank you to everyone who contributed to this review, helping us enhance Swift as a programming language!

Maarten Engels
Review Manager

5 Likes
2

The conclusion here doesn't logically follow the premise. I think most Swift APIs strive to facilitate fluent reading, yet spurious words like with are generally avoided in label names except when they significantly change the meaning, and isn't generally considered to impact fluency.

For this API in particular, I'm highly skeptical that the general consensus would be that this can't be read fluently:

#expect(processExit: .signal(SIGTERM)) {
    fooFunction()
}

Or that this somehow adds clarity to a programmer:

#expect(processExitsWith: .signal(SIGTERM)) {
    fooFunction()
}

The actual reason to require this transition word (based on the previous review thread) seems to stem from a preference to have a verb following #expect, which forces the label (exit/terminate) to have an -s inflection (exits/terminates) to match "process", which in turn requires the addition of "with" (exitsWith/terminatesWith) to clarify that the parameter is not the direct object (as in the .signal(SIGTERM) is not the thing being exited/terminated).

I agree the intermediate version would be genuinely confusing:

#expect(processExits: .signal(SIGTERM)) { ... } // ⚠️

But this is a self-imposed constraint. Why do the authors feel so strongly that only a verb can follow #expect here? I've read "consistency" being floated around, but I don't see how using a noun instead of a verb when it comes more naturally harms consistency or much less discoverability. And there's already #expect( _ condition: ), which is also not a verb[1].

  1. Unless we're trying to be consistent with "the #expect APIs that take a trailing closure" specifically, but at that point that's like drawing a trend line with two data points. ↩︎

7 Likes
3

If process is important enough to justify being somewhere in the API, but not important enough to justify changing the base name, then the parameter it adheres to more closely (and which is most directly affected by 'test writing capabilities') is the trailing closure.

Thus, I'd suggest that the name which then adheres to all of these criteria would be: #expect(exit:process:).

4 Likes
4

This rule is present for overloads of #expect that take a trailing closure, specifically. Using a noun would not be consistent here with existing overloads that take a trailing closure (i.e. #expect(throws:) { ... }).

Using a noun or noun phrase in label position does not follow Swift's general guidelines for naming parameters (except those of initializers).

For technical reasons involving macro expansion, the label on the trailing closure is currently fixed as performing. But ignoring that—trailing closure labels are elided, so this would have no practical effect on legibility and would just result in #expect(exit: ...) { ... }.

2 Likes
5

There are no parts of speech forbidden as such for labels—it's whatever promotes clarity and the rest of the criteria by which we evaluate API naming. If the base name of a function is a verb, then what tends to follow that verb grammatically and best describes the parameter would be appropriate for the label—this may indeed be a preposition, or it may be a noun which is the direct object; it is rarely another verb though.

That is indeed part of the elegance of it: since the rationale for its inclusion is that execution in a separate process affects test writing, a label which is visible in all API documentation and at the declaration site but (at the writer's discretion) optionally elided for the reader is the best of both worlds: it would fulfill the workgroup's criteria without increasing verbosity at the use site!

3 Likes
6

As stated previously, we would have to reject that proposal on technical grounds. I'm not sure this argument holds any water though, and it sounds like you're proposing #expect(exit:) again, which I have already stated is not a candidate name.

7

I thinks this just make the label more mouthful without meaningful gain.

I would like to suggests#expect(reports:). It's short and aligns well with #expect(throws:).

If with is a must I would suggest #expect(terminatesWith:) because it avoids the word repetition when using .exitCode.

6 Likes
8

Now that the workgroup has set out explicit criteria, I'm just pointing out that it meets all the criteria! :wink:

But in all seriousness, I appreciate the point that "process" helps describe test writing capabilities that are affected. However, if "process" must be included in the name, but must not be included in the base name and cannot be included in the label for the closure that is actually run on that process and which is affected in terms of test writing capabilities, that is a pretty unsatisfying reason indeed for putting the word in the label for the first argument.

Taking as given that there is a technical block to labeling the second argument with "process," I would strenuously urge reconsideration of the only discretionary part of these premises: That is, if "process" is important enough to be part of the name because it affects test writing capabilities and important enough not to be elided, then the naming guidelines tell us it ought to be part of the base name because it applies to the whole macro and not the first argument. "Consistency" applies to things which are alike being spelled alike, and the workgroup is arguing that there is something different here which merits highlighting in the name, which cuts against the argument that the base name ought to be the same.

It bears mentioning that past practice, in line with the naming guidelines, is that we extract words which apply to all arguments into the base name without considering that it disturbs "consistency"—see, for example, top-level swap(_:_:) versus Array.swapAt(_:_:).

Having said that, if "process" is read as an adjective attributive which modifies another noun, it could be shoehorned into the first label. Then we could have something like:

#expect(processTermination:)

But that's still a mouthful, with "termination" just being a longer synonym for the much more searchable "exit" (again, read as a noun):

#expect(processExit:)
8 Likes
9
Off-topic procedural remarks

As a matter of process, I do not think it is advisable for an author of a proposal to engage in repeatedly arguing against suggestions that other people post in a review thread.

The author’s views should already be sufficiently explained in the text of the proposal, and if they feel that is not the case then they may communicate with the review manager about revising the proposal.

In a targeted re-review, the author is of course welcome to participate and share their preferences and opinions. But for them to continually argue against those of other reviewers seems to me rather contrary to the spirit of the review.

If nothing else, new people coming to the thread may perceive new ideas being met with immediate opposition from a proposal author, and then feel disinclined to share their own views because they do not want to feel jumped on.

The purpose of the re-review is to welcome different ideas and suggestions for the workgroup to consider. We want participants to feel welcome to share their ideas. That is how we end up with a broader variety of possibilities for the workgroup to consider, and thus a higher likelihood of finding the “best possible” result.

5 Likes
10

As review manager, I believe constructive dialogue is crucial for the evolution of ideas. If suggestions and arguments are well articulated and bring novel insights, it enriches the discussion and should be encouraged, including contributions from the proposal's author.

However, it does feel like we are at the risk of repeating arguments instead of bringing in novel ideas. We might not even be able to come to consensus. Which is also a valid conclusion off course.

For now, lets try and find new insights. Just as with the previous review, I keep track of all suggestions and in the workgroup we look at all suggestions from all participants.

2 Likes
11

Thank you @Nevin. As a meta commentary, the Testing Workgroup has recently been established, and many members of our group are new to participating in the Swift Evolution process at this level. Many of us have offered our thoughts in other SE reviews, often passionately, but now we are operating in a new capacity and it’s an adjustment.

I agree with the spirit of your critique, and think we need to stay mindful of this in the future — especially when a proposal author is a member of the workgroup. All authors should take great care to maintain an open stance during a review and be cautious about repeatedly defending their proposal, but this can be hard since they often hold strong views about the subject! I do feel @grynspan's comments have maintained a professional tone throughout this review, FWIW.

I think it is perfectly reasonable and expected that a proposal author participate in the discussion and clarify things, and even explain rationale, but there’s a balance that needs to be struck and sometimes it’s best to withhold responses after a certain point and let the discussion play out among other community members. I will make sure that topic is discussed in the next workgroup meeting. Also, I’m not sure this has been stated elsewhere but FYI: In the meeting where we decided to review this proposal, @grynspan proactively volunteered to recuse himself from the final decision since he's the author. This is a precedent which will be applied to future situations when a proposal author is a workgroup member.

7 Likes
12

There's a single macro that matches that extremely narrow criteria, #expect(throws:) :grinning_face_with_smiling_eyes:

If the argument is that #expect(processExit:) is inconsistent with the existing overloads because it's a noun, an analogous argument can be made that #expect(processExitsWith:) is also inconsistent with the existing overloads, on the grounds that existing labels for that API don't include prepositions, or that #expect(processExitsWith:) is similarly inconsistent by not keeping verbosity below a certain (arbitrary) level that throws: meets but processExitsWith: doesn't.

What makes the first a rule but not the others?

There's definitely precendent for Swift APIs using nouns in labels, even in the Swift API Guidelines: split(maxSplits: 12), fadeFrom(red: b, green: c, blue: d), subscript(index: Int)...

And, since the topic of guidelines has been brought up, the general guidelines for naming also mention this:

More words may be needed to clarify intent or disambiguate meaning, but those that are redundant with information the reader already possesses should be omitted.

Yet I'm having trouble thinking of what possible new information is conveyed in processExitsWith that processExit did not already have, unlike —for example— fadeFrom(red:green:blue), where dropping the "from" would leave users clueless as to whether the API fades from that color or to that color.

What other possible interpretation of processExit: .signal(SIGTERM) could there be, that the addition of "with" disambiguates? It's not possible to "exit a signal" (or failure, or exit code).

2 Likes
13

I, for one, have thoroughly appreciated @grynspan taking the time and effort to engage :slight_smile:

3 Likes
14

I mean well, I promise! But I'll take a step back at this point and let other contributors discuss more. My intent is certainly not to dominate the discussion.

(@maartene is continuing to collate feedback either way.)

4 Likes
15

Let me chime in with one last point bouncing off of this, and I'll also let others have at it.

What I, at least, thought was an exciting new direction with #expect(...) as compared to XCTest APIs—and, if I recall, this was something that was explicitly heralded at its introduction—is that you could now learn one or two new names (such as #expect) and then spell everything else with idiomatic Swift. (If there is any compelling argument for not varying the base name, it is this one.) So, for instance, is it XCTAssertEqual or XCTAssertEquals or XCTAssertEquivalent or ...? Now you don't have to remember, because you just write ==.

So what would one expect to be the label of the API that tests for a thrown error? Of course, it should be either throw or throws, because that's how you spell it in idiomatic Swift. One of these reads better in context, and that's exactly what Swift Testing calls it.

So what would one expect to be the label of the API that tests for a process exit? To my mind, this should be the overarching line of thinking. If there is an obvious answer that people will want to try first, and it is not egregious in the broader context of idiomatic API naming, then that is the answer. Anything else is going back on the distinguishing innovation of Swift Testing #expect(...) APIs as compared to XCTest, which (in part) got us away from having to worry about parts-of-speech, and verb declensions, and noun-verb agreement, and...

9 Likes
16

I understand the need for generality and covering all the bases, but I find myself wishing for the very simple:

#expect(crashes: eat(nonDeliciousTaco))

// acceptable alternatives:
// #expect(fatal: ...)
// #expect(failsAssertion: ...)
// #expect(failsPrecondition: ...)

(Disclaimer: this has probably come up in previous reviews I skipped, but since you're soliciting naming feedback, this is really all I want. I'm not interested in tests about process spawning and collecting output and exit codes. I want to make sure my precondition() and fatalError() calls get hit)

EDIT: ok so apparently a closure is necessary, which means I want:

#expect(crashes: { 
  var taco = Taco()
  taco.isDelicious = false
  eat(taco)
 })
2 Likes
17

I know this review is targeted at the spelling of the parameter label, but while we’re here I want to mention the enum StatusAtExit. I would have expected it to be named something like ExitStatus or ProcessExitStatus.

• • •

As for the parameter label, one possibility is to spell it exitCode and make the parameter type expressible by an integer literal. That way call sites would look like this:

#expect(exitCode: 42) { ... }
#expect(exitCode: .success) { ... }
#expect(exitCode: .failure) { ... }
#expect(exitCode: .signal(3) { ... }

When a signal is expected this reads a bit oddly, but in practice I think it works well. Of course, if the parameter type gets used in other places without the label exitCode, then the integer literal conformance might be confusing.

• • •

Another possibility is to spell the label processResult, so call sites look like this:

#expect(processResult: .exitCode(42)) { ... }
#expect(processResult: .success) { ... }
#expect(processResult: .failure) { ... }
#expect(processResult: .signal(3) { ... }

That’s somewhat more verbose, especially when .exitCode(...) is spelled out instead of being implied by an integer literal, but it reads well in every case. The label clearly identifies what the parameter represents, and it includes the word “process” to indicate that an exit test will be run.

• • •

Alternatively, taking a cue from the parameter type itself, which is ExitTest.Condition, the label could be exitCondition:

#expect(exitCondition: .exitCode(42)) { ... }
#expect(exitCondition: .success) { ... }
#expect(exitCondition: .failure) { ... }
#expect(exitCondition: .signal(3) { ... }
2 Likes
18

Given the StatusAtExit name other possibility could be #expect(status: .failure)

1 Like
19

Although this second review is meant to be focused, renaming StatusAtExit to ExitStatus is perfectly reasonable. (I'm actually surprised it didn't come up already—I had originally meant it as a placeholder but forgot to change it for the initial review!)

6 Likes
20

Unfortunately, that shortens to:

#expect {
  var taco = Taco()
  taco.isDelicious = false
  eat(taco)
}

Which we've previously agreed isn't a good idea.

Additionally, "crashes" is very ambiguous. I think of a crash as when the code unintentionally exits. But how do you actually measure the unintentionally of an exit? If we want to define a crash as "exits with a non-zero exit code, or due to a signal raised", then we should come up/use more precise terminology to represent that concept.

1 Like