Nope. Could be a signal code. Could be an exception code (Windows only.) Could be the code you wrote. Could be a secret code (encrypted, of course.)
I really can't see us shipping this interface without some parameter label in that position. Let me offer a couple of potential alternatives:
exitsReporting:
reports:
processExitsWith: (or processExitsReporting:)
spawnedProcessExitsAndReports: (hah no)
I am open to more suggestions for alternatives to exitsWith that keep the same basic grammatical position (i.e. conjugated verb with an object).
Our established naming pattern for these macros is that, if there is a trailing closure, the macro's labels spell out something like "expect [the following closure] does xyz". In this case, we'd read that line as "expect the following closure exit failure" (or "expect the following closure exit is failure").
Consistency within a library is often more important than consistency across libraries, because the former comes up much more often (or, er, more consistently?) when you're using that library. Hence why I'm leaning back on exit vs. exits.
The existing precedent for expecting the closure to throw an error is spelled #expect(throws: MyError.foo) { ... }
This is highly non-fluent English.
It reads as âexpect verbs object the subjectâ. For example, âexpect eats honey the bearâ. It is possible to interpret this in a Yoda-like manner as âexpect that the bear eats honeyâ, but that is emphatically not what is written.
And yet, it is absolutely the correct spelling for that API.
⢠⢠â˘
Programmers are perfectly capable of reading #expect(throws: MyError.foo) { ... } and understanding it to mean âexpect that the following closure throws MyError.fooâ.
Similarly, programmers are perfectly capable of reading #expect(exit: .failure) { ... } and understanding it to mean âexpect an exit of .failure from the following closureâ.
This is much closer to fluent English. The words are in the correct order, and only a few semantically-empty connector words have been omitted.
⢠⢠â˘
In the throws: version, the label is a transitive verb. In the exit: version, the label is a noun. This is not an inconsistency, but rather a different grammatical construction.
It would be inconsistent to use the label exits: with an âsâ, because that would read as a transitive verb like throws:, and thus give rise to the incorrect interpretation âthe closure exits the argumentâ.
But using a noun has no such issue. Nobody would interpret exit as a verb in that position, it is clearly a noun. And as a noun, its argument answers the question âWhich exit do we expect from the closure?â
⢠⢠â˘
The purpose of the API design guidelines is to guide API designers toward spellings which promote clarity and ease of understanding. The overarching goal is to make it easy for readers and writers of code to understand what that code does.
In the case of #expect, the natural thing for it to say is what we expect:
In the no-label version, it is a boolean expression we expect to be true.
In the throws: version, it is what we expect the closure to throw.
In the exit: version, it is the type of exit we expect from the closure.
These are all different, and they should each be spelled in the most intuitive way for programmers to understand. There is no sense in contorting the exit: label into something more verbose and less grammatical, just to try to use the same sentence structure as another API.
Thanks for the feedback! We won't be proposing exit as the label for this parameter. I'm still interested in alternative suggestions for labels though!
reports: as the label here might maybe satisfy folks?
It doesn't repeat the word exit when used with .exitCode();
It's sufficiently distinct from normal operation that it wouldn't be confused with something like XFAIL;
It doesn't use with since it takes a direct object (whereas "exits" takes an indirect object and so needs an adverb/adposition);
It's a verb, which is consistent with the established pattern; and
It reads reasonably fluently, e.g. "expect [this closure] reports signal SIGINT".
However, "report" is not AFAIK an established term of art in the relevant literature. I picked it on somewhat technical grounds (i.e. the kernel observes the process exiting and reports its status back to the parent process), so we might want to break out the thesaurus and bikeshed alternatives to it too. A good source of ideas might be the documentation for wait(2) or exit(3).
For what it's worth, this is generally true of most functions in Swift that take trailing closures in addition to other arguments, so I think the Yoda-speak in particular is just par for the course. If I had total control over the language (and we can all agree it's a good thing I don't!) I might want to be able to structure a function call so that the trailing closure syntax can be used mid-argument-list, and then you could write:
await #expect {
// ...
} (reports: .success)
(Or some other way to spell it that puts the closure in the grammatically non-Yoda position.)
It could be, but it would be surprising to me to hear someone saying "expect this process to exit with code XYZ" and have XYZ be something other than an exit code.
Since there's going to be extra context at the call site, it's not like ".code" is going to be interpreted in isolation. For example:
Feels like it's hammering the "exit" point a little too much.
FWIW, reports doesn't suggest to me that there's an exit or any sort of process involved, so IMO it's much worse at describing what the API does than #expect(exit: .failure).
Except, perhaps, for the .exitCode case, like:
#expect(reports: .exitCode(EX_IOERR))
But in that case I'd argue that reports: doesn't convey any information that isn't already there from either #expect or .exitCode.
Is there a reason for only allowing verbs in Swift Testing macros, other than being consistent with the (relatively few) macros already there? What does this consistency achieve? (Not in general, but this particular enforcement of this pattern). A noun after #expect is as natural as a verb, perhaps more.
I was writing something else about #expect + exit:, but I don't think I can argue my point any better than @Nevin already did. So I'll just say that I couldn't agree more.
Actually, to match the janky throws error parameter, I think we still have to throw out the idea of ExitTest.Condition, but have success be modeled as Never, adopting an ExitFailure protocol.
Overall +1 on this feature. This is solving a real need in the ecosystem. Besides allowing to test preconditions it also enables us to have process isolation for tests. Without nit picking the name of the proposed feature here too much I see them more like subprocess tests.
I read through the feedback regarding the naming of this parameter. I sympathize with some of the previous feedback around making it clearer that the closure is running in a subprocess. Additionally, the Subprocess review arrived at using TerminationStatus for this concept. Staying consistent across the ecosystem is in my opinion very important. What about #expect(subprocessTerminatesWith: ...)? To me this reads very fluently Expect (this closure running in a) subprocess terminates with ....
Lastly, I do agree with the previous feedback around using compiler guards for this API. Having worked on libraries like swift-nio that support multiple platforms, we learned that putting APIs behind compiler conditions that are based on platforms is only pushing the problem one layer up. In this case, it means that any cross platform package that uses exit-tests would need add compiler guards around those tests. In my opinion, the API should be available on every platform but exit-tests should be implicitly skipped on non-supported platforms.
The âsubprocessâ aspect applies to all parts of this APIâas your sounding out of the API shows, it is just as salient to the closure as the termination status. So if âsubprocessâ is important enough to be in the spelling of this API, then the naming guidelines would suggest it should be #expectSubprocess(...).
With TerminationStatus, the cases are named exited and unhandledException, and thereâs isSuccessâso at the use site, if using a similar type, the âverbâ part would be subsumed by that and wouldnât require a label: #expectSubprocess(.isSuccess) { ... }.
I suspect we want to keep the interface unavailable on platforms that don't support it, otherwise developers may think their tests are passing when they simply aren't running at all. We could lower the diagnostic to a warning rather than an error (or we could record an Issue at runtime), although I'm not sure if that's much better than an error and it could go unnoticed in CI. Either way, I'll raise this with the Testing Workgroup at our next meeting.
subprocessTerminatesWith generally meets our requirements for this API:
We chose "exit" over "terminate" here for rather weak reasons and could switch easily enough. I'll make sure to raise this question with the Testing Workgroup when we next meet.
The word does help to clarify to a reader that the code is going to run in a subprocess. But the fact we spawn a subprocess is a (necessary) implementation detailâthis interface is meant to test that some code causes the process to terminate in a specific way, not to test that a subprocess can be spawned.
I'm not sure how that affects the naming calculus, but given that Swift Testing intentionally keeps to just two expression macro names (#expect and #require) for documentation and discoverability reasons, there's some wiggle room in where we put the opening parenthesis. Changing the naming policy for macros in Swift Testing is beyond the scope of this proposal, but that policy is not set in stone; we'll make sure to discuss it in the Workgroup.
TerminationStatus.isSuccess is an instance property, so you wouldn't be able to write #expectSubprocess(.isSuccess) anyway.
Regardless⌠we aren't able to directly use the enum from swift-subprocess anyway, so we have to create our own enumeration, and I don't think we need to exactly preserve the names of the cases in our own code if they're not ergonomic. (Besides, unhandledException isn't even correct as a signal is not the same thing as an exceptionâI'll send the Foundation team a bug report about that.)
High level, very much +1 from me. To echo the proposalâs introduction, this is a very frequently requested feature and has historically been asked for often with XCTest too, so Iâm very excited to see it being proposed in Swift Testing. A few comments/questions below:
Optional Result properties
The ability to observe the standard output and standard error streams of an exit test subprocess is very useful, and I agree it should be opt-in the way itâs currently proposed. I notice the corresponding properties on ExitTest.Result named standardOutputContent and standardErrorContent, respectively, are not Optional types and instead, if the test author did not opt in to observing one of those streams, the array is empty. Thereâs a chance that could be confusing or surprising when a test author first attempts to inspect one of those properties if they donât know they need to opt in to observing them.
This behavior is documented adequately, but still I wonder if those two properties should be Optional to avoid ambiguity about whether a stream wasnât observed or whether it actually had no content.
Future platform support
Thank you for clarifying the platform support. Itâs good to see that this is implemented with enough flexibility that it could, in theory, be expanded to support other mechanisms of spawning processes in the future which would make the API available in more places.
If this proposal is accepted, and later it becomes feasible to make these APIs available on more platforms with the same interfaces and behaviors which are described in this proposal, then I think it would be ideal for that future platform expansion to be âpre-approvedâ such that it would not require a pro-forma Evolution proposal. I doubt there would be much resistance to such expansion, and even suspect making such a change without a proposal would be unobjectionable, so this is just a suggestion to make it explicit in the proposal that expanding platform availability is acceptable as long as the interface and behavior are the same. (Iâm curious if thereâs other Swift evolution precedent here.)
Equatable conformance
Are either, or both, the nested StatusAtExit or Condition types intended to conform to Equatable? I would expect the former to have such a conformance, at least, to allow validating its value in the âouterâ test, but I didnât see such a conformance stated in the proposal. If it was intended but unstated, could that be added explicitly? (I do support the change made during the pitch phase to separate those concepts!)
We could make them optional without any technical difficulties. My thinking here was that it would be an extra bit of boilerplate if you do ask for them, which I expect will be the common case if you are observing the result. If you don't ask for them, you are probably going to discard the result.
Note the "alternatives considered" section discusses returning a tuple matching the inputs requested; this unfortunately runs into type checker limitations specific to macro expansion, otherwise that might be a more ergonomic choice here.
Yes, I would expect new platform support to not require separate Swift Evolution threads (I'm not even sure what there would be to comment on! "I don't want to see this on Wasm!" is a pretty short threadâŚ) I will make sure to update the proposal to make it explicit.
I'm not aware offhand of any specific examples of this sort of thing in other Swift Evolution proposals, but it's a long list.
Yes, StatusAtExit is Equatable. It should be listed in the proposal; the omission is a typo. Condition is not Equatable because the == operator would not be transitive (see the alternatives considered section.)
I'm still interested in alternative suggestions for labels though!
