One of the primary motivations for the trait system in Swift Testing, as described in its vision document, is to provide a way to customize the behavior of tests which have things in common. If all the tests in a given suite type need the same custom behavior, init and/or deinit (if applicable) can be used today. But if only some of the tests in a suite need custom behavior, or tests across different levels of the suite hierarchy need it, traits would be a good place to encapsulate common logic since they can be applied granularly per-test or per-suite. This aspect of the vision for traits hasn't been realized yet, though: the Trait protocol does not offer a way for a trait to customize the execution of the tests or suites it's applied to.
This proposal introduces API which enables a custom Trait-conforming type to customize the execution of test functions and suites, including running code before or after them.
I invite you to read the full proposal for more details, and add any comments or questions here!
We are really excited to have this feature in Swift Testing! We experimented with the @_spi version of this feature back when the Xcode 16 beta briefly allowed doing an @_spi import Testing. It is absolutely necessary to have this for any library using task locals for global configuration, and we have multiple libraries that use task locals.
One thing we noticed with the experimental version of this feature was that execute would be called multiple times for a single test. Has that issue been fixed in this newest re-imagining?
The execute(...) method is invoked once for the test its associated trait is applied to, and then once for each test case in that test, if applicable. So for parameterized test functions in particular, this method will be called multiple times for the same test: once for each of its cases.
Is that the behavior you're referring to, or is it something else? If you consider some aspect of the proposed behavior problematic, mind elaborating?
What’s the rationale behind naming these “Custom Test Executors” instead of just “Test Executors?” If the whole point of traits is to customize tests, I don’t feel like the additional “custom” adds anything.
That being said, this has made me realize we may want to refine the behavior of this feature when a trait which customizes test execution is applied to a suite and inherited by its children. As things stand, when one of these traits is applied to a suite and inherited by a (non-parameterized) test function in that suite, the execute(...) method will be called twice: once for the suite, before all its test functions and sub-suites, and then once for the test function which inherited the trait from its suite.
I think the most common usage scenario for this feature is to apply custom behavior per-test function, though. There are certainly times when a custom trait on a suite may need to perform a custom action once before all its child tests, but I think that's more rare and probably shouldn't be the default behavior since it could lead to accidentally duplicated set-up and tear-down work.
I have an idea about how we could modify the proposal such that by default, it would only apply to test functions and still avoid "unnecessarily lengthy backtraces" problem. It would still permit those rarer situations when you do need to perform "once per suite" behaviors, though, it would just require opting in. I'll experiment with that idea and update the PR and this thread if it seems viable. I'm curious what people think about it in the meantime.
This looks good overall, and it's good to see that it allows to wrap the test body, this way one could provide some task-local around a test which may be useful
I'm iffy about the type ending with "...ing", and maybe would be less weirded out with a CustomTestExecution name? I'd also re-consider if you really need the word Custom... in it. Swift also has custom executors, and we don't prefix them with Custom... even though every instance of such is a custom executor (at least today).
For a moment I was wondering if Executor could be confused with Concurrency executors, but as long as all names include customTestExecutor / testExecutor I think it's not a problem -- in theory you could add some other executor property there if you'd want then
That’s something that confused me at first too. I guess it might be fine since is not something user code will deal with often, but there was a bit of a clash in my mind for sure.
Yes sorry it wasn’t my intent to misquote you, just to point out that I (and others I’ve linked this proposal to) had the same initial impression. Apologies.
Thanks for the naming feedback, @j-f1! @ktoso mentioned that too.
I went back and forth on this myself, and am now convinced we don't need the word Custom as a prefix on these proposed API names. I've pushed changes to the PR to adjust those names accordingly.
I discussed the ingsuffix in a comment thread on the PR, and based on the API Design Guidelines I still think those suffixes are appropriate. In summary:
The new protocol is (now) TestExecuting because it describes one optional capability that a type (typically a Trait) can have. But the ability to execute tests is not necessarily its primary focus.
The associated type on Trait is (now) TestExecutor, because it's the type of a trait type's executor.
In particular: since only the associatedtype is suffixed "…Executor", and not the protocol, I think that should sufficiently avoid any potential confusion with Concurrency executors.
By accepting the test and test case, its default implementation is able to distinguish suites and test functions, and can ensure that by default, traits with custom behaviors only run once per scope.
For more details about this change, see my latest commit on the PR and the newly-added Avoiding unnecessary (re-)execution section in the proposal text.
