# \[Pitch\] Custom Test Execution Traits **URL:** \ **Category:** Swift Testing\ **Tags:** testing, pitch, swift-testing\ **Created:** [September 30, 2024, 10:06pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055 "2024-09-30T22:06:17Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![smontgomery](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/smontgomery/32/965_2.png) [@smontgomery](https://forums.swift.org/u/smontgomery)\ **Post date:** [September 30, 2024, 10:06pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/1 "2024-09-30T22:06:18Z")
Hi everyone, One of the primary motivations for the trait system in Swift Testing, as [described in its vision document](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#trait-extensibility), is to provide a way to customize the behavior of tests which have things in common. If all the tests in a given suite type need the same custom behavior, `init` and/or `deinit` (if applicable) can be used today. But if only _some_ of the tests in a suite need custom behavior, or tests across different levels of the suite hierarchy need it, traits would be a good place to encapsulate common logic since they can be applied granularly per-test or per-suite. This aspect of the vision for traits hasn't been realized yet, though: the `Trait` protocol does not offer a way for a trait to customize the execution of the tests or suites it's applied to. This proposal introduces API which enables a custom `Trait`-conforming type to customize the execution of test functions and suites, including running code before or after them. I invite you to read the [full proposal](https://github.com/stmontgomery/swift-testing/blob/publicize-CustomExecutionTrait/Documentation/Proposals/NNNN-custom-test-execution-traits.md) for more details, and add any comments or questions here! Stuart ---
**Author:** ![mbrandonw](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/mbrandonw/32/349_2.png) [@mbrandonw](https://forums.swift.org/u/mbrandonw)\ **Post date:** [October 1, 2024, 1:41pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/2 "2024-10-01T13:41:14Z")
We are really excited to have this feature in Swift Testing! We experimented with the `@_spi` version of this feature back when the Xcode 16 beta briefly allowed doing an `@_spi import Testing`. It is absolutely necessary to have this for any library using task locals for global configuration, and we have multiple libraries that use task locals. One thing we noticed with the experimental version of this feature was that `execute` would be called multiple times for a single test. Has that issue been fixed in this newest re-imagining? ---
**Author:** ![smontgomery](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/smontgomery/32/965_2.png) [@smontgomery](https://forums.swift.org/u/smontgomery)\ **Post date:** [October 1, 2024, 2:05pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/3 "2024-10-01T14:05:48Z")
Appreciate the enthusiasm, @mbrandonw! > [@mbrandonw](#): > > One thing we noticed with the experimental version of this feature was that `execute` would be called multiple times for a single test. Has that issue been fixed in this newest re-imagining? The `execute(...)` method is invoked once for the test its associated trait is applied to, and then once for each test case in that test, if applicable. So for parameterized test functions in particular, this method _will_ be called multiple times for the same `test`: once for each of its cases. Is that the behavior you're referring to, or is it something else? If you consider some aspect of the proposed behavior problematic, mind elaborating? ---
**Author:** ![mbrandonw](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/mbrandonw/32/349_2.png) [@mbrandonw](https://forums.swift.org/u/mbrandonw)\ **Post date:** [October 1, 2024, 2:38pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/4 "2024-10-01T14:38:13Z")
I think I'm referring to something else. It was mentioned in [this post](https://forums.swift.org/t/how-do-i-apply-a-customexecutiontrait-to-a-suite/71317/2): > [@How do I apply a CustomExecutionTrait to a Suite?](https://forums.swift.org/t/how-do-i-apply-a-customexecutiontrait-to-a-suite/71317/2): > > (Be aware that there's a known issue in the implementation of `CustomExecutionTrait` that may cause it to execute at the wrong time relative to a suite.) When I would put `print` in the `execute` function I would notice it printed in an unexpected order. ---
**Author:** ![j-f1](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/j-f1/32/12287_2.png) [@j-f1](https://forums.swift.org/u/j-f1)\ **Post date:** [October 1, 2024, 3:25pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/5 "2024-10-01T15:25:40Z")
What’s the rationale behind naming these “_Custom_ Test Executors” instead of just “Test Executors?” If the whole point of traits is to customize tests, I don’t feel like the additional “custom” adds anything. ---
**Author:** ![smontgomery](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/smontgomery/32/965_2.png) [@smontgomery](https://forums.swift.org/u/smontgomery)\ **Post date:** [October 1, 2024, 3:58pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/6 "2024-10-01T15:58:07Z")
Ah, right. The known issue that was referring to was fixed in [Ensure that a `CustomExecutionTrait`'s teardown logic occurs \_after\_ nested tests run. by grynspan · Pull Request #526 · swiftlang/swift-testing · GitHub](https://github.com/swiftlang/swift-testing/pull/526). That being said, this has made me realize we may want to refine the behavior of this feature when a trait which customizes test execution is applied to a suite and inherited by its children. As things stand, when one of these traits is applied to a suite and inherited by a (non-parameterized) test function in that suite, the `execute(...)` method will be called twice: once for the suite, before _all_ its test functions and sub-suites, and then once for the test function which inherited the trait from its suite. I think the most common usage scenario for this feature is to apply custom behavior _per-test function_, though. There are certainly times when a custom trait on a suite may need to perform a custom action once before _all_ its child tests, but I think that's more rare and probably shouldn't be the default behavior since it could lead to accidentally duplicated set-up and tear-down work. I have an idea about how we could modify the proposal such that by default, it would only apply to test functions and still avoid "unnecessarily lengthy backtraces" problem. It would still permit those rarer situations when you _do_ need to perform "once per suite" behaviors, though, it would just require opting in. I'll experiment with that idea and update the PR and this thread if it seems viable. I'm curious what people think about it in the meantime. ---
**Author:** ![grynspan](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/grynspan/32/15315_2.png) [@grynspan](https://forums.swift.org/u/grynspan)\ **Post date:** [October 2, 2024, 12:31am UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/7 "2024-10-02T00:31:53Z")
I suspect we already have the building blocks to support these patterns, but I'm curious what you've got in mind. ---
**Author:** ![ktoso](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ktoso/32/44526_2.png) [@ktoso](https://forums.swift.org/u/ktoso)\ **Post date:** [October 2, 2024, 12:34pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/8 "2024-10-02T12:34:38Z")
This looks good overall, and it's good to see that it allows to wrap the test body, this way one could provide some task-local around a test which may be useful 👍 I'm iffy about the type ending with "...ing", and maybe would be less weirded out with a `CustomTestExecution` name? I'd also re-consider if you really need the word `Custom...` in it. Swift also has custom executors, and we don't prefix them with `Custom...` even though _every_ instance of such is a custom executor (at least today). For a moment I was wondering if Executor could be confused with Concurrency executors, but as long as all names include `customTestExecutor` / `testExecutor` I think it's not a problem -- in theory you could add some other executor property there if you'd want then 🤔 ---
**Author:** ![Alejandro\_Martinez](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/alejandro_martinez/32/821_2.png) [@Alejandro\_Martinez](https://forums.swift.org/u/Alejandro_Martinez)\ **Post date:** [October 2, 2024, 4:29pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/9 "2024-10-02T16:29:25Z")
> [@ktoso](#): > > I was wondering if Executor could be confused with Concurrency executors That’s something that confused me at first too. I guess it might be fine since is not something user code will deal with often, but there was a bit of a clash in my mind for sure. ---
**Author:** ![ktoso](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ktoso/32/44526_2.png) [@ktoso](https://forums.swift.org/u/ktoso)\ **Post date:** [October 2, 2024, 11:42pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/10 "2024-10-02T23:42:19Z")
That's some _very_ selective quoting right there 😅 My sentence is in support of the current wording: > [@ktoso](#): > > **For a moment** I was wondering if Executor could be confused with Concurrency executors, **but as long as**... ---
**Author:** ![Alejandro\_Martinez](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/alejandro_martinez/32/821_2.png) [@Alejandro\_Martinez](https://forums.swift.org/u/Alejandro_Martinez)\ **Post date:** [October 3, 2024, 6:48am UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/11 "2024-10-03T06:48:18Z")
Yes sorry it wasn’t my intent to misquote you, just to point out that I (and others I’ve linked this proposal to) had the same initial impression. Apologies. ---
**Author:** ![ktoso](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ktoso/32/44526_2.png) [@ktoso](https://forums.swift.org/u/ktoso)\ **Post date:** [October 3, 2024, 7:20am UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/12 "2024-10-03T07:20:48Z")
No worries, found it a bit funny how it made it sound the opposing opinion 😅 thank you for making sure your point comes across 🙂 ---
**Author:** ![smontgomery](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/smontgomery/32/965_2.png) [@smontgomery](https://forums.swift.org/u/smontgomery)\ **Post date:** [October 3, 2024, 4:19pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/13 "2024-10-03T16:19:34Z")
Thanks for the naming feedback, @j-f1! @ktoso mentioned that too. I went back and forth on this myself, and am now convinced we don't need the word `Custom` as a prefix on these proposed API names. I've pushed changes to the [PR](https://github.com/swiftlang/swift-testing/pull/733) to adjust those names accordingly. > [@ktoso](#): > > I'm iffy about the type ending with "...ing", and maybe would be less weirded out with a `CustomTestExecution` name? I discussed the `ing` _suffix_ in a [comment thread](https://github.com/swiftlang/swift-testing/pull/733#discussion_r1786486309) on the PR, and based on the [API Design Guidelines](https://www.swift.org/documentation/api-design-guidelines/#protocols-describing-what-is-should-read-as-nouns) I still think those suffixes are appropriate. In summary: - The new protocol is (now) `TestExecuting` because it describes one optional _capability_ that a type (typically a `Trait`) can have. But the ability to execute tests is not necessarily its primary focus. - The associated type on `Trait` is (now) `TestExecutor`, because it's the type of a _trait_ type's executor. In particular: since only the `associatedtype` is suffixed "…Execut **or**", and not the protocol, I think that should sufficiently avoid any potential confusion with Concurrency executors. ---
**Author:** ![grynspan](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/grynspan/32/15315_2.png) [@grynspan](https://forums.swift.org/u/grynspan)\ **Post date:** [October 3, 2024, 5:04pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/14 "2024-10-03T17:04:59Z")
What about `TestRunning` instead of `TestExecuting`? To avoid confusion with concurrency executors as discussed above. ---
**Author:** ![noremac](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/noremac/32/7222_2.png) [@noremac](https://forums.swift.org/u/noremac)\ **Post date:** [October 6, 2024, 6:13pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/15 "2024-10-06T18:13:34Z")
I gave the branch a shot this weekend. I don't have much to add other than: it's a super helpful addition and works exactly as advertised! ---
**Author:** ![smontgomery](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/smontgomery/32/965_2.png) [@smontgomery](https://forums.swift.org/u/smontgomery)\ **Post date:** [October 8, 2024, 7:49pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/16 "2024-10-08T19:49:25Z")
Following up about this: > [@smontgomery](#): > > I have an idea about how we could modify the proposal such that by default, it would only apply to test functions and still avoid "unnecessarily lengthy backtraces" problem. It would still permit those rarer situations when you _do_ need to perform "once per suite" behaviors, though, it would just require opting in. I'll experiment with that idea and update the PR and this thread if it seems viable. I have now revised the [implementation PR](https://github.com/swiftlang/swift-testing/pull/733) and [proposal](https://github.com/stmontgomery/swift-testing/blob/publicize-CustomExecutionTrait/Documentation/Proposals/NNNN-custom-test-execution-traits.md) to replace the `testExecutor` property of `Trait` with this method: ```swift func executor(for test: Test, testCase: Test.Case?) -> TestExecutor? ``` By accepting the test and test case, its default implementation is able to distinguish suites and test functions, and can ensure that by default, traits with custom behaviors only run once per scope. For more details about this change, see my latest commit on the PR and the newly-added [Avoiding unnecessary (re-)execution](https://github.com/stmontgomery/swift-testing/blob/publicize-CustomExecutionTrait/Documentation/Proposals/NNNN-custom-test-execution-traits.md#avoiding-unnecessary-re-execution) section in the proposal text. ---
**Author:** ![smontgomery](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/smontgomery/32/965_2.png) [@smontgomery](https://forums.swift.org/u/smontgomery)\ **Post date:** [December 20, 2024, 4:56pm UTC](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055/17 "2024-12-20T16:56:49Z")
This pitch has advanced to a proposal, and is now being reviewed in [[Proposal] Test Scoping Traits](https://forums.swift.org/t/proposal-test-scoping-traits/76676).