We are busy updating our libraries to support swift-testing, and the last bit of functionality we see missing from XCTest is a way to wrap a test's execution in some additional work. For example, in our SnapshotTesting library we'd like to be able to enable a "record" mode for a single test or for a full test suite.
It seems that CustomExecutionTrait will be that feature for swift-testing, but I haven't seen much information on it since April. Is there anything new to share about its status? Is it planned for the first official release of swift-testing, both as an SPM package and the version included in Xcode?
Hi @mbrandonw, thanks for your interest in CustomExecutionTrait and your patience on a reply.
Unfortunately, this experimental feature isn't going to make it into the first official Swift Testing release. We've been holding off on creating a release/6.0 branch, but are expecting to do so in the very near future, with the work to install Swift Testing into toolchains as the only noteworthy item still planned, apart from any critical bug fixes. (We haven't been transparent enough about release planning this summer; that's something that we need to improve for future releases.)
Looking beyond the initial release, this would be a great feature to stabilize and promote to API for a following release. And I do think it's close! For next steps, we were still hoping to:
Get feedback from experimental early adopters on how well the current interface serves the use case of applying pre- and post- actions to tests and suites. (It sounds like you have positive feedback already!)
One specific question I have is whether additional work is needed to allow a CustomExecutionTrait applied to a suite to execute its "before" and "after" logic around each child test, instead of only being able execute before and after the entire suite's execution.
Consider whether this functionality should be rolled directly into the Trait protocol rather than needing an additional one.
Bikeshed the spelling of whatever new interfaces we're proceeding with.
Begin considering what other kinds of custom traits we might want to provide in the future, and whether those future directions suggest adjustments to CustomExecutionTrait's design.
With those learnings in hand, we should be able to run a feature proposal to get things wrapped up. We'd love to get your feedback on any of the above, before we start writing that proposal!
From my experimentation it seemed like we were able to execute the work around each suite and each test function by using isRecursive = true. Does this behave differently from what you are describing above?
I don't have any concrete feedback about the functionality of the trait. It serves the exact purpose we need, and without it our testing tools are going to be a lot more cumbersome to use. Each individual test function will need to be wrapped in a withABC { … } kind of configuration function, or people will need to use class-based (and .serialized) suites where they configure global variables in the init/deinit.
If there's anything we can do to help things along please feel free to reach out.
Want to add a +1 that CustomExecutionTrait, exactly as it was (is!), was working well for me. I was using it to remove the boilerplate of setting some TaskLocals for the duration of a test.