SwiftPM has long run XCTest tests serially by default, which I usually override with the --parallel flag. However, it runs the new Testing tests in parallel by default, so if a codebase uses both, as many might be transitioning to now, SwiftPM will by default run all the XCTest tests serially, then run the Testing tests in parallel.
I don't much care which is chosen as the default- I'd opt for making both parallel, as I have not seen the XCTest issues mentioned and always pass the --parallel flag to the mostly XCTest tests I run- but I think it doesn't make sense for SwiftPM to do different things for the different tests like this.
Perhaps the members of the new Testing workgroup can chime in here, and if members of the community agree with me, you can like this post.
Another advantage of parallel tests is that they promote more test isolation.
My concern with the proposal is that most likely many XCTest test suites either explicitly or implicitly rely on the current serial behaviour. So in a way, this would be a source breaking change. And therefore I don't know if changing XCTest behaviour that has been with us for so long this late in its lifecycle is a good idea.
So for me, I'm not yet pro or con this proposal and interested to hear more feedback.
The reason we[1] did not make this change when introducing Swift Testing is that it absolutely does cause a large number of existing tests written with XCTest to fail. (Yes, we've checked.)
Even though it is asymmetrical, I am not sure why it is a problem?
One old testing harness operates in one way, the new in another - hopefully we soon will only have the new - parallel as default is really good I think too, provides faster turnaround out of the box...
I agree on principle it would nice if XCTests could be run in parallel by default. But I am also fairly confident that enabling this for existing tests would break a significant portion of them. We're not investing very heavily in XCTest generally since it's considered legacy at this point, so I'm not sure it would be worth the effort to undertake a community-wide transition like this. I think that energy would be better spent encouraging usages of XCTest to transition to Swift Testing, and fixing any parallelization blockers that are encountered along the way.
I agree with @hassila that it's not necessarily a problem for the behavior of these two testing libraries to mismatch each other in certain ways. Thinking long term, we might eventually try to add support for integrating with other testing libraries or systems—potentially even ones not written in Swift—and it's not a guarantee that every testing system in the world can or should run in parallel by default.
Performance testing is another example to consider: if/when Swift Testing gains some level of support for performance testing, it may be deemed appropriate for the subset of Swift Testing tests which are marked as performance-related to not be run in parallel, to more accurately measure performance. (Perf testing is a big topic and I'm not trying to take this thread on a tangent, but just offering it as an example.) But if we did that, we'd be intentionally creating another scenario where the parallelization behavior differs based on the nature of the tests.
Despite my opposition to changing the default behavior, I do think it would useful to add a mechanism to the SwiftPM manifest (Package.swift) format to explicitly declare that a .testTarget should always run in parallel, including its XCTest tests (if it contains any).
This would remain completely opt-in, preserving existing behavior by default. The main benefit is that users would not need to remember to pass the --parallel flag to swift test, or know whether it is safe to do so in advance, because test targets with that setting would automatically be parallelized. Plus, having it be a target-level setting means it would be more granular, so any test targets which aren't suitable for parallelization could remain serialized.
Conceptually, I think it would be best expressed as a tri-state setting like:
enum ParallelizationBehavior {
/// Each testing library should use its default behavior.
/// (Swift Testing: true; XCTest: false).
/// This is the default parallelization behavior.
case automatic
/// Enable parallelization whenever it is supported.
/// (Not all testing libraries may support parallelization, or may
/// only support it on certain platforms or environments.)
case enabledIfSupported
/// Completely disable parallelization, even when it is
/// possible to support it.
case disabled
}
// Usage example:
.testTarget(
name: "MyTests",
...
parallelization: .enabledIfSupported
)
The last place I am blocked is one of my open source macro packages… we support the 5.10 toolchain and we also support swift-syntax 510.
It looks like there is more support for Swift Testing macros from the 600 release… but that would then lead to "two tests" if I still want to support the 510 release.