SwiftPM has long run XCTest tests serially by default, which I usually override with the --parallel flag. However, it runs the new Testing tests in parallel by default, so if a codebase uses both, as many might be transitioning to now, SwiftPM will by default run all the XCTest tests serially, then run the Testing tests in parallel.
I don't much care which is chosen as the default- I'd opt for making both parallel, as I have not seen the XCTest issues mentioned and always pass the --parallel flag to the mostly XCTest tests I run- but I think it doesn't make sense for SwiftPM to do different things for the different tests like this.
Perhaps the members of the new Testing workgroup can chime in here, and if members of the community agree with me, you can like this post.
Another advantage of parallel tests is that they promote more test isolation.
My concern with the proposal is that most likely many XCTest test suites either explicitly or implicitly rely on the current serial behaviour. So in a way, this would be a source breaking change. And therefore I don't know if changing XCTest behaviour that has been with us for so long this late in its lifecycle is a good idea.
So for me, I'm not yet pro or con this proposal and interested to hear more feedback.
The reason we[1] did not make this change when introducing Swift Testing is that it absolutely does cause a large number of existing tests written with XCTest to fail. (Yes, we've checked.)
Even though it is asymmetrical, I am not sure why it is a problem?
One old testing harness operates in one way, the new in another - hopefully we soon will only have the new - parallel as default is really good I think too, provides faster turnaround out of the box...
I agree on principle it would nice if XCTests could be run in parallel by default. But I am also fairly confident that enabling this for existing tests would break a significant portion of them. We're not investing very heavily in XCTest generally since it's considered legacy at this point, so I'm not sure it would be worth the effort to undertake a community-wide transition like this. I think that energy would be better spent encouraging usages of XCTest to transition to Swift Testing, and fixing any parallelization blockers that are encountered along the way.
I agree with @hassila that it's not necessarily a problem for the behavior of these two testing libraries to mismatch each other in certain ways. Thinking long term, we might eventually try to add support for integrating with other testing libraries or systems—potentially even ones not written in Swift—and it's not a guarantee that every testing system in the world can or should run in parallel by default.
Performance testing is another example to consider: if/when Swift Testing gains some level of support for performance testing, it may be deemed appropriate for the subset of Swift Testing tests which are marked as performance-related to not be run in parallel, to more accurately measure performance. (Perf testing is a big topic and I'm not trying to take this thread on a tangent, but just offering it as an example.) But if we did that, we'd be intentionally creating another scenario where the parallelization behavior differs based on the nature of the tests.
Despite my opposition to changing the default behavior, I do think it would useful to add a mechanism to the SwiftPM manifest (Package.swift) format to explicitly declare that a .testTarget should always run in parallel, including its XCTest tests (if it contains any).
This would remain completely opt-in, preserving existing behavior by default. The main benefit is that users would not need to remember to pass the --parallel flag to swift test, or know whether it is safe to do so in advance, because test targets with that setting would automatically be parallelized. Plus, having it be a target-level setting means it would be more granular, so any test targets which aren't suitable for parallelization could remain serialized.
Conceptually, I think it would be best expressed as a tri-state setting like:
enum ParallelizationBehavior {
/// Each testing library should use its default behavior.
/// (Swift Testing: true; XCTest: false).
/// This is the default parallelization behavior.
case automatic
/// Enable parallelization whenever it is supported.
/// (Not all testing libraries may support parallelization, or may
/// only support it on certain platforms or environments.)
case enabledIfSupported
/// Completely disable parallelization, even when it is
/// possible to support it.
case disabled
}
// Usage example:
.testTarget(
name: "MyTests",
...
parallelization: .enabledIfSupported
)
The last place I am blocked is one of my open source macro packages… we support the 5.10 toolchain and we also support swift-syntax 510.
It looks like there is more support for Swift Testing macros from the 600 release… but that would then lead to "two tests" if I still want to support the 510 release.
Correct! I was just pointing out that for engineers building macro packages there are two dimensions of constraints that are blocking them from migrating away from XCTest: the toolchain version and the swift-syntax version.
FWIW running swift package init --type macro still creates an XCTest target… not a Swift Testing target. And that template ships with a 6.0 toolchain requirement and a swift-syntax 600 requirement.
Note that I'm only talking about choosing the defaults for the OSS toolchain here: what Apple decides to do with your own proprietary builds doesn't concern me.
So what? That is what the --no-parallel flag is there for, for such legacy use.
Yes, sounds useful.
As for why I think a single default is best, defaults are for the average user, the recommendation of the Swift toolchain devs of what the casual or mid-level user should be using. Given the prevalence of multi-core CPUs and the parallel default already applied to Testing, I think that should now be parallel by default for XCTest too. As for splitting the defaults as it is now, this is confusing to that average user and unnecessarily slows down running their XCTest tests, that some tests are run in parallel and others aren't because no uniform default has been chosen.
One additional feature that would be useful is to be able to choose to apply parallelism to one type of tests and not another, ie to be able to pass --parallel=xctest --no-parallel=testing.
Realistically, we are stuck with XCTest for years to come, might as well make the experience consistent for the average Swift dev.
A change that fundamentally breaks existing code is not a change we take lightly. It's something we could consider for the next major version of Swift (i.e. Swift 7), but even then we would have to seriously consider whether the benefits of parallel-by-default for XCTest are worth the source breakage that would ensue.
We'll add this to the agenda for the next Testing Workgroup meeting and can discuss there.
This is a vast exaggeration, like Maarten above. This has nothing to do with source breakage, it is merely a different default that will break running some minority of XCTest tests that are run serially by default. Telling those expert users to stick --no-parallel in their CI config is no big deal.
We made a similar switch to --enable-test-discovery by default in recent years and I saw almost no complaint about it in these forums.
I'm sure it was, and in this case we have a much simpler solution: use --no-parallel.
You chose the wrong example: take a look at what that pull actually does. I was involved with that revert, so I know.
That transition was so successful that Max jumped the gun and tried to remove the --enable-test-discovery flag altogether. I pointed out in the linked issue there that the explicit flag itself was still needed because of some packages that unfortunately hard-coded errors that you should use it, ie from before when it was the default.
So that pull actually paints a picture of a transition so successful that we even tried to remove the --enable-test-discovery flag prematurely, because it was already successfully the default!
Hi @Finagolfin, thanks for bringing up this inconsistency across the wider testing ecosystem!
The @testing-workgroup discussed this at our last meeting and we came to the following conclusions:
While the inconsistency is unfortunate, fixing it will definitely be a breaking change. As @grynspan mentioned, there are actual, known tests that will fail in the ecosystem. And more broadly, it is not hard to imagine that there are tests out there making changes to the outside world (such as API requests) for which it would be necessary for those tests to run in serial in order to guarantee side effects from multiple tests do not interleave and interfere with each other. This is true even though XCTest runs parallel tests in separate processes. Not to mention that there may be tests that depend on the order in which the tests run.
And while the application of a --no-parallel would fix those tests, it is still a breaking change and that alone makes it a non-starter for making this change without a major release of Swift.
If we do not want to wait for a major Swift version to make some headway towards this problem, the workgroup agrees that @smontgomery's suggestion is the best path forward to resolve this issue. The new SPM configuration would allow setting parallel testing on a per-project basis, and the SPM template could be updated to apply the config by default.
This work is not planned for Swift 6.2, and the best way to get it moving forward would be for someone from the community to pursue those changes. It would need to start with a pitch for feedback and then a proposal with implementation.
And beyond any specifics of how to move towards a consistent behavior between XCTest and Testing, the community has not yet shown an interest in this change in great numbers. Consistency is generally great, but it's not clear that this is a huge win that many currently care about and worth the effort. The workgroup thinks more support would need to be corralled to make sure it's well understood what the ramifications are and if the suggestions above would be used by the community.
