Hi! Can anyone help me understand more about how swift-testing plans to handle controlling for tests that are unsafe to run in parallel with each other? There seems to be a page in the documentation[1] that is maybe a WIP placeholder? I don't see many clues here (maybe those clues are coming soon)?
I have a situation where a set of tests needs to use a test-double spy that adopts a protocol with a static function as a requirement. In a "serial" world… I can spy with a static variable to save (spy) on parameters that are passed in. In a "parallel" world… you could see why that pattern breaks down (without adding locking or other protection).
My tests currently "pass" with probability one… but this seems inconsistent with what I am reading (that tests run in parallel by default and my test should eventually crash or throw with undefined behavior from unsafe global state mutation).
You'll use the .serialized trait to mark suites or parameterized tests whose content needs to run serially. My colleague @Dennis is preparing an API pitch for this trait in the near future. Keep an eye out.
For more information about the trait (which is available with Xcode 16 Beta 1), watch Go further with Swift Testing. In that video, I talk about when you might want to use .serialized and how it affects (or doesn't affect) your test runs.
To add more information to this, you can use the .serialized trait today in Swift 6.0. However, to use it you must add this to your import for the Testing:
@_spi(Experimental) import Testing
Also, you need to build and run your tests with the --enable-experimental-swift-testing flag as well. This at least applies for Linux platforms and the open source Swift 6.0 toolchain. It may be enabled by default already in Xcode 16 Beta (I haven't checked).
This is something I'm using even in the experimental state for now just to get some tests to run serially. I do hope they will eventually make this official since it is incredibly important and useful to have this sort of control over tests.
I'm afraid you have that backwards. --enable-experimental-swift-testing is only required right now on macOS, not Linux. On Linux, Swift Package Manager is able to detect your dependency on the Swift Testing package, while on macOS Swift Testing is included with Xcode and you don't need to write out the dependency in your package (which means that Swift Package Manager can't detect it.)
We'll be adding Swift Testing directly to the Swift 6.0 toolchain in the near future, at which point you won't need to specify a package dependency nor will you need to specify --enable-experimental-swift-testing on any platforms.
As I wrote, my colleague is working on an API pitch to do exactly that.
@grynspan Hmm… I migrated a bunch of tests in my project from XCTest to swift-testing. Those tests (passing from XCTest) depend on shared mutable state (like static properties). I see no reason why those tests should pass when run concurrently with each other… but I see no failures from swift-testing. Could you think of any reason why Xcode_16_beta_2 would default to serialized as of today?
It looks like parallel testing is (currently) opt-in from SwiftPM. If we know that parallel testing will (eventually) be opt-out… I guess that means we should expect for that value to flip before 6.0 ships?
@Dennis would you know if there was any kind of potential plan to support the ability for a testTarget (from a package) to support opting-out of parallel testing across the target? I understand that we are planning to add support for suites (and individual tests) to opt-out of parallel testing… but would there also be an ability to force that preference across the whole target (which would then have precedence over the shouldRunInParallel flag passed at command-line)?
Parallel testing of XCTest code is off by default. Parallel testing of Swift Testing code is on by default. I know this is confusing when looking just at the SwiftPM source, but in effect Swift Testing does not use the default value of shouldRunInParallel.
As of right now, Swift Testing is responsible for re-parsing the command-line arguments passed to swift test. In a future release, I hope to clean that up and have the parsing only done once, but that's how it works in Swift 6.
I have observed today that my Swift Testing tests with multiple @Suite(..., .serialized) test suites will fail nevertheless when I run them in Xcode - unless I set the Parallelization option explicitly to Disabled for the Test Plan.
The same thing happens when I run them in a Docker container, unless I a specify the --no-parallel option in the Dockerfile (which the man-page claims to be the default...).
Is this by any chance related? And expected?
P.S. this is a Vapor/Fluent app using a database, and most tests expect to run from a clean state, so should be serialized.
…then the tests in Suite1 will be run serially, and the tests in Suite2 will also be run serially, but a test from Suite1andSuite2 can still run in parallel. That is probably causing your failures since you can still be accessing the same database from two different tests.
In order to truly make all tests run serially you need a base suite that serializes, and then apply that suite to all tests:
Now all of the tests nested in SerializedSuite (which includes Suite1 and Suite2) will be run serially.
However, it may also be worth looking into making it so that each test is provisioned its own unique, temporary database. That would allow tests to run in parallel without them trampling on each other.
I have applied @mbrandonw's suggestion to our codebase and that works, but it's not really a sustainable solution. Our CI pipeline runs tens of thousands of test cases and we rely on having multiple iOS simulators running in parallel. What we really need is a mechanism that runs suites serially on each simulator instance.
That has been the XCTest behavior that we have relied on for a long time: we have parallelization enabled at the Scheme level, with the understanding that it means "run on multiple simulators when available" vs. "run XCTest suites in parallel on individual simulators". Now we need the equivalent functionality from Swift Testing.
