Running tests serially or in parallel

Hi! Can anyone help me understand more about how swift-testing plans to handle controlling for tests that are unsafe to run in parallel with each other? There seems to be a page in the documentation[1] that is maybe a WIP placeholder? I don't see many clues here (maybe those clues are coming soon)?

I have a situation where a set of tests needs to use a test-double spy that adopts a protocol with a static function as a requirement. In a "serial" world… I can spy with a static variable to save (spy) on parameters that are passed in. In a "parallel" world… you could see why that pattern breaks down (without adding locking or other protection).

My tests currently "pass" with probability one… but this seems inconsistent with what I am reading (that tests run in parallel by default and my test should eventually crash or throw with undefined behavior from unsafe global state mutation).


  1. https://swiftpackageindex.com/apple/swift-testing/main/documentation/testing/parallelization ↩︎

1 Like

You'll use the .serialized trait to mark suites or parameterized tests whose content needs to run serially. My colleague @Dennis is preparing an API pitch for this trait in the near future. Keep an eye out. :slight_smile:

For more information about the trait (which is available with Xcode 16 Beta 1), watch Go further with Swift Testing. In that video, I talk about when you might want to use .serialized and how it affects (or doesn't affect) your test runs.

1 Like

To add more information to this, you can use the .serialized trait today in Swift 6.0. However, to use it you must add this to your import for the Testing:

@_spi(Experimental) import Testing

Also, you need to build and run your tests with the --enable-experimental-swift-testing flag as well. This at least applies for Linux platforms and the open source Swift 6.0 toolchain. It may be enabled by default already in Xcode 16 Beta (I haven't checked).

This is something I'm using even in the experimental state for now just to get some tests to run serially. I do hope they will eventually make this official since it is incredibly important and useful to have this sort of control over tests.

2 Likes

I'm afraid you have that backwards. --enable-experimental-swift-testing is only required right now on macOS, not Linux. On Linux, Swift Package Manager is able to detect your dependency on the Swift Testing package, while on macOS Swift Testing is included with Xcode and you don't need to write out the dependency in your package (which means that Swift Package Manager can't detect it.)

We'll be adding Swift Testing directly to the Swift 6.0 toolchain in the near future, at which point you won't need to specify a package dependency nor will you need to specify --enable-experimental-swift-testing on any platforms.

As I wrote, my colleague is working on an API pitch to do exactly that. :slight_smile:

3 Likes

@grynspan Hmm… I migrated a bunch of tests in my project from XCTest to swift-testing. Those tests (passing from XCTest) depend on shared mutable state (like static properties). I see no reason why those tests should pass when run concurrently with each other… but I see no failures from swift-testing. Could you think of any reason why Xcode_16_beta_2 would default to serialized as of today?

1 Like

Xcode's test plans have the option to globally enable or disable parallelization. You might have it disabled.

1 Like

Ahh… correct! I see it now. Thanks!

Okay thanks for the clarification! I thought this was required because the VSCode Swift extension automatically enables it. I stand corrected.

It looks like parallel testing is (currently) opt-in from SwiftPM. If we know that parallel testing will (eventually) be opt-out… I guess that means we should expect for that value to flip before 6.0 ships?

@Dennis would you know if there was any kind of potential plan to support the ability for a testTarget (from a package) to support opting-out of parallel testing across the target? I understand that we are planning to add support for suites (and individual tests) to opt-out of parallel testing… but would there also be an ability to force that preference across the whole target (which would then have precedence over the shouldRunInParallel flag passed at command-line)?

Parallel testing of XCTest code is off by default. Parallel testing of Swift Testing code is on by default. I know this is confusing when looking just at the SwiftPM source, but in effect Swift Testing does not use the default value of shouldRunInParallel.

As of right now, Swift Testing is responsible for re-parsing the command-line arguments passed to swift test. In a future release, I hope to clean that up and have the parsing only done once, but that's how it works in Swift 6.

2 Likes