Test much slower (and faster!) using parameterized tests

Related Projects swift-testing
1

I have an existing XCTest that:

  • Reads in a text file with 512 lines of short strings
  • Splits the string and iterates over each line in a for loop
  • Tests that the string is processed as expected

The test takes about 0.231 seconds to run.

I moved this to be a parameterized test in Swift Testing with the following changes:

  • Reading the text file and splitting the string happen in a static fixture method
  • The resulting arguments are passed to the parameterized test

The migrated test took an average of 3.383 seconds over five runs, a good deal slower.

And after running the test Xcode remains unresponsive for an average of 16.218 seconds waiting to 'acknowledge that testing finished'.

This is a Swift package building and testing in Xcode 16 beta.

I have made sure the autogenerated test plan has the "Execute in parallel (if possible)" checkbox checked. (Otherwise the test takes about 23 seconds).

This is on a M3 Max MacBook Pro, Xcode 16 beta, macOS 14.5.

Is this amount of performance hit expected moving an existing test to parameterized tests? (Both the longer test itself and the Xcode delay at the end)

Is there anything I can do to make it faster (possibly turn off logging of each parameter test to the console unless it is a failure?)

Note that migrating the test from XCTest to Swift Testing directly, without a parameterized test, the time is in the same .21 second range and there is no Xcode pause at the end of the test.

So, I do have an easy workaround, but unfortunately it is "keep using loops inside the test". (EDIT: a second, better workaround is to run the package tests using swift test instead of in Xcode.)

EDIT: Moving the post to the correct category.

EDIT 2: TL;DR It appears that how the tests are run for a package in Xcode may be causing the slowness.

Running the same test using swift test --enable-swift-testing in the first Xcode beta results in an average test time of 0.065 seconds. Definitely faster than the XCTest loop version of the test!

8 Likes
2

Hi @James_Dempsey! Thank you for continuing to explore parameterized tests and sharing your experience. This is definitely a workflow we encourage and we want to ensure It performs well.

Within swift-testing itself, there should not be a significant difference between the performance of a single test with a loop and a parameterized test. When parallelization is in use, the latter will generally finish faster, but roughly the same total amount of work is performed in either case. In my experience, there's not much overhead involved in using parameterized testing — at least within the testing library itself.

The tool being used to invoke the tests can sometimes introduce additional overhead however, and I suspect this may partly explain what you're seeing. I attempted to reproduce your example using Xcode 16 Beta and I was able to observe some slower performance with a large number of arguments. I filed a bug for the relevant team to investigate this further.

I'll also note that the console logging is not likely to be the source of slowness. Running the tests using the much simpler swift test command still logs each argument to the console, and it finished in 0.008 seconds on my machine (albeit for a trivial test which doesn't do much work).

1 Like
3

It’s unfortunate swift-testing has inherited or recreated XCTest’s bottleneck here. It’s always been far faster to loop test logic within a test method than to try and repeatedly invoke a test method. You can see this using Xcode’s repeat test functionality. When I needed to invoke logic a million times to narrow down a threading issue, repeated invocations would take hours where doing it directly would take minutes. I hope swift-testing can remove this overhead so larger scale testing is actually viable.

6 Likes
4

You're very welcome. I find Swift Testing to be very nicely designed and so it's the first thing I'm digging into post-WWDC. I appreciate your responses a great deal.

I'm glad to hear what I am encountering is not fundamental to swift-testing itself.

Do you know if using swift test as you describe is expected to work with the first Xcode 16 beta?

I am finding that when I run test with my package open in Xcode, Xcode finds both my swift-testing and XCTest tests.

When I run swift test with the exact same package, the swift-testing tests do not get run.

5

It's a known issue in Xcode 16 Beta 1, per the release notes. Pass --enable-swift-testing explicitly for now.

3 Likes
6

Thank you, and my apologizes, I should have checked the release notes.

Running the same test using swift test --enable-swift-testing in the first Xcode beta results in an average test time of 0.065 seconds.

Definitely faster than the XCTest loop version of the test which took about 0.231 seconds.

I hope the issue in running the tests via Xcode can be resolved. The difference in results (3.383 seconds with Xcode vs 0.065 seconds with Swift PM) is quite significant.

I've updated the title of this thread and added a TL;DR to the original post.

Thank you for your responsiveness and patience @smontgomery and @grynspan.

2 Likes
7

Would you mind sending Apple feedback via Feedback Assistant about this issue? Please make sure to attach the specific package/project you're using that's showing the issue (if you can!) It'll help us narrow down where the slowdown is in Xcode.

1 Like
8

I am traveling for the next few days, but will file when I am able.

1 Like
9

I have filed this as FB13986710: Parameterized Swift Testing test much slower in Xcode than using swift test.

I've attached an example package to the feedback that demonstrates the issue.

5 Likes