An update about Swift Testing in the Swift 6 toolchain

grynspan · August 20, 2024, 10:09pm

Dearest Swift Testing adopters!

If you happened to download the 2024-08-19 main-branch toolchain, you might have noticed that Swift Testing is included! We're inching closer to adding Swift Testing into the Swift 6 toolchain, and I wanted to let you all know our plans for the Swift Testing package!

We’ll be creating our release/6.0 branch very shortly, in line with the other components of the Swift 6 release. Because Swift Testing will be in the Swift 6 toolchain soon, it won't be necessary for package authors to include it as a package dependency anymore.

So, shortly after the release of the Swift 6 toolchain, we will be creating a final pre-6.0 tag (0.99.0) of the Swift Testing package that deprecates the @Test and @Suite attributes. If you’re including Swift Testing as a package dependency, you’ll see a warning like this one when you use @Test or @Suite:

ExampleTests.swift:30:4 'Test' is deprecated: Swift Testing is now included in the Swift 6 toolchain. Remove your 'swift-testing' package dependency to silence this warning.

Oh no!

Don't panic! We aren't deprecating the package as a whole. Our intent here is to notify developers that are still using these early/prerelease tags of Swift Testing that they don't need to do so anymore. These deprecations won't be present in the version of Swift Testing included with the toolchain, nor will it be present on future tags corresponding to future prerelease or release versions of the Swift toolchain.

If you wish to continue using Swift Testing 0.x, you can modify your package manifest file to explicitly specify the tagged version you wish to use. Keep in mind that these early/prerelease tags of Swift Testing are not guaranteed to work with future Swift releases.

I still want to try out new features!

When we start working on the next version of Swift in earnest, you'll be able to get the latest version of Swift Testing by downloading a prerelease toolchain. If you want to try out new Swift Testing features on a release toolchain, you'll be able to add a package dependency such as:

.package(
  url: "https://github.com/swiftlang/swift-testing.git",
  from: "700.0.0-latest"
)

Where 700.0.0 would correspond to a hypothetical Swift 7 toolchain release^[1]. Note that doing so opts you back into a prerelease version of Swift Syntax as well.

This versioning strategy matches the one used by Swift Syntax. ↩︎

Finagolfin · August 21, 2024, 12:31am

Great, looking forward to using it.

Note that you'll want to add snapshot tags of this repo too, so those like me who build the toolchain from the source snapshot tags know what commit to use. For example, swift-foundation just started adding such snapshot tags, now that it is in Swift 6 also.

I believe Mishal is the guy to talk to about setting that up.

grynspan · August 21, 2024, 12:29pm

Mishal is in the loop and is helping us with our infrastructure/integration checklist.

Edit: Nightly snapshots are now being tagged.

vanvoorden · August 27, 2024, 7:08pm

https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/temporarygettingstarted

Note

If your package does not explicitly list the testing library as a dependency, pass --enable-experimental-swift-testing to the swift test command to ensure your tests are run.

Does this imply that we should now be able to run swift test without listing swift-testing as an explicit dependency?

Max_Desiatov · August 27, 2024, 7:32pm

Yes, this is the case already for Linux swift:nightly (not nightly-6.0 though) snapshots and toolchains shipped with latest Xcode betas.

grynspan · August 27, 2024, 10:06pm

(We're workin' on it!)

grynspan · September 17, 2024, 7:33pm

Now that Swift 6 is available, we've deployed 0.99.0 as described above.

sliemeobn · September 18, 2024, 8:15am

I love how "Don't panic!" is becoming the unofficial tagline of Swift 6.0

Also: Fantastic work on Swift Testing! It is a much appreciated breath of fresh air compared to the somewhat dusty and dated XCTest.

Josh_Wisenbaker · September 18, 2024, 1:41pm

The core team are some real hoopy froods that really know where their towels are.

MahdiBM · September 18, 2024, 6:07pm

Just created this issue. This is absolutely horrendous :

github.com/swiftlang/swift

Absolutely massive regressions on `swift:6.0-noble`

opened 06:05PM - 18 Sep 24 UTC

MahdiBM

bug triage needed

### Description ## Absolutely massive build regressions on linux, worsened by… swift-testing ### Environment * GitHub Actions * 8 CPU x 16GB RAM c7x EC2 Instance (we use [RunsOn](https://runs-on.com/)) * Migrating from `swift:5.10-noble` to `swift:6.0-noble` * 200K-300K LOC project * Updated to `// swift-tools-version:6.0` in `Package.swift` * All targets still have Swift 5 language mode enabled. * No actual tests migrated to swift-testing just yet. ### What Happened? * Our Tests CI started getting stuck after the toolchain update. * The CI machines were getting killed by AWS. Considering this has been a common problem specially in the Server Side Swift ecosystem, which also has been brought up for ages and numerous times, my first guess was that the machine is **running out of RAM** when building the Swift project. * After a lot of tries, I noticed a `c7x`-family machine with **64 CPU x 128 GB RAM (8x larger than before)** runs the tests as well as they were being run before, on swift 5.10. * My first guess was that maybe Swift testing is causing an issue, so tried `--disable-swift-testing`. * After that, our tests have been running on an only 2x larger machine, and the rest of the things are back to normal in tests CI. * With this bigger machine, it still almost takes as long as before for the tests CI to complete (we run tests in parallel with 64 workers). * So this means that not only Swift 6 has a significant build regression, but also swift-testing makes it go from `significant` to `absolutely massive`. * This is borderline unusable for the Swift-Server ecosystem. I hope this problem requires more specific situations to happen that what meets the eye. * Still trying things out with the deployment CI. even 2x CPU and 4x RAM isn't proving helpful even though I did throw the `--disable-swift-testing` flag into the mix. ### Reproduction Not sure. ### Expected behavior No regressions. Preferably even faster than previous Swift versions. ### Environment Mentioned above. ### Additional information _No response_

grynspan · September 18, 2024, 6:17pm

Thanks for filing the issue. I'm not sure that this announcement thread is the right place to share it though. Perhaps a new thread would be a better place to discuss?

MahdiBM · September 18, 2024, 6:28pm

ahh hmm just wanted to catch some eyes I guess. Otherwise we can have the discussion somewhere else like in the issue.