Standardise `swift test` across platforms

This has been bought up in various forms over the last couple of years, but it's worth revisiting as it's been a stumbling block for several people with the release of the Swift for VSCode extension.

Currently, when running swift test on most platforms, it uses SwiftPM's test discovery features to find tests and execute them. This works well for most cases, though there may be a few edge cases that are yet to be solved (generics and extensions come to mind).

However, on macOS, the behaviour is completely different. macOS uses the Objective-C runtime to discover the tests and may produce different results compared to other platforms. This provides an inconsistent experience for Swift users and something that should be solved.

I'm proposing that the default behaviour for all platforms for swift test to be consistent and use test discovery. On macOS, XCTest will need to be added to the Swift toolchain you can install from swift.org. This will make it consistent with all other platforms and should also resolve the problem of needing Xcode to be installed to be able to test Swift packages on macOS.

I propose we add a new command line flag to enable the old behaviour for those that need it, but stick to using SwiftPM by default. This makes cross platform work easier, improves Swift across platforms and makes for a much better user experience. This does break the consistency with Xcode, however Xcode links Apple-platform-only frameworks like UIKit already, so that consistency is already there. xcodebuild would still use the Objective-C runtime.

Edited to add XCTest to the macOS Swift toolchain

40 Likes

XCTest is not part of the Swift toolchain on macOS, so I don't think this would change.

SwiftPM also integrates with Xcode (or, Xcode integrates with SwiftPM), but it is part of swift.org toolchains on macOS, isn't it?

One of my computers has Xcode 13.2.1, and I just checked:

% xcrun -toolchain swift swift-build --version
Swift Package Manager - Swift 5.6.0
% xcrun swift-build --version 
Swift Package Manager - Swift 5.5.0

Why couldn't XCTest do the same thing?

I guess part of this pitch then is to bundle XCTest in the Swift toolchain like it is on other platforms

4 Likes

Strictly consistent swift test behavior across platforms, including both logic and dependencies, would be very welcome. The issues listed here bit me hard last time I tried to create a portable Swift exercise for students using a mix of Mac, Windows, and Linux.

3 Likes

Is there any reason the XCTest library can not be included with the Swift toolchain on macOS?

Developing for IDEs other than Xcode would be much simplified if the initiating of tests and the output was consistent across platforms. On top of the test discovery behaviour there are various other inconsistencies

  • Building Linux tests generates an executable which links in the XCTest library while macOS tests are run via the XCTest executable that comes with the Xcode install.
  • Test results are output to stderr on macOS, while they are output to stdout on linux
  • When running tests with the thread or address sanitizer macOS needs to set DYLD_INSERT_LIBRARIES environment variable to insert sanitizer libraries while linux doesn't

If we could have a flag that switched between generating test executables that use a XCTest library (bundled with swift install, consistent with non macOS platforms) and the current method of using Xcode XCTest executable that would simplify things considerably when doing cross platform work. Also the non macOS setup is just easier to work with.

1 Like

I don't know the answer to this, but if adding XCTest to the toolchain on macOS is to be part of the pitch, I would strongly encourage making it explicit. That certainly wasn't what I expected from the description. Hopefully an XCTest expert can weigh in.

1 Like

I've edited the pitch to make this explicit

1 Like

The lack of support in the Swift toolchain for testing on macOS without installing Xcode is something that really ties using Swift as a programming language to Apple's own proprietary, closed-source software like Xcode. (In a way ofc compiling swift on a mac requires a macOS which is also closed-source but I think my point still stands)

As long as Swift isn't independent from Apple's proprietary technology it definitely will not gain the momentum Apple touted they wish it would. Decoupling the swift toolchain from XCTest (or making XCTest available outside Xcode) would be a big step in the right direction.

7 Likes

I think there's another one to pull in โ€” @compnerd do you think it's reasonable to distribute XCTest with the toolchain, so we can get rid of the strict layout convention on Windows the sooner day?

No, I don't think that is really reasonable.

XCTest is built for the host that you are building for, not the host that the toolchain is built for. This is problematic as it complicates the builds for both the Windows ARM64 and Windows X64 toolchains. Furthermore, it complicates other things as well. There are questions around ownership of the files as Windows does allow multiple parallel SDKs to be installed (not to mention architecture variants are split out as well). Overall, I think that the current structure is the desirable structure. I have even been experimenting with the host SDK being unavailable for a cross-compilation target only.

@0xTim this discussion made me think about this some more, I wonder how hard it is to write a 3rd party swift test runner? Discovery of tests via swiftsyntax, a runner that runs the tests, track asserts and produces a report ... it might as well not be compatible with XCTest at all, it could just be a SwiftPM plugin that interested cross-platform packages adopt. What do you think?

Test discovery using symbol graphs would be better than using SwiftSyntax, because you may need semantic information about types that SwiftSyntax can't provide. For example, if your test runner uses class-based test suites, you can't always detect test classes that come from inheritance hierarchies from syntax alone.

For Bazel, I implemented the symbol-graph-based approach to add support for XCTest discovery to the swift_test rule. I don't believe it's been merged into the main branch yet, but the implementation of the tool can be seen here: rules_swift/tools/test_discoverer at upstream ยท bazelbuild/rules_swift ยท GitHub. It's nothing too fancy; it basically does what SPM does to generate the runner, just using symbol graphs instead of the index store.

In my vision of an ideal world, test runner support in Swift Package Manager (including XCTest support) would be plug-in-based where SPM merely extracts the symbol graphs from the modules it built and hands them off to the appropriate plug-in that generates the runner sources, and then SPM compiles and runs those. SPM should also not have any XCTest-specific awareness about running the test; a "test" should just be an execution that returns 0 for "everything passed" or non-zero for "something failed", and the runner would generate its output in a commonly accepted format (to the console, xUnit/jUnit-style XML, etc.).

5 Likes

Oh, that's great, I like that direction โ€” getting the symbol graph instead will also give the parsed documentation (the way we did in docc) and developers could use that to add meta-information to test cases a-la types-in-comments in typescript or the way the swift team does that for the compiler tests.

As for the runner, re-implementing the XCTest asserts isn't much trouble, I had few of them running in my play-ide in few hours here: https://twitter.com/icanzilb/status/1486728440350789647

Long story short, I think that if "standardizing" swift test across platforms isn't something Apple would like to do, a 3rd party test runner isn't far fetched if there are few interested companies that need one.

Definitely gonna like this idea. If possible we can have a standardized test runner interface and related SwiftPM options instead.

A test runner interface is an interesting idea and would make SwiftPM far more extendible but is out of scope for this pitch. It would be nice to get what we currently have working consistently :sweat_smile:

2 Likes