An update on swift-testing progress and stable release plans

Related Projects swift-testing
, , , ,
1

Hi everyone,

I wanted to give an update about swift-testing and discuss our plans for declaring the first stable version.

We announced swift-testing last September as a new open source package intended to rethink and modernize testing Swift software. In the announcement forum post, we publicized the GitHub repo and shared a draft Vision document outlining our philosophy behind the project and mentioning various design considerations and potential future directions. We received over a hundred replies to that post with thoughtful feedback and enthusiasm, so I'd like to thank everyone who participated there and has engaged with us on the project thus far.

Project status

Since its initial introduction, swift-testing has been considered experimental. We gave it this designation because we knew it was missing some important functionality and lacked integration with common tools like Swift Package Manager. It would take more time to deliver those pieces, but we wanted to begin soliciting feedback as those efforts got underway.

The project's experimental status was always intended to be temporary, of course; it was meant to discourage adoption in production codebases until the project was more stable and supporting work had landed, while prompting discussion about the core concepts. We have published several tags of the package so far (e.g. 0.7.0, the latest as of this writing), and have intentionally chosen pre-1.0 version numbers to reflect its unstable status.

Recent progress

In recent months we have made significant progress towards lifting the experimental label and declaring the first stable version:

  • SwiftPM integration: @grynspan has landed experimental support in Swift Package Manager for building and running swift-testing style tests, alongside any existing XCTests. This is included in recent Swift 6.0 development snapshot toolchains, and is enabled automatically in packages which declare an explicit dependency on swift-testing.

    Directly integrating this way renders the XCTestScaffold compatibility shim unnecessary, so there are fewer barriers to start writing tests and the console output is easier to understand. See the updated Getting Started documentation for the latest usage instructions.

  • Integration with other tools and IDEs: One of the project's goals stated in the Vision document is to integrate with IDEs and tools beyond SwiftPM. To support this, my colleague @dennis added experimental private APIs for receiving structured result data. These "snapshot" types, along with related private APIs for programmatically driving test execution, are intended to eventually facilitate direct integration with IDEs like VSCode—a topic we've begun discussing on the Forums.

  • Macro improvements: swift-testing relies on Swift macros in several ways, including for test function declaration using @Test. Since its introduction, there have been some known limitations with this macro's implementation which require information about the surrounding usage context to resolve.

    Thankfully, these limitations have now been resolved: swift-syntax introduced a new API for macros to retrieve their lexical context, which swift-testing has adopted, and we've now been able to remove workarounds and surface more helpful diagnostics.

These are in addition to many other refinements to expectation and parameterized testing APIs, documentation, and more.

Achieving stability

swift-testing is getting closer to production-ready. With that in mind, we intend to align the first stable release of swift-testing with Swift 6.0. This means that a Swift 6 toolchain and swift-syntax version 600 or later will soon be required. Using swift-testing to validate code built using the Swift 5 language mode will still be supported, however.

As we continue to stabilize the project, we will soon begin two efforts worth mentioning:

  • Vision document review: We aspire for swift-testing to succeed XCTest as the official, default testing solution in the Swift ecosystem. To that end, we plan to revise our draft Vision document based on feedback and lessons learned during the project's first few months and submit it for review by the Swift Core Team. If this vision is accepted, we will propose promoting the currently-experimental SwiftPM integration to an officially supported feature starting in Swift 6.

  • Public API audit: Before declaring a stable version, we need to ensure we are confident about the design of the package’s public APIs. We have landed some exploratory work on APIs and features which aren't mature enough yet to be suitable for inclusion in the first stable version, so we will soon begin auditing all APIs in the package and lowering some to experimental API or removing them completely.

    Conversely, some experimental APIs have been added which we believe may be important and mature enough to merit inclusion in the first stable version. We will review those and propose elevating them to public API, using our newly-added feature proposal template.

We look forward to swift-testing reaching stability and inviting wider adoption starting in Swift 6. Please let us know in this thread if you have questions or comments about these upcoming plans.

28 Likes
2

According to the updated Getting Started document, the recommended way to run tests written using the framework will be swift test command. I wonder if there is plan to add support in Xcode to run these tests? I'm asking because swift command doesn't work for code that can only be compiled for or executed on iOS (see here).

Example scenario

(Note: my scenario is a corner case. I don't know if there are more general scenarios).

I'm developing app using Xcode 15 on macOS 13.6 (my MBP has Intel CPU, so it can't be upgraded to macOS 14). Because I use swift language features (e.g. varadic generics, macros) that requires macOS 14, I have the following setting in my packages:

let package = Package(
    name: "storage",
    platforms: [
        .iOS(.v17), .macOS(.v14)
    ],
    ...
}

With that setting I'm able to compile and test my code in Xcode, using iOS simulator as deployment target. However, swift build or swift test doesn't work because these comands use macOS.

2 Likes
3

Hi @rayx! Right now we are focused on building the basic hooks which will allow integration with a variety of tools and IDEs. We have no way to comment on whether or when any particular product will deliver that integration, and I should note that even official integration with SwiftPM is contingent on the approval of our upcoming vision and proposal(s) by the Swift Core Team and the SwiftPM maintainers, respectively. But we believe our efforts here will help unblock broader integration whenever downstream codebases decide to adopt.

2 Likes
4

Happy to see some progress on this! Is there also any movement on automatic test discovery at the moment? The vision document outlines that SE-0385: Custom Reflection Metadata is planned to be used, but I guess this is a thing for a post 1.0 version of the library then?

1 Like
5

Thanks for the question, @BastianKusserow. swift-testing already performs automatic test discovery at runtime by walking emitted Swift metadata sections. :slight_smile: SE-0385 was returned for revision and is, in part, superseded by macros as a language feature.

@kubamracek and his colleagues are investigating future language features that would allow emitting custom metadata in a format that swift-testing could adopt. For more information about that effort, see Kuba's first pitch here and his second pitch here.

4 Likes