Swift Testing and VSCode

Hi,

I've just started looking at how I can integrate swift-testing into the VSCode swift extension. I have a few questions

  1. At the moment I am parsing the xUnit output to get the test results. Is this the intention that IDEs use this output to update their UIs or is there or will there be another solution.

  2. With XCTest I parsed the stdio output as it was produced and could update the VSCode TestExplorer as each test finished. With the xUnit output this is no longer possible as I have to wait for the test process to finish. Has there been any thought into how providing results as tests are completed might be possible with swift-testing.

  3. To debug tests I have to run the test process directly and not via swift test. So far I have been working in Linux and it works similar to xctest where a separate executable is built and you run that executable. Will this be the case on macOS also? As currently macOS xctest based tests are run through an xctest executable that comes with Xcode.

  4. Does test code coverage work with swift-testing yet? It doesn't appear to be working.

cheers

6 Likes

Hi Adam! Our short-term goal is feature parity with XCTest, so for the most part command-line functionality in swift-testing mirrors it. To answer your questions in no particular order:

swift-testing supports XML output with the same schema as XCTest. It is possible to build a more specialized interface for VSCode to use, however we haven't done so at this time.

To speculate a bit: we were considering providing a dedicated entry point to use in place of __swiftPMEntryPoint() (i.e. __vsCodeEntryPoint()) that would write output to a pipe or other stream controlled by VSCode (instead of simply writing to stderr or a file on disk.) This entry point's output format could be tailored to e.g. produce JSON blobs that could be read/decoded by VSCode into whatever data structure would be appropriate. We've written a fair amount of the infrastructure here (in the form of Snapshot types on Test, Event, etc.) but haven't built out the outer interfaces yet.

If all that sounds like a direction worth pursuing, we'd be more than happy to define a formal ABI for VSCode to consume like we have with SwiftPM.

As for debugging tests, swift build --build-tests produces a binary (just like XCTest) on Linux and on Darwin (unlike XCTest!) This has already been implemented in SwiftPM on the main (soon to be release/6.0) branch.

We are not aware of any issues with code coverage, as that part of the build system is implemented independently of any particular test library. If you're having trouble with it, please file a GitHub issue against the swift-testing repo with some repro steps and we'll take a look. Thanks!

I believe my colleague @smontgomery may also have some thoughts he wants to share.

2 Likes

I'll add a few more thoughts on this topic:

We have been planning to provide a more generalized SPI for integrating swift-testing with tools/IDEs, and VSCode would be a great candidate to use such an SPI. An important consideration in designing something like this is maintaining compatibility while allowing both swift-testing and the tool/IDE to evolve independently. In particular, it’s critical to minimize the SPI/ABI surface area and rely primarily on serialized data and Codable types to pass data between components. As @grynspan mentioned, we already have some infrastructure here (the Snapshot types) but a remaining missing piece is a single “entry point” which ties them all together.

Roughly speaking, the way such an entry point could work is that a tool/IDE such as VSCode would first create an instance of Configuration in code, and assign the relevant properties on it which describe how to perform testing—such as specific tests to include/exclude, and other settings. Then it would convert that into a new Snapshot type on Configuration, and serialize that using Codable. (If doing these preliminary steps in code is incompatible with VSCode, this serialized configuration could also be constructed manually and formatted into JSON.) From there, it would pass the serialized configuration data to the runner executable (possibly via a file or pipe), which would use the hypothetical new SPI described above. Internally, this SPI would deserialize the Configuration.Snapshot, convert it back into a Configuration, and then proceed to run tests according to the settings in that configuration. In order to be notified of test results, this SPI would also accept a callback block, which would be passed a serialized Event.Snapshot , and each time an event is received through that channel it could be passed back in the opposite direction to the tool/IDE to be deserialized, interpreted, and presented in UI.

Hi,

I'm not sure what the configuration object provides that can't be done with simple command line arguments at the moment. But I'm happy to go along with this. The VSCode extension won't be able to take advantage of any Codable conformance as it is written in Typescript, but JSON output isn't exactly a problem there.

I'm more interested in what parseable format is output and how it is output. I wrote a simple spec for a parseable format which could easily be output as stdout and parsed. I only spent a very short time on it, but I thought it might be enough to start a discussion.

I've had my eye on getting swift-testing support added to the test discovery that I implemented in Bazel's Swift rules, so I'm very interested in a stable API that arbitrary build systems can hook into.

Right now our XCTest test discovery is implemented similar to what SPM does. We consume symbol graphs extracted from the test targets (instead of indexstore, as SPM does) and generate a runner that does some filtering/sharding and then invokes XCTest's APIs. We also configure an XCTestObservation object to get those notifications and write the results out in xUnit-style format. It looks like supporting swift-testing will be significantly less work, since the tests are discovered by the library via runtime metadata and you've already implemented an xUnit-style output generator.

So, it looks like I could probably start by writing my own small runner based on the implementation in EntryPoint.swift, but any future work that adds stability guarantees to those SPIs would be very welcome!

I've opened a draft PR on swift-testing to add an experimental JSON output stream that can be used with a regular file or named pipe. The precise format of the stream is not fixed in stone (yet) but I figure this is a good starting point, yeah?

1 Like

This is great. I'll try and find some time to have a closer look.