Listing Tests with extended information

It would be really useful if the output of swift --list-tests had an option to include location information for each test. ie File name and line number. I don't know if this information is available to SwiftPM or XCTest but if it is it would be very helpful for the VSCode Swift extension.

Currently the extension is using a DocumentSymbol request to SourceKit-LSP to augment the test list with location information. This is done whenever you open a file containing tests, so location information is only available once you have opened the file containing the tests. I could run it on every file at startup but this would be quite cumbersome and impact the quality of results from other uses of SourceKit-LSP.

The other option is have SourceKit-LSP have a custom request for providing a list of all tests with location information.

5 Likes

@adam-fowler sorry for the slow response. how about filing a ticket for an enhancement request to be able and produce structured information about known test? e.g. swift list-tests --output json (or some better verb for this)

I looked into this some more and currently XCTest does not provide this information (file, line number) back to SwiftPM, so we would need to start by modifying that before we can make progress.

@smontgomery any input on how to best do that? can XCTest even provide this information? based on swift-corelibs-xctest/XCTestCase.swift at main · apple/swift-corelibs-xctest · GitHub I did not see a way.

1 Like

@tomerd Yeah that's correct — XCTest itself doesn't have information about the source code location of tests themselves. XCTest does include #file and #line default parameter values in some of its APIs, such as the XCTAssert functions, so that the location of calls to those APIs can be logged if/when they fail. But test methods (or classes) themselves are declarations, not API calls, so we cannot leverage #file or #line to gather location info of test declarations.

SwiftPM might have a viable path to obtain this information, though. It already leverages indexer data when building test targets to discover the full list of their tests and synthesize source files containing those lists. So conceivably, this logic in SwiftPM could be extended to also retrieve the source code locations from that indexer data and include those locations in swift test --list-tests output.

I've added an issue to the swift-package-manager repo Add option to provide location information when listing tests · Issue #5601 · apple/swift-package-manager · GitHub to track this