Here I want to assert what I know (and ask what I don't know) about what's missing in SPM, that we need in order to be able to discover tests on Linux.
Currently
Currently we need a LinuxMain.swift file in order to run tests on Linux. This is a fragile dependency, as when tests are added, removed or modified, it's not automatically changed. This leads to misleading CI outputs and leaves Linux devs a hard time trying to port existing Swift libraries to Linux. This is specially hard, since the file can be generated under Darwin but not under Linux, where you do notice if it's missing.
Options
There are three options for CI and Linux devs.
Just Scripts
One is to generate this file through scripts, made on Ruby, Python or Swift itself. Anything works, really. The problem with this approach is that, without building an entire compiler, you'll only reach a lexical knowledge of the files. Test classes inherit from XCTest, can be extended outside of their original file, and can inherit indirectly from XCTest, like so: A : XCTest, B : A. Without semantic knowledge, all scripts will be fragile at best.
SourceKit-based Scripts
Semantic knowledge can be obtained using SourceKit-derived tools such as SourceKitten and Sourcery. This is the second option: scripting with SK-based tools.
There are two problems with this:
- The easiest tool for this, Sourcery, and many other SourceKit-based tools, don't work on Linux.
- Even though you can construct a tool to do this with SourceKitten, which works on Linux, that doesn't help most developers. SourceKitten is not part of SPM, and therefore it's not available to all Swift users directly from their toolchains.
Moreover, knowing SourceKitten can achieve this, being it only a wrapper of SourceKit, we know that SPM, which has access to SourceKit, should be able to achieve it as well.
Therefore, it could produce theLinuxMain.swiftfile on its own, avoiding this problem altogether.
Completing SPM
The third option is to give SPM what it's missing, such that the code that's being used to discover tests on Darwin can be used to discover tests on Linux. How to do this, is explained in the next point.
Giving SPM what it needs
Here I'm guessing that SPM could discover tests on Linux without a huge revamp of its design. If I'm wrong, then we should be working on a redesign. But we'll assume that that's not the case. Here's my analysis on what's missing, then, from the codebase:
This is the command that triggers the generation of the LinuxMain.swift file:
case .generateLinuxMain:
#if os(Linux)
warning(message: "can't discover new tests on Linux; please use this option on macOS instead")
#endif
// No relevant #ifs
let graph = try loadPackageGraph()
// No relevant #ifs
let testPath = try buildTestsIfNeeded(options, graph: graph)
// This must be the culprit
let testSuites = try getTestSuites(path: testPath)
// No relevant #ifs
let generator = LinuxMainGenerator(graph: graph, testSuites: testSuites)
try generator.generate()
As you can see, I've marked the lines with comments pointing out if there are or not cross-platform concerns. All of the functions and classes, except for getTestSuites, seem to work fine on Linux.
Let's jump to getTestSuites then:
fileprivate func getTestSuites(path: AbsolutePath) throws -> [TestSuite] {
//// Run the correct tool.
#if os(macOS)
//
// No relevant #ifs here.
// This class is also used on Linux code.
//
let tempFile = try TemporaryFile()
//
// Maybe this is macOS only.
// `xctestHelperPath` is valid in Linux, but it throws a `fatalError`
// on failure. It looks for a specific library in the system. If this
// library is not present on Linux, then this line is not
// cross-platform.
//
let args = [SwiftTestTool.xctestHelperPath().asString, path.asString, tempFile.path.asString]
//
// No relevant #ifs here.
// This function has both valid Linux and macOS branches.
//
var env = try constructTestEnvironment(sanitizers: options.sanitizers, toolchain: try getToolchain())
//// Add the sdk platform path if we have it. If this is not present, we
//// might always end up failing.
//
// Maybe this is macOS only.
// The function's documentation hints that it might be designed
// with only macOS in mind.
//
if let sdkPlatformFrameworksPath = Destination.sdkPlatformFrameworkPath() {
env["DYLD_FRAMEWORK_PATH"] = sdkPlatformFrameworksPath.asString
}
//
// No relevant #ifs here.
// This class is also used on Linux code.
//
try Process.checkNonZeroExit(arguments: args, environment: env)
//// Read the temporary file's content.
let data = try fopen(tempFile.path).readFileContents()
#else
let args = [path.asString, "--dump-tests-json"]
let data = try Process.checkNonZeroExit(arguments: args)
#endif
//// Parse json and return TestSuites.
return try TestSuite.parse(jsonString: data)
}
This function already has inline comments, which I've modified to have 4 preceding /s so that it's overall easier to read.
As you can see, there are only two candidates here: SwiftTestTool.xctestHelperPath and Destination.sdkPlatformFrameworkPath. Which one is it? Are both not available on Linux? I do not know.
These seem to be the only hurdle to pass before we can have automatic test discovery on Linux. What can we do about it? I think we should push SPM forward. This is, if only second to Foundation, the most important key piece in the Swift environment that isn't yet available to all of the Swift community.
I will be happy to help. Let's do this.
-- Félix
