SPM and Linux

Here I want to assert what I know (and ask what I don't know) about what's missing in SPM, that we need in order to be able to discover tests on Linux.

Currently

Currently we need a LinuxMain.swift file in order to run tests on Linux. This is a fragile dependency, as when tests are added, removed or modified, it's not automatically changed. This leads to misleading CI outputs and leaves Linux devs a hard time trying to port existing Swift libraries to Linux. This is specially hard, since the file can be generated under Darwin but not under Linux, where you do notice if it's missing.

Options

There are three options for CI and Linux devs.

Just Scripts

One is to generate this file through scripts, made on Ruby, Python or Swift itself. Anything works, really. The problem with this approach is that, without building an entire compiler, you'll only reach a lexical knowledge of the files. Test classes inherit from XCTest, can be extended outside of their original file, and can inherit indirectly from XCTest, like so: A : XCTest, B : A. Without semantic knowledge, all scripts will be fragile at best.

SourceKit-based Scripts

Semantic knowledge can be obtained using SourceKit-derived tools such as SourceKitten and Sourcery. This is the second option: scripting with SK-based tools.

There are two problems with this:

  1. The easiest tool for this, Sourcery, and many other SourceKit-based tools, don't work on Linux.
  2. Even though you can construct a tool to do this with SourceKitten, which works on Linux, that doesn't help most developers. SourceKitten is not part of SPM, and therefore it's not available to all Swift users directly from their toolchains.
    Moreover, knowing SourceKitten can achieve this, being it only a wrapper of SourceKit, we know that SPM, which has access to SourceKit, should be able to achieve it as well.
    Therefore, it could produce the LinuxMain.swift file on its own, avoiding this problem altogether.

Completing SPM

The third option is to give SPM what it's missing, such that the code that's being used to discover tests on Darwin can be used to discover tests on Linux. How to do this, is explained in the next point.

Giving SPM what it needs

Here I'm guessing that SPM could discover tests on Linux without a huge revamp of its design. If I'm wrong, then we should be working on a redesign. But we'll assume that that's not the case. Here's my analysis on what's missing, then, from the codebase:

This is the command that triggers the generation of the LinuxMain.swift file:

case .generateLinuxMain:
  #if os(Linux)
    warning(message: "can't discover new tests on Linux; please use this option on macOS instead")
  #endif
    // No relevant #ifs
    let graph = try loadPackageGraph()
    // No relevant #ifs
    let testPath = try buildTestsIfNeeded(options, graph: graph)
    // This must be the culprit
    let testSuites = try getTestSuites(path: testPath)
    // No relevant #ifs
    let generator = LinuxMainGenerator(graph: graph, testSuites: testSuites)
    try generator.generate()

As you can see, I've marked the lines with comments pointing out if there are or not cross-platform concerns. All of the functions and classes, except for getTestSuites, seem to work fine on Linux.

Let's jump to getTestSuites then:

fileprivate func getTestSuites(path: AbsolutePath) throws -> [TestSuite] {
    //// Run the correct tool.
  #if os(macOS)

    // 
    // No relevant #ifs here. 
    // This class is also used on Linux code.
    // 
    let tempFile = try TemporaryFile()

    //
    // Maybe this is macOS only.
    // `xctestHelperPath` is valid in Linux, but it throws a `fatalError` 
    // on failure. It looks for a specific library in the system. If this 
    // library is not present on Linux, then this line is not
    // cross-platform.
    //
    let args = [SwiftTestTool.xctestHelperPath().asString, path.asString, tempFile.path.asString]
    
    // 
    // No relevant #ifs here.
    // This function has both valid Linux and macOS branches.
    // 
    var env = try constructTestEnvironment(sanitizers: options.sanitizers, toolchain: try getToolchain())
    //// Add the sdk platform path if we have it. If this is not present, we
    //// might always end up failing.
    
    //
    // Maybe this is macOS only.
    // The function's documentation hints that it might be designed
    // with only macOS in mind.
    //
    if let sdkPlatformFrameworksPath = Destination.sdkPlatformFrameworkPath() {
        env["DYLD_FRAMEWORK_PATH"] = sdkPlatformFrameworksPath.asString
    }
    // 
    // No relevant #ifs here.
    // This class is also used on Linux code.
    // 
    try Process.checkNonZeroExit(arguments: args, environment: env)
    //// Read the temporary file's content.
    let data = try fopen(tempFile.path).readFileContents()
  #else
    let args = [path.asString, "--dump-tests-json"]
    let data = try Process.checkNonZeroExit(arguments: args)
  #endif
    //// Parse json and return TestSuites.
    return try TestSuite.parse(jsonString: data)
}

This function already has inline comments, which I've modified to have 4 preceding /s so that it's overall easier to read.

As you can see, there are only two candidates here: SwiftTestTool.xctestHelperPath and Destination.sdkPlatformFrameworkPath. Which one is it? Are both not available on Linux? I do not know.

These seem to be the only hurdle to pass before we can have automatic test discovery on Linux. What can we do about it? I think we should push SPM forward. This is, if only second to Foundation, the most important key piece in the Swift environment that isn't yet available to all of the Swift community.

I will be happy to help. Let's do this.

-- Félix

4 Likes

Thanks for opening a discussion on this @felix91gr!

It'll be awesome if someone can work on fixing this problem, I'll be more than happy to help. Tests are discovered using Objective-C runtime on Darwin but that is not available on Linux so we're required to list all test methods by hand right now.

We've talked about using SourceKit in past to solve this problem. Thanks to work done by several contributors in the community, it works on linux and is available in the toolchains! However, there is plenty of work that needs to be done in order to fix this problem. Here is a summary:

  • Adopt llbuild as a library. This is already in-progress thanks to @hartbit! See: https://github.com/apple/swift-package-manager/pull/1537

  • Land some Swift bindings (in SwiftPM) for SourceKit's C-API. This will be very useful as we can leverage SourceKit for several SwiftPM features in future. For test discovery, we need bindings for SourceKit's indexing feature.

  • Write a custom llbuild tool to run SourceKit using the above bindings at build time and generate LinuxMain.swift as a build artifact.

1 Like

Ah, I see. Then, here I was indeed wrong:

I was worried that maybe the ObjC runtime was part of SPM's test discovery.


Hey, at least it's been discussed! That's something.

That PR looks really good! Good job, @hartbit :slight_smile:

For this I thoroughly recommend reading the bindings used at SourceKitten. They are extensive, so much so that I would even recommend forking SourceKitten or using it as a dependency directly, because they've been battle tested for at least a couple of years already and... I wouldn't want to go through that task myself.

SourceKitten uses the MIT license and the owner is @jpsim. You'd also probably like to have the help/counsel of @norio_nomura here, as he's been an important force in bringing SK to Linux, and on making SourceKitten correct.

Sounds good :slight_smile:


All in all I think it sounds like a solid plan. I'm really glad it's in the cards.

I don't know if my skills can help with anything other than testing, though. This sounds very low level, and I don't have a Mac where to run Xcode. But maybe I can help with step 3 :slight_smile:

(I understand that using or forking SKitten might not be plausible under the rules that Apple uses for its repos. That's fine :slight_smile: but even still, I think SourceKitten can be used as a reference to know how to make good C bindings for SK :slight_smile:)