Cross‐compile from WSL to Windows with SwiftPM?

Development Package Manager
,
1

I have grown weary of CMake, and since I noticed that a few weeks ago WSL was added to GitHub Actions’ hosts, I thought I would see if it was possible to run SwiftPM inside WSL and cross‐compile to the external Windows host.

I have gotten Swift 5.2.1 installed both in and out of WSL (Ubuntu 18.04) such that Windows successfully builds for Windows (with CMake) and WSL successfully builds for Linux with SwiftPM. However, in attempting to cross‐compile, I’ve gotten stuck on the part where WSL’s toolchain needs to be directed to use Windows’ SDK.

In every attempt below, graph resolution succeeds and everything is fetched, but then it fails when it tries to build the first source file (meaning all the manifests are correctly building and executing on WSL for WSL and the error has to do with the actual cross‐compile for Windows).

The destination JSON contains these entries (because the --triple and --sdk flags didn’t exist yet in 5.2.1):

  • sdk: /mnt/c/Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk (the one in Windows)
  • toolchain-bin-dir: /usr/bin (the one in WSL)
  • target: x86_64-unknown-windows-msvc

  1. Without any extra -Xswiftc flags, it errors like this:

    <unknown>:0: error: unable to load standard library for target 'x86_64-unknown-windows-msvc'
swift: /home/buildnode/jenkins/workspace/oss-swift-5.2-package-linux-ubuntu-18_04/swift/lib/Frontend/FrontendInputsAndOutputs.cpp:111: void swift::FrontendInputsAndOutputs::assertMustNotBeMoreThanOnePrimaryInput() const: Assertion `!hasMultiplePrimaryInputs() && "have not implemented >1 primary input yet"' failed.

    Based on this Tensorflow issue, I assume SwiftPM is not passing the SDK to swiftc in all the necessary ways.

  2. Upon adding -Xswiftc -sdk -Xswiftc //mnt/c/Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk, it errors like this:

    //mnt/c/Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk/usr/include/dispatch/module.modulemap:1:8: error: redefinition of module 'Dispatch'
module Dispatch {
       ^
/usr/lib/swift/dispatch/module.modulemap:1:8: note: previously defined here
module Dispatch {
       ^

    That seems to mean that it is now using both the SDK from the command line and the host SDK at / in WSL.

  3. Remembering similar issues in the past dealing with Android, I tried mimicking the Android cross‐compilation flags for 5.2.1, and doing -Xswiftc -resource-dir -Xswiftc //mnt/c/Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk/usr/lib/swift instead. Then it errors like this:

    <module-includes>:1:10: note: in file included from <module-includes>:1:
#include "AssertionReporting.h"
         ^
//mnt/c/Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk/usr/lib/swift/shims/AssertionReporting.h:16:10: note: in file included from //mnt/c/Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk/usr/lib/swift/shims/AssertionReporting.h:16:
#include "SwiftStdint.h"
         ^
//mnt/c/Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk/usr/lib/swift/shims/SwiftStdint.h:28:10: error: 'stdint.h' file not found
#include <stdint.h>
         ^
<unknown>:0: error: could not build C module 'SwiftShims'
swift: /home/buildnode/jenkins/workspace/oss-swift-5.2-package-linux-ubuntu-18_04/swift/lib/Frontend/FrontendInputsAndOutputs.cpp:111: void swift::FrontendInputsAndOutputs::assertMustNotBeMoreThanOnePrimaryInput() const: Assertion `!hasMultiplePrimaryInputs() && "have not implemented >1 primary input yet"' failed.

    I found this swift‐build issue, which makes it look like Swift either doesn’t know about or cannot find the MSVC components. They are present; CMake can build with them. So my hunch is that CMake automatically infers some -I flags or --sysroot based on knowledge of Visual Studio, which I need to pass manually while inside WSL.

So answers to any of these questions, or any other ideas that might help me along would be appreciated:

  • What are the proper --sysroot and include (-I) directories?
  • Does the SDK contain hard‐coded paths? (The slashes would have to be reversed in WSL, and prepended with //mnt.)
  • Does anyone out there already have a working cross‐compilation from WSL to Windows?
2

I added -Xswiftc -I -Xswiftc '//mnt/c/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.26.28801/include', which is a translation of ${VCToolsInstallDir}/include. It seems to have now found <stdint.h>, but it is hitting the same sort of error with <stddef.h>.

3

I haven't tested this in a while, but I did have a working WSL -> Windows SwiftPM cross-compilation setup working a few months ago. What's below is all from memory and notes I've taken so forgive me it's a little vague.

I had copied the lib directory for the target into the Linux Swift /usr/lib (i.e. to /usr/lib/swift/windows), avoiding having to pass the -sdk. My destination JSON file looked like this:

{
    "version": 1,
    "sdk": "/mnt/c/Users/troug/Documents/swift-toolchain/usr/bin/swiftc",
    "toolchain-bin-dir": "/mnt/c/Users/troug/Documents/swift-toolchain/usr/bin",
    "target": "x86_64-unknown-windows-msvc",
    "dynamic-library-extension": "lib",
    "extra-cc-flags": [
        "-DUNICODE=1",
        "-isystem",
        "/mnt/c/Users/troug/Documents/swift-toolchain/usr/lib/clang/7.0.0/include",
        "-isystem",
        "/mnt/c/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.24.28314/include",
        "-isystem",
        "/mnt/c/Program Files (x86)/Windows Kits/10/Include/10.0.18362.0/ucrt",
        "-isystem",
        "/mnt/c/Program Files (x86)/Windows Kits/10/Include/10.0.18362.0/um",
        "-isystem",
        "/mnt/c/Program Files (x86)/Windows Kits/10/Include/10.0.18362.0/shared"
    ],
    "extra-swiftc-flags": [
        "-use-ld=lld",
        "-Xcc",
        "-isystem",
        "-Xcc",
        "/mnt/c/Users/troug/Documents/swift-toolchain/usr/lib/clang/7.0.0/include",
        "-Xcc",
        "-isystem",
        "-Xcc",
        "/mnt/c/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.24.28314/include",
        "-Xcc",
        "-isystem",
        "-Xcc",
        "/mnt/c/Program Files (x86)/Windows Kits/10/Include/10.0.18362.0/ucrt",
        "-Xcc",
        "-isystem",
        "-Xcc",
        "/mnt/c/Program Files (x86)/Windows Kits/10/Include/10.0.18362.0/um",
        "-Xcc",
        "-isystem",
        "-Xcc",
        "/mnt/c/Program Files (x86)/Windows Kits/10/Include/10.0.18362.0/shared",
​
        "-L/mnt/c/Users/troug/Documents/swift-toolchain/usr/lib/swift/windows",
        "-L/mnt/c/Users/troug/Documents/swift-toolchain/usr/lib/swift/windows/x86_64",
        "-L/mnt/c/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.24.28314/ATLMFC/lib/x64",
        "-L/mnt/c/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.24.28314/lib/x64",
        "-L/mnt/c/Program Files (x86)/Windows Kits/NETFXSDK/4.8/lib/um/x64",
        "-L/mnt/c/Program Files (x86)/Windows Kits/10/lib/10.0.18362.0/ucrt/x64",
        "-L/mnt/c/Program Files (x86)/Windows Kits/10/lib/10.0.18362.0/um/x64",
        "-L/mnt/c/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.24.28314/ATLMFC/lib/x86",
        "-L/mnt/c/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.24.28314/lib/x86",
        "-L/mnt/c/Program Files (x86)/Windows Kits/NETFXSDK/4.8/lib/um/x86",
        "-L/mnt/c/Program Files (x86)/Windows Kits/10/lib/10.0.18362.0/ucrt/x86",
        "-L/mnt/c/Program Files (x86)/Windows Kits/10/lib/10.0.18362.0/um/x86"
    ],
    "extra-cpp-flags": [
    ],
}

I used swift build --destination WindowsDest.json -Xlinker "/mnt/c/Program Files (x86)/Windows Kits/10/Lib/10.0.18362.0/um/x64/Ole32.Lib" -Xlinker "/mnt/c/Program Files (x86)/Windows Kits/10/Lib/10.0.18362.0/um/x64/shell32.lib" as the invocation; I think the only reason those linker flags are on the command line was because I was testing things and didn't bother to add them to the JSON file.

The order of the includes is important; in particular, you need to make sure the Clang overlays are included before the Windows headers.

1 Like
4

Thanks, @Torust!

I had kept hunting for the files the compiler couldn’t find and quickly figured out that most were in the directories where the SDK module maps had been installed. After adding those directories with -I it got much further. There are still a few things missing, but now with your list I have a much better idea where to look for them. I will come back to this on Monday.

5

@Torust, do you remember what you did to make it build everything as dynamic libraries? Or was your package graph just simple enough it didn’t matter?

At least, I think the default static linking is the problem, since I know that isn’t supported on Windows yet. SwiftPM compiles several modules successfully, but then it dies when it tries to link the executable.

With the default linker:

LINK : warning LNK4044: unrecognized option '[...]/Windows.sdk/usr/lib/swift/windows/x86_64/swiftrt.obj'; ignored
LINK : warning LNK4044: unrecognized option '[...]/.build/x86_64-unknown-windows-msvc/debug/[file].build/[file].swift.o'; ignored
[↑ One of these for each file.]
LINK : fatal error LNK1104: cannot open file '[...]/.build\x86_64-unknown-windows-msvc\debug\[executable].exe'
clang-7: error: linker command failed with exit code 80 (use -v to see invocation)
[[x]/138] Linking [executable].exe

After installing lld and explicitly selecting it:

/usr/bin/lld-link: error: [...]/.build/x86_64-unknown-windows-msvc/debug/[first file].build/[first file].swiftmodule.o is not a COFF file
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
[41/137] Linking [executable].exe
6

Actually, I do remember something like that error. Does a release build work for you? I think the issue is the wrapped module debug info (swift-package-manager/BuildPlan.swift at 933c5d70c1cba4d096953769466cd3056f39ca03 · apple/swift-package-manager · GitHub) which isn't produced in a release build.

7

Thanks. The release configuration successfully linked the first executable. (The rest of the build hasn’t finished yet though.)

8

More complicated modules are giving me trouble, though I’m not sure what the difference is between those that work and those that don’t. (All of them build successfully with native CMake.)

Without the clang entry (which seemed not to be necessary at first), the error is:

Declaration may not be in a Comdat!
%swift.metadata_response (i64)* @"[mangled symbol]"
<unknown>:0: error: fatal error encountered during compilation;
<unknown>:0: note: Broken module found, compilation aborted!

The mangled symbol is from a module for which no Compiling notification has yet appeared, so I don’t know if it is occurring while compiling the module itself, or when importing into another later. (Either way, the output log seems to have been interrupted early by the crash, truncating the most recent actions taken by SwiftPM.)

When I add the clang import path, the error changes:

<unknown>:0: error: fatal error encountered while reading from module 'Foundation'; please file a bug report with your project and the crash log
*** DESERIALIZATION FAILURE (please include this section in any bug report) ***
top-level value not found
Cross-reference to module 'ucrt'
... memcmp
... with type (Optional<UnsafeRawPointer>, Optional<UnsafeRawPointer>, Int) -> Int32

I don’t know if that is closer or farther from the solution.

Neither error sounds like the sort of thing that would have an easy workaround.

I should ask though, is it deliberate that you’ve added both x64 and x86 to with -L? That seems a little fishy to me.

9

Adding the Clang module path needs to be done before starting the build, since otherwise the Windows headers are pulled in first – you get those circular references if you add Clang later since the module caches are already somewhat built. I don’t know if that’s your issue specifically, but it’s worth trying a clean build (i.e. remove .build) with the Clang headers on the include path from the start if you haven’t already.

With that said, the COMDAT error seems like something different and not something I’ve encountered. In terms of the truncated output, I’ve found piping to a text file to be useful (e.g. > CompileOutput.txt on the end of the swift build invocation); I don’t think things get truncated in that case. You can also try passing -Xcc -v to see the header search paths actually being used, which may or may not be useful.

x86 being on the library path is I think just an artefact of me copying from the LIB environment variable; you should be able to safely remove it.

10

They have all been clean builds. I posted those last errors mostly for anyone else following the thread. There was the off chance you might recognize them, but since they look more like something broken than something missing, I wasn’t really expecting you to have an answer. Thanks for your help.

11

The COMDAT thing seems to be a bug in the compiler’s handling of a particular module—and quite possibly a clue to the SegFault errors that I kept hitting at run time with native builds of the same module. I have now successfully built half a dozen other modules.

The only thing that still isn’t working is the test executable:

/usr/bin/lld-link: error: entry point must be defined
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
[5/6] Linking MyPackagePackageTests.xctest

Any idea of a way to work around it?

12

I'm not sure how the test setup works, sorry, although I wonder if it might be due to something like the LinuxMain difference between macOS and Linux – I had a quick look through SPM sources and didn't spot any Linux-specific testing code apart from excluding C-language test targets.

For both that and the COMDAT issue @compnerd may have a better idea. It looks like @compnerd had to define a Windows-specific WindowsMain.swift file for testing with CMake at least –e.g. build: support the test suite on Windows · compnerd/swift-numerics@697e935 · GitHub, so maybe something similar is needed under SPM, or else if you already have that file due to the existing CMake builds some flag is needed to make SPM see it as an entry point.

13

Yes, I’m familiar with WindowsMain.swift from the CMake set‐up I have been using until now. It is basically the same as LinuxMain.swift. SwiftPM also has an --enable-test-discovery mode that generates a similar file inside .build instead. All of them have virtually the same contents, they are just generated different ways and have different names. And they all produce a simple executables which are directly launched to run the tests—Android testing is done by coping that executable onto a device or emulator first and launching it. I just can’t figure out how the compiler is being told that these files should be treated as though they were named main.swift. Windows is an oddball for having several entry points to choose from and CMake seems to magically know how to set them up, but SwiftPM seems to be missing something in that respect.

14

I have stabilized the script that builds the main package. It’s here if anyone wants to see it. Since I still haven’t gotten it to build the tests successfully, I still have to use CMake for those at the moment.

3 Likes
15

Still trying to debug the test target.

Upgrading to lld 9 changed the error to lld-link: error: subsystem must be defined. Also adding -Xlinker -subsystem:console resulted in an exit code of 1 but no text description of what went wrong.

16

I found a workaround to enable testing! It’s ugly, but not as ugly or complicated as CMake.

Normal executable targets do work, so...

I added the following to the bottom of the manifest. Whenever the TARGETING_WINDOWS environment variable is enabled, the package converts all its test targets into normal targets, and then assembles an additional target that points at Tests/WindowsTests and depends on all of them. Then all you need is a main.swift file inside that target’s folder, and you have a normal executable that runs the tests when you launch it. The main.swift can presumably be a symlink to LinuxMain.swift, but in my case I moved and renamed the very similar WindowsMain.swift file I had already been generating for use with CMake.

import Foundation
if ProcessInfo.processInfo.environment["TARGETING_WINDOWS"] == "true" {
  var tests: [Target] = []
  var other: [Target] = []
  for target in package.targets {
    if target.type == .test {
      tests.append(target)
    } else {
      other.append(target)
    }
  }
  package.targets = other
  package.targets.append(
    contentsOf: tests.map({ test in
      return .target(
        name: test.name,
        dependencies: test.dependencies,
        path: test.path ?? "Tests/\(test.name)",
        exclude: test.exclude,
        sources: test.sources,
        publicHeadersPath: test.publicHeadersPath,
        cSettings: test.cSettings,
        cxxSettings: test.cxxSettings,
        swiftSettings: test.swiftSettings,
        linkerSettings: test.linkerSettings
      )
    })
  )
  package.targets.append(
    .target(
      name: "WindowsTests",
      dependencies: tests.map({ Target.Dependency.target(name: $0.name) }),
      path: "Tests/WindowsTests"
    )
  )
}

The only downside was that several uses of #file in the tests needed reworking, because the path looks different to the executable (running on Windows) than it did to the compiler (running inside WSL).

Asides:

  1. Saved by executable manifests—again! (Have I mentioned how thankful I am for that design decision?)

  2. This means SwiftPM should theoretically be able to bootstrap itself across from WSL to Windows, which will make contributing to its Windows port much easier.

    However, when I tried it, one of its dependencies, llbuild, needed to be able to attach .when(platforms: [.windows]) to a C compilation condition. While the compiler believes .windows to exist in 5.2.4, it just leads to a runtime crash. You can add it unconditionally while you are working on the package, but leaving it there breaks all the other platforms.