Can't get a working 5.2.x cross-compiler

I've been trying to get an ARM64 Linux cross-compiler toolchain working with Swift 5.2.x and haven't had any luck; the same steps work for 5.1.x just fine. The toolchain is built from the standard 5.2.3-RELEASE for Mac x86-64 and the futurejones ARM AArch64 Linux 5.2.3-RELEASE, both compilers work normally on their respective platforms, and the ARM one works within QEMU on the Mac as a sanity check too.

The failure is a very generic one:

$ swift build  --destination /tmp/Destinations/arm64-5.2.3-RELEASE.json  
<unknown>:0: error: unable to execute command: <unknown>
[3/4] Linking hello

Adding -v shows that compilation works, and the failure occurs at the link stage:

/private/tmp/Toolchains/arm64-5.2.3-RELEASE.xctoolchain/usr/bin/swiftc \
  -target aarch64-unknown-linux -use-ld=gold \
  -tools-directory /private/tmp/Toolchains/arm64-5.2.3-RELEASE.xctoolchain/usr/bin \
  -L /Users/scott/Developer/hello/.build/aarch64-unknown-linux/debug \
  -o /Users/scott/Developer/hello/.build/aarch64-unknown-linux/debug/hello \
  -module-name hello -emit-executable -Xlinker '-rpath=$ORIGIN' \
  @/Users/scott/Developer/hello/.build/aarch64-unknown-linux/debug/hello.product/Objects.LinkFileList \
  -target aarch64-unknown-linux \
  -L /private/tmp/Toolchains/arm64-5.2.3-RELEASE.xctoolchain/usr/lib

Adding -v -Xcc -v to that invocation doesn't reveal anything insightful:

$ /private/tmp/Toolchains/arm64-5.2.3-RELEASE.xctoolchain/usr/bin/swiftc -v -Xcc -v -target aarch64-unknown-linux -use-ld=gold -tools-directory /private/tmp/Toolchains/arm64-5.2.3-RELEASE.xctoolchain/usr/bin -L /Users/scott/Developer/hello/.build/aarch64-unknown-linux/debug -o /Users/scott/Developer/hello/.build/aarch64-unknown-linux/debug/hello -module-name hello -emit-executable -Xlinker '-rpath=$ORIGIN' @/Users/scott/Developer/hello/.build/aarch64-unknown-linux/debug/hello.product/Objects.LinkFileList -target aarch64-unknown-linux -L /private/tmp/Toolchains/arm64-5.2.3-RELEASE.xctoolchain/usr/lib

Apple Swift version 5.2.3 (swift-5.2.3-RELEASE)
Target: aarch64-unknown-linux
<unknown>:0: error: unable to execute command: <unknown>

Tips/suggestions for debugging further would be appreciated!

I've noticed a couple of differences in the verbose build log compared to 5.1.1, specifically:

  • the -sdk parameter is missing from 5.2.3 builds, though adding it by hand doesn't fix the issue
  • a new -modulewrap phase that is performed using the host compiler instead the target, resulting in a x86-64 .o file rather than AArch64, but fixing this by hand doesn't fix the issue

That regression affected Android as well. I fixed it for Android here in this pull request. If you look at the conversation and follow its links to the two preceding attempts, you’ll see I originally tried putting it back for Linux too. But that lead to continuous integration failures, so not realizing anything was actually broken for Linux, we just gave up and left it as it was. Maybe reading that history might help you home in on a fix or workaround faster.

That sounds strange to me. For Android, it was the opposite. Module wrapping was introduced in 5.1.2, but ran for the host instead of the Android target, preventing debug builds. It was then definitively fixed for Android by 5.2.1 (but I haven’t tried any higher patch number yet).

Just to be clear, while there are definitely SwiftPM issues, running the compiler by hand step-by-step with equal parameters as 5.1.x still fails in the same way.

<unknown>:0: error: unable to execute command: <unknown>

That error is here, looks like it's unable to execute a clang cross-compiler or cross-linker for your target. You may need to stick a log line in there to see exactly what command it's trying to invoke and then figure out why that's failing.

Your -tools-directory has a clang cross-compiler, linker, and libc files for your target?

Even better, -driver-show-job-lifecycle already exists with which I was able to figure out what was going on:

$ file .build/aarch64-unknown-linux/debug/hello
.build/aarch64-unknown-linux/debug/hello: ELF 64-bit LSB shared object,
 ARM aarch64, version 1 (SYSV), dynamically linked,
 interpreter /lib/, for GNU/Linux 3.7.0,
 with debug_info, not stripped

Guess I should write a PR...