Cross-architecture compilation on Linux

I'd love to see aarch64 Swift support on the Compiler Explorer. Right now it's limited to x86-64 because Compiler Explorer happens to use x86-64 Linux servers, and the official Swift toolchains for Linux don't support cross-architecture compilation.

I saw that SE-0387: Swift SDKs for Cross-Compilation got accepted, but I don't see any apparent implementation of it…?

I saw that @Finagolfin tried to implement this support but it was rejected by Apple…?

Is there a viable way to compile for aarch64 from x86-64 Linux today?

See also compiler-explorer issue #566: ARM compiler option for Swift.

3 Likes

They do, it's just not drop-dead simple or documented yet.

I believe it is implemented and will ship with the upcoming Swift 5.9, but can't speak to any remaining issues that might hold that back, as I have not used it yet.

That is more of an ancillary issue with placing the Swift runtime libraries for multiple architectures of one non-Darwin/Windows platform in a single Swift resource directory, which is easily worked around.

I have gone into more detail there.

1 Like

i have sort of the opposite (or the same?) problem right now where i am developing on an x86_64 host, but need to deploy applications to Graviton (aarch64) servers.

right now i am working around this by compiling the application in the cloud on a Graviton host, but this requires the code to be pushed to a central git repository before it can be built, which makes it difficult to iterate on the application.

i am very curious if there is a simple way to cross-compile on Linux today.

Linux or macOS?

If you are iterating with small source changes and then want to run the tests on a different platform architecture, I would think pushing small source diffs to that central git would be faster than pushing a cross-compiled test binary. If you simply want to make sure the source still builds okay for the other arch, then local builds may be faster.

It has been simple for awhile now, as I laid out late last year. 1. And 3. there are very easy to download from swift.org, with only 2. depending on the distro and your level of expertise with it.

Then, you simply configure all those together with a SwiftPM target destination config file, as I showed with my Android example.

Alternately, you could try the new Swift SDK generator, which currently generates linux SDK bundles from macOS, that you could then use on linux also for cross-arch compilation.

@Max_Desiatov or @ktoso, are there any plans to start distributing official linux SDK bundles on swift.org, for @taylorswift's use case of building on desktop macOS/linux and deploying to linux servers? I think this would help those using Swift on the server.

since the SDK generator is macOS only, i tried following the steps in the linked post.

i did not know what to supply for the -tools-directory, and also did not know where to obtain the glibc sysroot you mentioned. so i tried installing gcc-aarch64-linux-gnu and using the following JSON:

{
    "version": 1,
    "target": "aarch64-unknown-linux-gnu",
    "toolchain-bin-dir": "/home/ubuntu/usr/bin",
    "sdk": "/usr/aarch64-linux-gnu",
    "extra-cc-flags": [
        "-fPIC"
    ],
    "extra-swiftc-flags": [
        "-resource-dir", "/home/ubuntu/aarch64/usr/lib/swift"
    ],
    "extra-cpp-flags": [
        "-lstdc++"
    ]
}

where /home/ubuntu/usr contains an x86_64 swift toolchain and /home/ubuntu/aarch64/usr contains the corresponding aarch64 toolchain.

i tried to compile swift-nio-ssl, which did not work:

swift build --destination /home/ubuntu/ubuntu-22.04-aarch64.json -Xlinker -rpath -Xlinker /usr/aarch64-linux-gnu/lib/
In file included from /swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_ssl.h:145:
/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_base.h:429:10: fatal error: 'memory' file not found
#include <memory>
         ^~~~~~~~
1 error generated.

i tried an apt install libstdc++-12-dev per this issue, but that did not change anything, probably because that does not install the aarch64 headers. any pointers would be appreciated…

dockerfile
FROM ubuntu:jammy
ENV DEBIAN_FRONTEND noninteractive

WORKDIR /home/ubuntu

RUN apt update
RUN apt -y dist-upgrade

RUN apt update
RUN apt -y install \
    binutils \
    git \
    gnupg2 \
    libc6-dev \
    libcurl4-openssl-dev \
    libedit2 \
    libgcc-9-dev \
    libsqlite3-0 \
    libstdc++-9-dev \
    libxml2-dev \
    libz3-dev \
    pkg-config \
    tzdata \
    unzip \
    zlib1g-dev

RUN apt -y install curl

RUN curl https://download.swift.org/swift-5.10-release/ubuntu2204/swift-5.10-RELEASE/swift-5.10-RELEASE-ubuntu22.04.tar.gz \
    -o toolchain.tar.gz
RUN tar --strip-components=1 -xf toolchain.tar.gz
RUN rm toolchain.tar.gz

RUN ln -s $PWD/usr/bin/swift /usr/bin/swift

RUN apt -y install gcc-aarch64-linux-gnu

WORKDIR /home/ubuntu/aarch64

RUN curl https://download.swift.org/swift-5.10-release/ubuntu2204-aarch64/swift-5.10-RELEASE/swift-5.10-RELEASE-ubuntu22.04-aarch64.tar.gz \
    -o toolchain-aarch64.tar.gz

RUN tar --strip-components=1 -xf toolchain-aarch64.tar.gz
RUN rm toolchain-aarch64.tar.gz

WORKDIR /home/ubuntu

COPY ubuntu-22.04-aarch64.json .

RUN rm /home/ubuntu/aarch64/usr/lib/swift/clang
RUN ln -s /home/ubuntu/usr/lib/swift/clang /home/ubuntu/aarch64/usr/lib/swift/clang

BTW: while researching the subject, i came across Cross-compiling packages which contain macros doesn't work · Issue #6950 · apple/swift-package-manager · GitHub which suggests projects that use macros do not support cross-compilation on 5.10. while i didn’t get far enough to try on my own, that’s obviously a showstopper because the application i am trying to build uses macros.

as for compiling in the cloud, i soon found that this is only really viable for debug builds. compiling an application of sufficient complexity in release mode uses too much memory and i was not able to get anywhere close to a successful release build, at least on a t4g.small instance.

That flag is only required for Android, a way to use their patched clang when linking, so it shouldn't be needed on linux.

The glibc sysroot for AArch64 should be automatically installed with that gcc-aarch64 package, else you would not be able to use it, so it's probably one of its dependencies.

Your JSON config file looks fine, with the exception of the sdk. The Swift compiler expects that path to have two directories, <sdk>/usr/include/, containing the glibc headers, and <sdk>/usr/lib/, containing the glibc library and object files.

Ubuntu probably uses the same headers in /usr/include/ across all arches it supports, so you want the sdk to be /, then it depends where your gcc-aarch toolchain places the AArch64 libc.so.

Try cross-compiling hello world in C using that gcc-aarch64 toolchain with the -v verbose flag, then look for which libc.so it links against. If it's linking against one in /usr/lib64/aarch64-linux-gnu/ instead, add "-L", "/usr/lib64/aarch64-linux-gnu/" to your extra-swiftc-flags, just as I do for Android with a similar non-standard cross-compilation layout.

That package is a bit more complicated, as it also links against libstdc++, which most Swift packages don't. I suggest you try a package like swift-argument-parser instead, as shown in my Android doc, and only try a more complicated package like NIO SSL once that is working.

Don't forget to add the right symlink in the Swift resource directory, as mentioned in my linked post. The four -Xlinker -rpath flags should not be necessary if you are then running the resulting binary in a linux AArch64 environment.

I suggest you add the --build-tests flag to build the Swift package's test runner, then copy it over to linux AArch and run it, exactly as I document doing so for Android AArch64 in my doc.

That issue was closed this week, so the next trunk snapshot toolchain may work for this.

T4g.large instances appear to be available for $20/month: that may be your only option to build Swift for linux AArch64 now, if you don't have your own local AArch64 hardware.

2 Likes

swift-argument-parser compiles successfully when using the original /usr/aarch64-linux-gnu value for the sysroot. it does not compile when using / for the sysroot. there is some kind of problem in the headers inside the aarch64 swift resource directory.

$ cat /home/ubuntu/ubuntu-22.04-aarch64.json
{
    "version": 1,
    "target": "aarch64-unknown-linux-gnu",
    "toolchain-bin-dir": "/home/ubuntu/usr/bin",
    "sdk": "/",
    "extra-cc-flags": [
        "-fPIC"
    ],
    "extra-swiftc-flags": [
        "-resource-dir", "/home/ubuntu/aarch64/usr/lib/swift"
    ],
    "extra-cpp-flags": [
        "-lstdc++"
    ]
}
$ ls -l /home/ubuntu/aarch64/usr/lib/swift/clang
lrwxrwxrwx 1 root root 32 Apr 19 19:52 /home/ubuntu/aarch64/usr/lib/swift/clang -> /home/ubuntu/usr/lib/swift/clang
/home/ubuntu/aarch64/usr/lib/swift/CoreFoundation/CoreFoundation.h:25:10: error: 'sys/types.h' file not found
#include <sys/types.h>
         ^
/home/ubuntu/aarch64/usr/lib/swift/CoreFoundation/CFStream.h:20:10: note: while building module 'CDispatch' imported from /home/ubuntu/aarch64/usr/lib/swift/CoreFoundation/CFStream.h:20:
#include <dispatch/dispatch.h>
         ^
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "dispatch.h"
         ^
/home/ubuntu/aarch64/usr/lib/swift/dispatch/dispatch.h:32:10: note: in file included from /home/ubuntu/aarch64/usr/lib/swift/dispatch/dispatch.h:32:
#include <os/generic_unix_base.h>
         ^
/home/ubuntu/aarch64/usr/lib/swift/os/generic_unix_base.h:26:10: error: 'sys/param.h' file not found
#include <sys/param.h>
         ^
<unknown>:0: error: could not build C module 'CoreFoundation'
error: fatalError

so the /usr/aarch64-linux-gnu sysroot path is probably correct.

there is not much you can do on the server that doesn’t depend on TLS, directly or indirectly. so most Swift packages we are interested in using on the server do link against libstdc++, by transitive swift-nio-ssl dependency.

i tried installing the cross-compilation headers for libstdc++.

$ apt install libstdc++-12-dev-arm64-cross

this was not sufficient to get swift-nio-ssl compiling. i can see the header in the sysroot directory

$ ls /usr/aarch64-linux-gnu/include/c++/12/ | grep memory
memory
memory_resource

so i’m not sure what’s going wrong here.

That is unusual, check if both have a param.h installed: find /usr -name param.h. What may be happening if /usr/aarch64-linux-gnu/ has its own set of headers is that the Swift ClangImporter looks in several other paths too, like <sdk>/include/ and finds the headers there instead.

You can check this by adding -v -Xcc -v to your current swift build command, which will have ClangImporter dump out all include paths it's using.

Did you build the test runner with --build-tests? If not, you haven't really figured out which sdk path works.

Why would you say that? sys/{param,types}.h come from the glibc sdk, not from the resource directory.

If the same error as before, likely something to do with how the Swift compiler only looks in certain paths relative to the sdk for the C++ headers. Use those verbose flags I listed to see what it's doing.

It sounds like the system glibc-aarch64 sysroot uses some weird non-standard layout to avoid colliding with the native x86_64 sysroot in /usr/lib/ and so on. You will have to adapt your destination config file to adjust for that, or you may be better off copying all the relevant files into a proper layout in a new /home/ubuntu/glibc-aarch64/ sysroot that you create, then passing that into the destination config as the sdk.

If you want an example of a proper C/C++ sysroot layout, you can download the Android NDK and look in its sysroot/ directory to see what the Swift compiler expects.

$ find /usr -name param.h
/usr/include/linux/param.h
/usr/include/x86_64-linux-gnu/asm/param.h
/usr/include/x86_64-linux-gnu/sys/param.h
/usr/include/x86_64-linux-gnu/bits/param.h
/usr/include/asm-generic/param.h
/usr/aarch64-linux-gnu/include/asm/param.h
/usr/aarch64-linux-gnu/include/sys/param.h
/usr/aarch64-linux-gnu/include/linux/param.h
/usr/aarch64-linux-gnu/include/asm-generic/param.h
/usr/aarch64-linux-gnu/include/bits/param.h

this produces a staggering amount of output, but i’m guessing this is the part we’re interested in:

clang version 15.0.0 (https://github.com/apple/llvm-project.git 5dc9d563e5a6cd2cdd44117697dead98955ccddf)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/ubuntu/usr/bin
ignoring nonexistent directory "/usr/aarch64-linux-gnu/usr/local/include"
ignoring nonexistent directory "/usr/aarch64-linux-gnu/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/ubuntu/aarch64/usr/lib/swift
 /home/ubuntu/aarch64/usr/lib/swift/clang/include
 /usr/aarch64-linux-gnu/include
End of search list.

for some reason, it is building the package plugins (GenerateManual, etc.) for x86_64 and not aarch64; not sure what’s going on there.

the /usr/aarch64-linux-gnu path still works with --build-tests.

i downloaded a copy of that, and noticed that /android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/sysroot/ contains a nested usr/ directory. so i tried replicating that:

mkdir /usr/aarch64-linux-gnu/usr
mv /usr/aarch64-linux-gnu/bin/ /usr/aarch64-linux-gnu/usr/
mv /usr/aarch64-linux-gnu/include/ /usr/aarch64-linux-gnu/usr/
mv /usr/aarch64-linux-gnu/lib/ /usr/aarch64-linux-gnu/usr/

but that was not enough to compile NIOSSL successfully.

clang -cc1 version 15.0.0 based upon LLVM 15.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/aarch64-linux-gnu/usr/local/include"
ignoring nonexistent directory "/usr/aarch64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/ubuntu/swift-nio-ssl/Sources/CNIOBoringSSL/include
 /home/ubuntu/usr/lib/clang/15.0.0/include
 /usr/aarch64-linux-gnu/usr/include
End of search list.
In file included from /home/ubuntu/swift-nio-ssl/Sources/CNIOBoringSSL/ssl/ssl_versions.cc:15:
In file included from /home/ubuntu/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_ssl.h:145:
/home/ubuntu/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_base.h:429:10: fatal error: 'memory' file not found
#include <memory>
         ^~~~~~~~

however, i can observe that memory.h exists at /usr/aarch64-linux-gnu/usr/include.

# ls /usr/aarch64-linux-gnu/usr/include
a.out.h      assert.h    endian.h           finclude       gnu             libgen.h   misc        neteconet   obstack.h       re_comp.h    sgtty.h        stdlib.h    tgmath.h     utmp.h
aio.h        bits        envz.h             fmtmsg.h       gnu-versions.h  libintl.h  mntent.h    netinet     paths.h         regex.h      shadow.h       string.h    thread_db.h  utmpx.h
aliases.h    byteswap.h  err.h              fnmatch.h      grp.h           limits.h   monetary.h  netipx      poll.h          regexp.h     signal.h       strings.h   threads.h    values.h
alloca.h     complex.h   errno.h            fpu_control.h  gshadow.h       link.h     mqueue.h    netiucv     printf.h        resolv.h     sound          sys         time.h       video
ar.h         cpio.h      error.h            fstab.h        iconv.h         linux      mtd         netpacket   proc_service.h  rpc          spawn.h        syscall.h   ttyent.h     wait.h
argp.h       ctype.h     execinfo.h         fts.h          ieee754.h       locale.h   net         netrom      protocols       sched.h      stab.h         sysexits.h  uchar.h      wchar.h
argz.h       dirent.h    fcntl.h            ftw.h          ifaddrs.h       malloc.h   netash      netrose     pthread.h       scsi         stdc-predef.h  syslog.h    ucontext.h   wctype.h
arpa         dlfcn.h     features-time64.h  gconv.h        inttypes.h      math.h     netatalk    nfs         pty.h           search.h     stdint.h       tar.h       ulimit.h     wordexp.h
asm          drm         features.h         getopt.h       langinfo.h      mcheck.h   netax25     nl_types.h  pwd.h           semaphore.h  stdio.h        termio.h    unistd.h     xen
asm-generic  elf.h       fenv.h             glob.h         lastlog.h       memory.h   netdb.h     nss.h       rdma            setjmp.h     stdio_ext.h    termios.h   utime.h

Yep, it shows that the last include path used is <sdk>/include/, as I suspected.

What about the test runner, built for AArch64? file .build/aarch64-unknown-linux-gnu/debug/swift-argument-parserPackageTests.xctest

That's not the header it's looking for, it wants the C++ header memory, which is at /usr/include/c++/13/memory in my linux distro. Whatever heuristic that Swift is using to find those C++ headers isn't working for you, though make sure it is there in your sysroot first.

1 Like

ah, that was exactly what i was missing. after doing some more investigation, i figured out how to cross-compile SwiftNIO SSL for Graviton. i wrote up the instructions here:

as i anticipated, i was not able to compile the actual server application because it depends on a library that uses macros, and for some reason the compiler builds the macros for the target architecture, which prevents itself from being able to run the macros to generate the library code. as it is a library i also wrote, i’m guessing my best option is to just rewrite the library to not use macros anymore, which i was already considering for the purposes of reducing compilation times.

1 Like

Thanks for writing it up, maybe it will be useful for someone else.

Cross-compilation of Swift packages that have macros has been broken since macros were enabled in the linux compiler seven months ago, but @Max_Desiatov pushed a fix to 6.1 trunk this week. Hopefully, that gets backported to 6.0 or a 5.10 patch release if it works well.

In the meantime, that is why I suggested your best bet may be to rent out AArch64 hardware, like that t4g.large instance.

1 Like

a cherry-picked 5.10.1 would be really great. who is in charge of that?

First we have to make sure it's working, as that fix still has other build errors when I cross-compile for Android. Download a trunk snapshot toolchain and see if it works for your cross-arch usecase now.