I'd love to see aarch64 Swift support on the Compiler Explorer. Right now it's limited to x86-64 because Compiler Explorer happens to use x86-64 Linux servers, and the official Swift toolchains for Linux don't support cross-architecture compilation.
They do, it's just not drop-dead simple or documented yet.
I believe it is implemented and will ship with the upcoming Swift 5.9, but can't speak to any remaining issues that might hold that back, as I have not used it yet.
That is more of an ancillary issue with placing the Swift runtime libraries for multiple architectures of one non-Darwin/Windows platform in a single Swift resource directory, which is easily worked around.
i have sort of the opposite (or the same?) problem right now where i am developing on an x86_64 host, but need to deploy applications to Graviton (aarch64) servers.
right now i am working around this by compiling the application in the cloud on a Graviton host, but this requires the code to be pushed to a central git repository before it can be built, which makes it difficult to iterate on the application.
i am very curious if there is a simple way to cross-compile on Linux today.
If you are iterating with small source changes and then want to run the tests on a different platform architecture, I would think pushing small source diffs to that central git would be faster than pushing a cross-compiled test binary. If you simply want to make sure the source still builds okay for the other arch, then local builds may be faster.
It has been simple for awhile now, as I laid out late last year. 1. And 3. there are very easy to download from swift.org, with only 2. depending on the distro and your level of expertise with it.
Then, you simply configure all those together with a SwiftPM target destination config file, as I showed with my Android example.
Alternately, you could try the new Swift SDK generator, which currently generates linux SDK bundles from macOS, that you could then use on linux also for cross-arch compilation.
@Max_Desiatov or @ktoso, are there any plans to start distributing official linux SDK bundles on swift.org, for @taylorswift's use case of building on desktop macOS/linux and deploying to linux servers? I think this would help those using Swift on the server.
since the SDK generator is macOS only, i tried following the steps in the linked post.
i did not know what to supply for the -tools-directory, and also did not know where to obtain the glibc sysroot you mentioned. so i tried installing gcc-aarch64-linux-gnu and using the following JSON:
where /home/ubuntu/usr contains an x86_64 swift toolchain and /home/ubuntu/aarch64/usr contains the corresponding aarch64 toolchain.
i tried to compile swift-nio-ssl, which did not work:
swift build --destination /home/ubuntu/ubuntu-22.04-aarch64.json -Xlinker -rpath -Xlinker /usr/aarch64-linux-gnu/lib/
In file included from /swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_ssl.h:145:
/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_base.h:429:10: fatal error: 'memory' file not found
#include <memory>
^~~~~~~~
1 error generated.
i tried an apt install libstdc++-12-dev per this issue, but that did not change anything, probably because that does not install the aarch64 headers. any pointers would be appreciated…
dockerfile
FROM ubuntu:jammy
ENV DEBIAN_FRONTEND noninteractive
WORKDIR /home/ubuntu
RUN apt update
RUN apt -y dist-upgrade
RUN apt update
RUN apt -y install \
binutils \
git \
gnupg2 \
libc6-dev \
libcurl4-openssl-dev \
libedit2 \
libgcc-9-dev \
libsqlite3-0 \
libstdc++-9-dev \
libxml2-dev \
libz3-dev \
pkg-config \
tzdata \
unzip \
zlib1g-dev
RUN apt -y install curl
RUN curl https://download.swift.org/swift-5.10-release/ubuntu2204/swift-5.10-RELEASE/swift-5.10-RELEASE-ubuntu22.04.tar.gz \
-o toolchain.tar.gz
RUN tar --strip-components=1 -xf toolchain.tar.gz
RUN rm toolchain.tar.gz
RUN ln -s $PWD/usr/bin/swift /usr/bin/swift
RUN apt -y install gcc-aarch64-linux-gnu
WORKDIR /home/ubuntu/aarch64
RUN curl https://download.swift.org/swift-5.10-release/ubuntu2204-aarch64/swift-5.10-RELEASE/swift-5.10-RELEASE-ubuntu22.04-aarch64.tar.gz \
-o toolchain-aarch64.tar.gz
RUN tar --strip-components=1 -xf toolchain-aarch64.tar.gz
RUN rm toolchain-aarch64.tar.gz
WORKDIR /home/ubuntu
COPY ubuntu-22.04-aarch64.json .
RUN rm /home/ubuntu/aarch64/usr/lib/swift/clang
RUN ln -s /home/ubuntu/usr/lib/swift/clang /home/ubuntu/aarch64/usr/lib/swift/clang
as for compiling in the cloud, i soon found that this is only really viable for debug builds. compiling an application of sufficient complexity in release mode uses too much memory and i was not able to get anywhere close to a successful release build, at least on a t4g.small instance.
That flag is only required for Android, a way to use their patched clang when linking, so it shouldn't be needed on linux.
The glibc sysroot for AArch64 should be automatically installed with that gcc-aarch64 package, else you would not be able to use it, so it's probably one of its dependencies.
Your JSON config file looks fine, with the exception of the sdk. The Swift compiler expects that path to have two directories, <sdk>/usr/include/, containing the glibc headers, and <sdk>/usr/lib/, containing the glibc library and object files.
Ubuntu probably uses the same headers in /usr/include/ across all arches it supports, so you want the sdk to be /, then it depends where your gcc-aarch toolchain places the AArch64 libc.so.
Try cross-compiling hello world in C using that gcc-aarch64 toolchain with the -v verbose flag, then look for which libc.so it links against. If it's linking against one in /usr/lib64/aarch64-linux-gnu/ instead, add "-L", "/usr/lib64/aarch64-linux-gnu/" to your extra-swiftc-flags, just as I do for Android with a similar non-standard cross-compilation layout.
That package is a bit more complicated, as it also links against libstdc++, which most Swift packages don't. I suggest you try a package like swift-argument-parser instead, as shown in my Android doc, and only try a more complicated package like NIO SSL once that is working.
Don't forget to add the right symlink in the Swift resource directory, as mentioned in my linked post. The four -Xlinker -rpath flags should not be necessary if you are then running the resulting binary in a linux AArch64 environment.
I suggest you add the --build-tests flag to build the Swift package's test runner, then copy it over to linux AArch and run it, exactly as I document doing so for Android AArch64 in my doc.
That issue was closed this week, so the next trunk snapshot toolchain may work for this.
swift-argument-parser compiles successfully when using the original /usr/aarch64-linux-gnu value for the sysroot. it does not compile when using / for the sysroot. there is some kind of problem in the headers inside the aarch64 swift resource directory.
/home/ubuntu/aarch64/usr/lib/swift/CoreFoundation/CoreFoundation.h:25:10: error: 'sys/types.h' file not found
#include <sys/types.h>
^
/home/ubuntu/aarch64/usr/lib/swift/CoreFoundation/CFStream.h:20:10: note: while building module 'CDispatch' imported from /home/ubuntu/aarch64/usr/lib/swift/CoreFoundation/CFStream.h:20:
#include <dispatch/dispatch.h>
^
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "dispatch.h"
^
/home/ubuntu/aarch64/usr/lib/swift/dispatch/dispatch.h:32:10: note: in file included from /home/ubuntu/aarch64/usr/lib/swift/dispatch/dispatch.h:32:
#include <os/generic_unix_base.h>
^
/home/ubuntu/aarch64/usr/lib/swift/os/generic_unix_base.h:26:10: error: 'sys/param.h' file not found
#include <sys/param.h>
^
<unknown>:0: error: could not build C module 'CoreFoundation'
error: fatalError
so the /usr/aarch64-linux-gnu sysroot path is probably correct.
there is not much you can do on the server that doesn’t depend on TLS, directly or indirectly. so most Swift packages we are interested in using on the server do link against libstdc++, by transitive swift-nio-ssl dependency.
i tried installing the cross-compilation headers for libstdc++.
$ apt install libstdc++-12-dev-arm64-cross
this was not sufficient to get swift-nio-ssl compiling. i can see the header in the sysroot directory
$ ls /usr/aarch64-linux-gnu/include/c++/12/ | grep memory
memory
memory_resource
That is unusual, check if both have a param.h installed: find /usr -name param.h. What may be happening if /usr/aarch64-linux-gnu/ has its own set of headers is that the Swift ClangImporter looks in several other paths too, like <sdk>/include/ and finds the headers there instead.
You can check this by adding -v -Xcc -v to your current swift build command, which will have ClangImporter dump out all include paths it's using.
Did you build the test runner with --build-tests? If not, you haven't really figured out which sdk path works.
Why would you say that? sys/{param,types}.h come from the glibc sdk, not from the resource directory.
If the same error as before, likely something to do with how the Swift compiler only looks in certain paths relative to the sdk for the C++ headers. Use those verbose flags I listed to see what it's doing.
It sounds like the system glibc-aarch64 sysroot uses some weird non-standard layout to avoid colliding with the native x86_64 sysroot in /usr/lib/ and so on. You will have to adapt your destination config file to adjust for that, or you may be better off copying all the relevant files into a proper layout in a new /home/ubuntu/glibc-aarch64/ sysroot that you create, then passing that into the destination config as the sdk.
If you want an example of a proper C/C++ sysroot layout, you can download the Android NDK and look in its sysroot/ directory to see what the Swift compiler expects.
for some reason, it is building the package plugins (GenerateManual, etc.) for x86_64 and not aarch64; not sure what’s going on there.
the /usr/aarch64-linux-gnu path still works with --build-tests.
i downloaded a copy of that, and noticed that /android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/sysroot/ contains a nested usr/ directory. so i tried replicating that:
but that was not enough to compile NIOSSL successfully.
clang -cc1 version 15.0.0 based upon LLVM 15.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/aarch64-linux-gnu/usr/local/include"
ignoring nonexistent directory "/usr/aarch64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
/home/ubuntu/swift-nio-ssl/Sources/CNIOBoringSSL/include
/home/ubuntu/usr/lib/clang/15.0.0/include
/usr/aarch64-linux-gnu/usr/include
End of search list.
In file included from /home/ubuntu/swift-nio-ssl/Sources/CNIOBoringSSL/ssl/ssl_versions.cc:15:
In file included from /home/ubuntu/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_ssl.h:145:
/home/ubuntu/swift-nio-ssl/Sources/CNIOBoringSSL/include/CNIOBoringSSL_base.h:429:10: fatal error: 'memory' file not found
#include <memory>
^~~~~~~~
however, i can observe that memory.h exists at /usr/aarch64-linux-gnu/usr/include.
Yep, it shows that the last include path used is <sdk>/include/, as I suspected.
What about the test runner, built for AArch64? file .build/aarch64-unknown-linux-gnu/debug/swift-argument-parserPackageTests.xctest
That's not the header it's looking for, it wants the C++ header memory, which is at /usr/include/c++/13/memory in my linux distro. Whatever heuristic that Swift is using to find those C++ headers isn't working for you, though make sure it is there in your sysroot first.
ah, that was exactly what i was missing. after doing some more investigation, i figured out how to cross-compile SwiftNIO SSL for Graviton. i wrote up the instructions here:
as i anticipated, i was not able to compile the actual server application because it depends on a library that uses macros, and for some reason the compiler builds the macros for the target architecture, which prevents itself from being able to run the macros to generate the library code. as it is a library i also wrote, i’m guessing my best option is to just rewrite the library to not use macros anymore, which i was already considering for the purposes of reducing compilation times.
Thanks for writing it up, maybe it will be useful for someone else.
Cross-compilation of Swift packages that have macros has been broken since macros were enabled in the linux compiler seven months ago, but @Max_Desiatov pushed a fix to 6.1 trunk this week. Hopefully, that gets backported to 6.0 or a 5.10 patch release if it works well.
In the meantime, that is why I suggested your best bet may be to rent out AArch64 hardware, like that t4g.large instance.
I then copy that file into the swift base image, install the aarch64 toolchain and try to cross-compile grpc-swift with a minimal Dockerfile as follows:
FROM --platform=$BUILDPLATFORM swift:6.0.3-noble AS grpc_swift
RUN mkdir -p /protoc-gen-swift /grpc-swift /root/aarch64
RUN apt-get update
RUN apt-get install -y \
build-essential \
curl \
gcc-aarch64-linux-gnu \
g++-aarch64-linux-gnu
WORKDIR /root
RUN curl -sSL https://download.swift.org/swift-6.0.3-release/ubuntu2404-aarch64/swift-6.0.3-RELEASE/swift-6.0.3-RELEASE-ubuntu24.04-aarch64.tar.gz | tar xz --strip 1 -C /root/aarch64
COPY ubuntu-24.04-aarch64.json .
WORKDIR /grpc-swift
RUN curl -sSL https://api.github.com/repos/grpc/grpc-swift/tarball/1.24.2 | tar xz --strip 1 -C /grpc-swift
RUN swift build -c release --product protoc-gen-swift --destination /root/ubuntu-24.04-aarch64.json -v
When invoked with docker buildx build . this fails with an Scrt1.o: unsupported ELF machine number 183 error:
I feel like I'm really close here, but I'm not sure why the linker is failing. I previously had a lot of hacky linker stuff going on like this, but I'm not sure if any of it is still necessary with Swift 6. Can anyone help me get this over the line?
I was able to work around this using the Swift 6 SDK as follows:
FROM --platform=$BUILDPLATFORM swift:${SWIFT_IMAGE_VERSION}-noble AS grpc_swift
ARG GRPC_SWIFT_VERSION SWIFT_IMAGE_VERSION
RUN apt-get update && \
apt-get install -y curl
RUN swift sdk install \
https://download.swift.org/swift-${SWIFT_IMAGE_VERSION}-release/static-sdk/swift-${SWIFT_IMAGE_VERSION}-RELEASE/swift-${SWIFT_IMAGE_VERSION}-RELEASE_static-linux-0.0.1.artifactbundle.tar.gz \
--checksum 67f765e0030e661a7450f7e4877cfe008db4f57f177d5a08a6e26fd661cdd0bd
RUN mkdir -p /grpc-swift
RUN curl -sSL https://api.github.com/repos/grpc/grpc-swift/tarball/${GRPC_SWIFT_VERSION} | tar xz --strip 1 -C /grpc-swift
WORKDIR /grpc-swift
ARG TARGETOS TARGETARCH
RUN <<EOF
case ${TARGETARCH} in
"amd64") SWIFTARCH=x86_64 ;;
"arm64") SWIFTARCH=aarch64 ;;
*) echo "ERROR: Machine arch ${TARGETARCH} not supported." ;;
esac
swift build -c release --product protoc-gen-swift --swift-sdk $SWIFTARCH-swift-linux-musl
swift build -c release --product protoc-gen-grpc-swift --swift-sdk $SWIFTARCH-swift-linux-musl
install -D /grpc-swift/.build/release/protoc-gen-swift /out/usr/bin/protoc-gen-swift && \
install -D /grpc-swift/.build/release/protoc-gen-grpc-swift /out/usr/bin/protoc-gen-grpc-swift
EOF
This results in two very large binaries including libraries and the Swift runtime however, it would be better to be able to dynamically link these. Is this possible using the SDK?
I’m not sure about dynamic linking, but you can at least strip the binaries using the strip command if you have build-essential packages installed. This strips debug symbols from the binaries and unused libraries as well too, I believe.
I am also interested in dynamic linking but as we’ve found, it’s somewhat incomplete in swiftpm right now.
Your sdk looks off: the Swift toolchain expects a directory containing <sdk>/usr/include/, <sdk>/usr/lib/, and so on, which I doubt /usr/aarch64-linux-gnu/ contains. I'd simply try / as the sdk first and then look for workarounds if that doesn't work.
No, the static Musl SDK doesn't contain dynamic libraries, so you'd have to go back to your original approach above.