We're happy to announce a new open-source utility that simplifies cross-compilation of Swift packages!
When working with Xcode, cross-compilation from macOS to other Darwin platforms is something that a lot of Swift developers use on a daily basis. At the same time, cross-compilation with command-line developer tools to Linux and other platforms that Swift supports has been not as easy to set up. With SE-0387 we'd like to reduce this gap and make cross-compilation a first-class feature in command-line interface of SwiftPM.
While SE-0387 specified a format and a file system layout for Swift SDK bundles, it did not prescribe how these bundles should be generated. We're providing a reference implementation of such generator that supports macOS as a host platform and a few major Linux distributions as a target platform.
It's important to distinguish Swift SDK authors from Swift SDK users. The new Swift SDK Generator should be primarily utilized by Swift SDK authors, who can customize it for their needs and publish their own Swift SDK bundles. In turn, Swift SDK users can rely on the swift experimental-sdk command introduced in Swift 5.9 to install a bundle previously generated by a Swift SDK author.
We're working on adding support for all Linux distributions officially supported by the Swift project. As always, all contributions are welcome!
Pretty good. It's always a pain having to compile on weak systems, such as older Raspberrys.
One of the problems I often encounter in Package.swift, is that #if refers to the host platform instead of the target – will there be a solution for this?
This is intended behaviour, #if executes on the platform that the code is running on, and package manifests run on the host platform when cross-compiling. Changing #if os conditions to refer to the target platform in Package.swift when cross-compiling would be semantically incorrect. For customizations on the target platform, packages should be using platform condition .when arguments in PackageDescription DSL, as it was done recently in sqlite-nio for example.
swift run swift-sdk-generator --swift-version 5.9-RELEASE --target-cpu-architecture arm64
swift experimental-sdk install /Users/rvs/Development/Projects/blucycles/xcompilers/Bundles/5.9-RELEASE_ubuntu_jammy_aarch64.artifactbundle
Building for debugging...
warning: Could not read SDKSettings.json for SDK at: /Users/rvs/Library/org.swift.swiftpm/swift-sdks/5.9-RELEASE_ubuntu_jammy_aarch64.artifactbundle/5.9-RELEASE_ubuntu_jammy_aarch64/aarch64-unknown-linux-gnu/ubuntu-jammy.sdk
<unknown>:0: warning: glibc not found for 'aarch64-unknown-linux-gnu'; C stdlib may be unavailable
<unknown>:0: warning: glibc not found for 'aarch64-unknown-linux-gnu'; C stdlib may be unavailable
warning: Could not read SDKSettings.json for SDK at: /Users/rvs/Library/org.swift.swiftpm/swift-sdks/5.9-RELEASE_ubuntu_jammy_aarch64.artifactbundle/5.9-RELEASE_ubuntu_jammy_aarch64/aarch64-unknown-linux-gnu/ubuntu-jammy.sdk
[6/6] Linking helloworld
I'm a little worried about the warnings about glibc there. But honestly I was getting them with my 5.8 cross-compilers since last spring too. Any ideas on what those are?
So the immediate question for me now is how I establish a transitively closed set of shlibs necessary to run the output executable on the target. In particular, my need is to create distro-less docker containers that can run the executable on a Raspberry Pi. So taking the helloworld example above, this means that I need to resolve all of the following symbols and provide the resolving shlibs in my docker container:
nm --demangle ./.build/aarch64-unknown-linux-gnu/debug/helloworld | grep U
U $sSS21_builtinStringLiteral17utf8CodeUnitCount7isASCIISSBp_BwBi1_tcfC
U $sSSN
U $sSaMa
U $ss27_allocateUninitializedArrayySayxG_BptBwlFyp_Tg5
U $ss5print_9separator10terminatoryypd_S2StF
U $sypN
U __libc_start_main
U abort
U swift_addNewDSOImage
U swift_bridgeObjectRelease
U swift_release
In the past this has meant starting with the libs in /usr/lib/swift/linux, running nm gathering any undefined symbols and figuring out what apt package has the shlib that resolves that symbol, adding that shlib to the list and recursing.
The transitively resolved list then informs all the apt packages that must be installed in order to run my executable. Is the SDK generator maintaining this list somewhere or do I still need to run that process?
In the default configuration, when generating packages for Ubuntu (or for any .deb-based distribution when those are supported), it downloads each package separately and unpacks them into the SDK tree. The full list of packages is maintained in LinuxDistributions.swift.
If you run the generator passing it --with-docker flag, it launches a Docker container and copies /usr/lib over from that container. The downside in the latter scenario of course is that it requires a valid Docker installation, and also makes copied /usr/lib hierarchy not as lean as it can be with a predefined list of .deb packages. The docker image is currently hardcoded in the swiftBaseDockerImage property of VersionsConfiguration, but the intention is to make it fully configurable. In the meantime, you can build your own image and hardcode your own. Contributions that expose the name of the image as a CLI option are, of course, welcome, if I don't get to it myself soon enough.
I'm confused by this answer. Is the idea that I deploy the contents of the SDK to my target? In the past I have distinguished between SDK and Runtime. SDK is what is used on the host (to do software development), Runtime is what I need on the target to run. Has that distinction gone away?
Ok, I see you're referring to Swift symbols here. In my answer I was referring to non-Swift system libraries you may want to link.
The idea is that you either deploy to the environment which has exact same version of Swift runtime libraries installed, or pass --static-swift-stdlib to swift build and have those linked statically. There are known issues with --static-swift-stdlib when cross-compiling, and I'm working on fixing those.
When Swift runtime/stdlib are linked statically, the only versions of dynamically linked libraries you care to be compatible with are Glibc and libstdc++, at least on most Linux distributions that use these libraries and not musl and/or LLVM's libc++ instead. And if you link to other system libraries dynamically with systemLibrary in your Package.swift, that needs to be sorted out too.
One reason I ask is that the list you link to the LinuxDistributions.swift is not close to sufficient to run (as I do), swift-nio, swifty-gpio, swift-nio-mqtt executables. currently the list of shlibs required to run those executables on an R/Pi running focal is maintained at arm64.json
I can still run my scripts to generate that package list, I'm just wondering if that is instead something that should move into the SDK-generator project
Yes this is precisely the problem. producing the apt package configuration that would support a specific executable using the minimal set of packages turns out to be really important if you want to deploy. Currently I do this with a complete set of packages that could be accessed from any program written using anything accessible from /usr/lib/linux/swift plus whatever is needed by nio, nio-mqtt and swifty-gpio. That is vastly too large (order of 750MB of packages).
What would be great would be a system that would allow us to look at the undefined symbols of the produced target executable and state a minimal set of packages required for that executable on that target. Then docker container generation could be a part of the build process..
I'm not sure that producing a list of packages that contain necessary dynamically linked libraries can scale for all Linux distributions, not even speaking of other platforms that the generator may want to support in the future. I suggest relying on static linking as much as possible, which simplifies the deployment process significantly, and I'm currently working on making static linking the default per SE-0342.
We're aware that --static-swift-stdlib doesn't link Glibc and libstdc++ dynamically, but ABI of those is stable and you can link against older versions of those to make it compatible with older Linux distributions.
I don't think that static compilation is going to work for docker deployment for the same reasons that it didn't work for pre-5.0 swift on mac and ios.
You still have to resolve all the symbols in /usr/lib/swift/linux plus nio, mqtt, gpio. If you are only deploying one container with one executable, that's fine, statically resolve. OTOH if you are deploying multiple containers with swift executables on one R/Pi host, or you are frequently redeploying with updates, you'd like a) for all your containers to share the common libs and b) to minimize your container deployment size. in my case, updating my containers has to be done through a 4G modem connection.
With a shared docker container housing all the shlibs shipping with the Pi, each docker container with NIO/MQTT/GPIO weighs in about 9MB stripped of debug symbols. If I force static resolution (which was what I did in the early days) each container and each update ends up being about 150MB of download. Bandwidth costs on that alone were prohibitive.
Swift 5 is ABI-stable on Darwin, it isn't on non-Darwin platforms. You will still need to redeploy stdlib/runtime even with a patch version change of the toolchain and the SDK (i.e. 5.9.0 to 5.9.1). For that reason alone I recommend statically linking on non-Darwin platforms. As for binary size concerns, IMO those should be discussed in the other thread: Pitch: Support LTO for Swift.
Overall, I think for the generator or even SwiftPM to produce some kind of manifest of dynamically linked libraries makes sense. If you're interested in fleshing out the design for it, please feel free to create a separate thread, or an issue on the generator repository.
what I will probably do with SwiftCrossCompilers now is get rid of the SDK and Toolchain generation and focus on creation of the shared runtime libs. Currently I'm creating a single container that houses all of the shlibs, shipping that with my Pi and mounting that container from the containers using my executable.
@Max_Desiatov should things like the glibc warning I mentioned above be reported against swift-idk-generator or somewhere else? It's been present since 5.8 when trying to cross-compile but there has been nowhere to report cross-compilation issues prior to now..
Yes, IMO the generator repository is a good place to track it for now. I'll transfer it to a corresponding repository if the root cause is found to be some other part of the infrastructure. Thanks!