Pitch: Cross-Compilation Destination Bundles

Hi everyone. We know there's space for improving cross-compilation features in Swift. Here's a pitch for standardizing a format of archives containing cross-compilation SDKs and toolchains for distribution and installation purposes. It also includes an overview of future directions to take in this space.

I look forward to all feedback from the community on this topic!

Many thanks to @tomerd @ktoso @abertelrud @NeoNacho @tbkka @Ben_Cohen @johannesweiss @lukasa @al45tair @jakepetroules and others for comments, suggestions, and contributions to this pitch.

Introduction

Cross-compilation is a common development use case. When cross-compiling, we need to refer to these two main concepts:

  • host platform, where developer's code is built;
  • target platform, where developer's code is running.

Another important term is toolchain, which is a set of executable binaries running on the host platform. Additionally, we define SDK as a set of dynamic and/or static libraries, headers, and other resources required to produce a binary for a target platform. Let’s call a toolchain and an SDK bundled together a destination.

Motivation

Swift cross-compilation (CC) destinations are currently produced on an ad-hoc basis for different combinations of host and target platforms. For example, scripts that produce macOS β†’ Linux CC destinations were created by both the Swift team and the Swift community. At the same time, the distribution process of CC destinations is cumbersome. After building a destination tree on the file system, required metadata files rely on hardcoded absolute paths. Adding support for relative paths in destination's metadata and providing a unified way to distribute and install destinations as archives would clearly be an improvement to the multi-platform Swift ecosystem.

The primary audience of this pitch are people who cross-compile from macOS to Linux. When deploying to single-board computers supporting Linux (e.g. Raspberry Pi), building on the target hardware may be too slow or run out of available memory. Quite naturally, users would prefer to cross-compile on their host machine when targeting these platforms.

In other cases, building in a Docker container is not always the best solution for certain development workflows. For example, when working with Swift AWS Lambda Runtime, some developers may find that installing Docker just for building a project is a daunting step that shouldn’t be required.

The solution described below is general enough to scale for any host/target platform combination.

Proposed Solution

Since CC destination is a collection of binaries arranged in a certain directory hierarchy, it makes sense to distribute it as an archive. We'd like to build on top of SE-0305 and extend the .artifactbundle format to support this.

Additionally, we propose introducing a new swift destination CLI command for installation and removal of CC destinations on the local filesystem.

We introduce a notion of a top-level toolchain, which is the toolchain that handles user’s swift destination invocations. Parts of this top-level toolchain (linker, C/C++ compilers, and even the Swift compiler) can be overridden with tools supplied in .artifactbundle s installed by swift destination invocations.

When the user runs swift build with the selected CC destination, the overriding tools from the corresponding bundle are invoked by swift build instead of tools from the top-level toolchain.

Detailed Design

CC Destination Artifact Bundles

As a quick reminder for a concept introduced in SE-0305, an artifact bundle is a directory that has the filename suffix .artifactbundle and has a predefined structure with .json manifest files provided as metadata.

The proposed structure of artifact bundles containing CC destinations looks like:

<name>.artifactbundle
β”œ info.json
β”œ <destination artifact>
β”‚ β”œ <host variant>
β”‚ β”‚ β”œ destination.json
β”‚ β”‚ β”” <destination file tree>
β”‚ β”” <host variant>
β”‚   β”œ destination.json
β”‚   β”” <destination file tree>
β”œ <destination artifact>
β”‚ β”” <host variant>
β”‚   β”œ destination.json
β”‚   β”” <destination file tree>
β”œ <destination artifact>
┆ β””β”„

For example, a destination bundle allowing to cross-compile Swift 5.7 source code to recent versions of Ubuntu from macOS would look like this:

swift-5.7_ubuntu.artifactbundle
β”œ info.json
β”œ ubuntu_jammy
β”‚ β”œ arm64-apple-darwin
β”‚ β”‚ β”œ destination.json
β”‚ β”‚ β”” <destination file tree>
β”‚ β”” x86_64-apple-darwin
β”‚   β”œ destination.json
β”‚   β”” <destination file tree>
β”œ ubuntu_focal
β”‚ β”” x86_64-apple-darwin
β”‚   β”œ destination.json
β”‚   β”” <destination file tree>
β”œ ubuntu_bionic
┆ β””β”„

Here each artifact directory is dedicated to a specific CC destination, while binaries for a specific host platform are placed in arm64-apple-darwin and x86_64-apple-darwin subdirectories.

Note the presence of destination.json files in each <host variant> subdirectory. These files should contain a JSON dictionary with an evolved version of the schema of existing destination.json files that SwiftPM already supports (hence "version": 2 )

{
  "version": 2,
  "sdkRootDir": <relative path to a sysroot directory in the destination tree>,
  "toolchainBinDir": <relative path to toolchain executables in the destination tree>,
  "runtimeDir": <optional relative path to runtime components in the destination tree>,
  "hostTriples": [<an array of supported host platform triples>],
  "targetTriples": [<an array of supported target platform triples>],
  "extraSwiftCFlags": [<an array of flags passed to the Swift compiler>],
  "extraCCFlags": [<an array of flags passed to the C compiler>],
  "extraCXXFlags": [<an array of flags passed to the C++ compiler>],
  "extraLinkerFlags": [<an array of flags passed to the linker>]
}

We propose that all relative paths in destination.json files should be validated not to "escape" the destination bundle for security reasons. That is, ../ components, if present in paths, will not be allowed to reference files and directories outside of a corresponding destination bundle. Symlinks will also be validated to prevent them from escaping out of the bundle.

Lastly, info.json bundle manifests at the root of artifact bundles should specify "type": "crossCompilationDestination" for corresponding artifacts. Artifact identifiers in this manifest uniquely identify a CC destination. The rest of the properties of bundle manifests introduced in SE-0305 are preserved.

Destination Bundle Installation

To manage CC destinations, we'd like to introduce a new swift destination command with three subcommands:

  • swift destination install <bundle URL or local filesystem path>, which downloads a given bundle if needed and installs it in a location discoverable by SwiftPM. For destinations installed from remote URLs an additional --checksum option is required, through which users of destinations can specify a checksum provided by publishers of destinations. The latter can produce a checksum by running swift package compute-checksum command (introduced in SE-0272) with the destination artifact bundle archive as an argument.
  • swift destination list, which prints a list of already installed CC destinations with their identifiers.
  • swift destination delete <identifier> will delete a given destination from the filesystem.

Using a CC Destination

After a destination is installed, users can refer to it via its identifier passed to the --destination option, e.g.

swift build --destination ubuntu-jammy

We'd also like to make --destination flexible enough to recognize destination triples when there's only a single CC destination installed for such triple:

swift build --destination x86_64-unknown-linux-gnu

When multiple destinations support the same triple, an error message will be printed listing these destinations and asking the user to select a single one via its identifier.

CC Destination Bundle Generation

CC destinations can be generated quite differently, depending on host and target platform combinations and user's needs. We intentionally don't specify how destination artifact bundles should be generated.

Authors of this document intend to publish source code for a macOS β†’ Linux CC destination generator, which community is welcome to fork and reuse for their specific needs. This generator will use Docker for setting up the build environment locally before copying it to the destination tree. Relying on Docker in this generator makes it easier to reuse and customize existing build environments. Important to clarify, that Docker is only used for bundle generation, and users of CC destinations do not need to have Docker installed on their machine to utilize it.

As an example, destination publishers looking to add a library to an Ubuntu 22.04 destination environment would modify a Dockerfile similar to this one in CC destination generator source code:

FROM swift:5.7-jammy

apt-get install -y \
  # PostgreSQL library provided as an example.
  libpq-dev
  # Add more libraries as arguments to `apt-get install`.

Then to generate a new CC destinations, a generator executable delegates to Docker for downloading and installing required tools and libraries, including the newly added ones. After a Docker image with destination environment is ready, the generator copies files from the image to a corresponding .artifactbundle destination tree.

Prior Art

Rust

In the Rust ecosystem, its toolchain and standard library built for a target platform are managed by the rustup tool. For example, artifacts required for cross-compilation to aarch64-linux-unknown-gnu are installed with rustup target add aarch64-linux-unknown-gnu. Then building for this target with Rust’s package manager looks like cargo build --target=aarch64-linux-unknown-gnu .

Mainstream Rust tools don’t provide an easy way to create your own destinations/targets. You’re only limited to the list of targets provided by Rust maintainers. This likely isn’t a big problem per se for Rust users, as Rust doesn’t provide C/C++ interop on the same level as Swift. It means that Rust packages much more rarely than Swift expect certain system-provided packages to be available in the same way that SwiftPM allows with systemLibrary .

Currently, Rust doesn’t supply all of the required tools when running rustup target add. It’s left to a user to specify paths to a linker that’s suitable for their host/target combination manually in a config file. We feel that this should be unnecessary, which is why destination bundles proposed for Swift can provide their own tools via toolchainBinDir property in destination.json .

Go

Go’s standard library is famously self-contained and has no dependencies on C or C++ standard libraries. Because of this there’s no need to install additional targets and destinations. Cross-compiling in Go works out of the box by passing GOARCH and GOOS environment variables with chosen values, an example of this is GOARCH=arm64 GOOS=linux go build invocation.

This would be a great experience for Swift, but it isn’t easily achievable as long as Swift standard library depends on C and C++ standard libraries. Any code interoperating with C and/or C++ would have to link with those libraries as well. When compared to Go, our proposed solution allows both dynamic and, at least on Linux when Musl is supported, full static linking. We’d like Swift to allow as much customization as needed for users to prepare their own destination bundles.

Alternatives Considered

Building Applications in Docker Containers

Instead of coming up with a specialized bundle format for destinations, users of Swift on macOS targeting Linux could continue to use Docker. But, as discussed in the Motivation section, building applications in Docker doesn’t cover all of the possible use cases and complicates onboarding for new users. It also only supports Linux as a target platform, while we’re looking for a solution that can be generalized for all possible platforms.

Alternative Bundle Formats

One alternative is to allow only a single host β†’ target platform combination per bundle, but this may complicate distribution of destinations bundles in some scenarios. The existing .artifactbundle format is flexible enough to support bundles with a single or multiple combinations.

Different formats of destination bundles can be considered, but we don't think those would be significantly different from the proposed one. If they were different, this would complicate bundle distribution scenarios for users who want to publish their own artifact bundles with executables, as defined in SE-0305.

Future Directions

Identifying Platforms with Dictionaries of Properties

Platform triples are not specific enough in certain cases. For example, aarch64-unknown-linux host triple can’t prevent a user from installing a CC destination bundle on an unsupported Linux distribution. In the future we could deprecate hostTriple and destinationTriple JSON properties in favor of dictionaries with keys and values that describe aspects of platforms that are important for destinations. Such dictionaries could look like this:

"destination": {
  "kernel": "Linux",
  "libcFlavor": "Glibc",
  "libcMinVersion": "2.36",
  "cpuArchitecture": "aarch64"
  // more platform capabilities defined here...
}

A toolchain providing this information could allow users to refer to these properties in their code for conditional compilation and potentially even runtime checks.

SwiftPM Plugins for Remote Running, Testing, Deployment, and Debugging

After an application is built with a CC destination, there are other development workflow steps to be improved. We could introduce new types of plugins invoked by swift run and swift test for purposes of remote running, debugging, and testing. For Linux as a target platform, these plugins could delegate to Docker for running produced executables.

swift destination select subcommand

While swift destination select subcommand or a similar one make sense for selecting a CC destination instead of passing --destination to swift build every time, users will expect swift run and swift test to also work for the target platform previously passed to swift destination select. That’s out of scope for this proposal on its own and depends on making plugins (from the previous subsection) or some other remote running and testing implementation to fully work.

SwiftPM and SourceKit-LSP improvements

It is a known issue that SwiftPM can’t run multiple concurrent builds for different target platforms. This may cause issues when SourceKit-LSP is building a project for indexing purposes (for a host platform by default), while a user may be trying to build for a target platform for example for testing. One of these build processes will fail due to the process locking the build database. A potential solution would be to maintain separate build databases per platform.

Another issue related to SourceKit-LSP is that it always build and indexes source code for the host platform. Ideally, we want it to maintain indices for multiple platforms at the same time. Users should be able to select to target platforms and corresponding indices to enable semantic syntax highlighting, auto-complete, and other features for areas of code that are conditionally compiled with #if directives.

Source-Based CC Destinations

One interesting solution is distributing source code of a minimal base destination, as explored by Zig programming language. In this scenario, a cross-compilation destination binaries are produced on the fly when needed. We don't consider this option to be mutually exclusive with solutions proposed in this document, and so it could be explored in the future for Swift as well. However, this requires reducing the number of dependencies that Swift runtime and core libraries have.

40 Likes

Wrt to "Prior Art": I've already implemented something like that for Swift: GitHub - SPMDestinations/homebrew-tap: Homebrew Formulas to install Swift Cross Compilers on macOS (e.g. targeting Ubuntu)., though it would be nice to have better builtin support!

2 Likes

Thanks! A link to this repository is already included in the "Motivation" section of the pitch.

5 Likes

I only have time to skim it, but +1. I've wanted us to productise this for a long time.

A couple of thoughts:

  1. I don't think we should reuse the .artifactbundle extension. Extensions are cheap; why not define a new one which precisely explains what this is?

  2. I think each CC bundle should be from 1 host to 1 target, not N hosts to N targets. Somebody on an x86 Mac will never need the ARM host tools or vice versa, and somebody targeting an ARM linux box with ubuntu focal won't need x86 jammy, etc.

  3. I'm sure clients will want to install additional libraries, either by providing an overlay or by modifying the local copy of the sysroot. Things that may help this are the ability to clone a CC destination and the ability to print the local path of an installed sysroot.

6 Likes

Overall +1 for the pitch! It would be great to have such β€œSwift SDKs” for cross compilation.

Some instant thoughts:

I would like to see destination bundles available from registries, which could make use of the proposed authentication system and identifier design. This should be natural for internal distribution.

While it’s a valid use case, I don’t think we should ask anyone who only wants an additional library to do so… As @Karl pointed out, we should explore some way to β€œextend” a CC destination bundle, something like the FROM keyword in Dockerfile.

Reusing the artifact bundle design is okay for me, but I'm wondering if a CC destination really is, by definition, an β€œartifact” for SwiftPM and its users. Maybe we should use just a different file extension β€” and maybe we should do so for the plugins.

This is, IMO, already a big problem for cross-platform developers (and tool authors!) even without extended destination support. SourceKit-LSP basically has no knowledge of β€œdestination”. I believe the two sides of work are decoupled at least for now.

3 Likes

Thanks for the feedback, happy to see that you feel positive about this in general!

Personally, I feel that sticking to the precedent makes user experience consistent with SE-0305. The extension is general enough to cover both cases, and SE-0305 did not go with .executableArtifact or anything else more concrete. If the community disagrees with the approach SE-0305 took and prefers very specific extensions for every artifact bundle type, that means we'd have to amend SE-0305 as well, if that's at all possible.

That's an overly broad generalization, on both sides.

For the host, Universal Binaries on macOS exist for a reason. Destination bundles against which I'm currently testing my prototype implementation reuse Swift toolchains downloaded from Swift.org - Download Swift, which are all Universal Binaries on macOS. Supporting N:N destinations is what allows reusing such toolchains with almost no modifications, and destination generator very closely matches the generation script that was already hosted in the SwiftPM repository for years. That generator script produces a 2:1 destination that supports both arm64 and x86_64 hosts.

On the target side, a developer interested in server-side Swift could deploy to both x86_64 and aarch64 servers in AWS. An engineer interested in comparing behaviors between two platforms could deploy to a raspbian Docker image running locally on their Intel laptop, and then compiling for aarch64 for deployment on the real hardware. A destination supporting both targets could reuse a lot of shared files like headers and whatever other files are portable enough between CPU architectures, just considering it from a perspective of saving disk space when compared to 2 separate 1:1 destinations.

What's proposed in the pitch does not preclude 1:1 host:platform destinations from existing. Users and destination authors are free to choose 1:1 destinations if they wish so.

That's addressed in the "CC Destination Bundle Generation" subsection. People interested in customizing a destination can do so freely if destination's generator source code is available. Artifact bundles on their own are ideally immutable, hence the checksum verification that we propose. We're also considering how viable it would be for destinations to be sandboxed on macOS in the future. Just for that reason, I don't think tweaking a destination bundle instead of regenerating a new one from a customized destination generator is a good idea.

By analogy, if one wishes to get a customized executable binary of an arbitrary program for which source is available, they'd just modify the source and rebuild instead of cloning program's binary executable and applying binary patches on that. Similarly, we envision destination generators as means of customization. As mentioned, we intend to provide source code for a Docker-based destination generator, where adding a library would be equivalent to adding a line in a Dockerfile. That's only provided as one possible option out of many. We're prescribing only the archive format, the generator will be provided as a potential reference.

4 Likes

Thanks! This was called "SDKs" in one of earlier drafts, but deemed confusing, especially as bits of toolchains can be included in bundles, not just SDKs. Additionally, Xcode already sets a precedent for calling something like this a "destination".

We could consider this in the future, but for now we're taking a piecemeal approach of just defining the distribution format and nothing else. If people feel positive about the format and we're successful with its adoption, then we can start thinking how to integrate with other tools and services, including registries.

Delegating to Docker for customizing a destination is not prescribed here, only given as an example. Other destination generators can freely coexist and support other ways of adding libraries. Even right now, if you're interested in modifying something built with destination.json V1 schema, it's potentially as easy as adding a package on a line defining pkg_names variable in the old generator script.

For this pitch limited in scope, it didn't feel right to reinvent something similar to Dockerfile from scratch. More specialized solution could emerge in the future, but right now we're focusing only on the archive format and a few CLI subcommands to consume these archives.

Yes, it really is an artifact for SwiftPM. The destination subcommands are developed in the SwiftPM codebase, and these bundles are fully operated on by SwiftPM, either when using swift build or the subcommands on their own. This could also tie in with an ability to specify supported destinations (maybe as dependencies?) in Package.swift in the future, in a similar way that binaryTarget bundles are specified as dependencies already. Obviously, if that expansion to Package.swift is ever considered, it would come as a separate pitch.

1 Like

This is a fantastic step in the right direction imho.

Does this then perhaps codify some of the process to add valid destination characteristics for folks bringing up new support? I have found that is definitely a stumbling block to advancing new platforms. For example - I would love to be able, for my personal projects, have atmel-avr as a target but adding that support to the toolchain is daunting to say the least.

How do you see the distribution and management of destination support (particularly things like SDK's or toolchain binaries)? Do you envision folks building these destination support packages and distributing them somehow? If that is the case then perhaps it might be interesting to have a default layout that works well for common systems that are already supported by Swift?

I definitely think that having cross compilation support will vastly improve our overall support for multiple platforms for all of the ecosystem.

4 Likes

BTW: This is one thing SPMDestination currently does, it splits that script into the three separate, distinct tasks (IIRC building the host toolchain, grabbing the destination toolchain and building the SDK).

2 Likes

As the maintainer of a cross compilation toolchain for Debian 11 armhf (Armv7), where the destination.json files needs to be generated on the host because of absolute paths, having a distributable bundle with relative paths and the needed binaries is exactly what is needed. I think this will greatly help Swift for Linux Arm, WASM and Android development. The proposal as presented seems very well thought out, even for some edge cases like when multiple destinations support the same triple. I like the idea of reusing .artifactbundle and having it support multiple hosts and destinations. While I don't have any proposed changes, I do have a couple of questions:

  1. A bundle like swift-5.7_ubuntu.artifactbundle could also have aarch64-unknown-linux-gnu toolchain, allowing destination bundles be installed on multiple host platforms?
  2. Will v1 destination files, e.g. swift build --destination destination.json, be deprecated?
  3. Will the Swift language VS Code plugin support selecting an installed destination for code completion, indexing and compiling?
  4. Will SwiftPM build tools be supported? They are completely broken right now with swift build --destination. It seems to me that this should be a simple fix and SwiftPM should just target the host platform it's running on when compiling the SwiftPM Build Tools, regardless if swift build --destination or swift destination is used, just like Package.swift manifest are always evaluated on the host platform. The plugin itself is being built for the host platform, it's just the build tool that is incorrectly cross compiled.
2 Likes

Can somehow speak for this: I’ve worked on the initial support for cross-compilation, but the work is greatly challenged by SourceKit-LSP, which totally doesn’t recognize destination files and doesn’t support multiple destinations correctly.

The experience we can reach so far is awkward and will result in a bunch of terrible workarounds. Once SK-LSP issue is resolved, I would pick up the work again.

2 Likes

Thank you!

The only part it would codify is how users of a new platform would cross-compile to it when it's fully supported. We could probably publish some guidelines for how to implement such support, but this is something that feels more suitable for the workgroups working on documentation and contributor experience. The process is too specific for every platform, as it spans multiple projects: LLVM, clang, Swift toolchain, Swift runtime, stdlib, core libraries, SwiftPM etc. One size clearly won't fit all platforms.

Yes, this is the goal. The plan is to publish an open-source package written in Swift that generates a destination for specific Linux distributions, as one possible "reference" destination generator. People interested in distributing destination bundles are then free to run the generator on their local machine or with some arbitrary CI setup. Then they can redistribute resulting bundles as they see convenient, according to open-source licenses of software included in the bundle. People interested in sharing their customizations are welcome to fork the generator repository, when it becomes available.

The layout is quite flexible, as we have separate customizable sdkRootDir, toolchainBinDir, and runtimeDir properties in destination.json for that. There's a certain directory layout for libraries and headers that the toolchain expects. We intend to document for people interested in writing their own destination generator from scratch.

1 Like

Thanks for the feedback!

That's correct. It could either contain multiple destinations, each for a specific host platform, or a single destination that relies on universal binaries for supporting both x86_64 and arm64 host macOS platforms.

This is a possibility, but we need to go through Swift Evolution process first if/when this becomes a proposal. I personally don't see standalone v1 destination.json files as flexible enough to keep them around. If v2 kept the same JSON property names as v1, it would be a superset of v1, only with added support for relative paths. But names of v1 properties were inconsistent with other APIs and info.json manifest files, so this seemed like a good opportunity to clean that up and bump the schema version number. What would be the case for sticking to v1 after v2 is available?

I can't speak for that, as I'm working on neither SourceKit-LSP, nor the VS Code plugin. I mentioned this area in "Future Directions", because to me personally this seems like a possible next step.

I'm aware of this issue, but I can't provide specific details for it right now.

1 Like

This is certainly something I would want to add support for, but as @stevapple points out there are changes needed to be made to SourceKit-LSP before we can have a satisfactory solution. Having this proposal though will focus minds which helps a lot.

1 Like

From what I understood of the proposal, this is great and could be a huge quality of life improvement. Thanks for pushing this forward!

I must say I don't really understand the .artifactbundle naming either and I have the impression that implementation detail is distracting from the substance of the proposal a bit (at least in my reading of it). As I was reading the proposal I kept jumping between "great!" and "huh? is this even relevant to me?" and I found the artifact bundle naming in particular to be a large factor in the potential misunderstandings.

Overall, I think I'm +1, but I'm still not actually 100% sure – it's late here and maybe when I re-read it tomorrow things will be clearer to me.

As for the destination.json schema, it is unclear to me what the sdkRootDir is as opposed to the runtimeDir. The sdkRoot in the current destination.json is actually what I'd call the sysroot in a C compilation (whereas I would have expected it to be the the root of a toolchain bundle). In the new schema, the idea of the resourceDir has disappeared – that is fine to me, because I found that naming quite confusing, but what has it been replaced with? And what is supposed to be in the runtimeDir?

One other thought, is there a strong reason we shouldn't use something like swift build --toolchain /Path/To/toolchain.xctoolchain in preference to all this? I generally find it pretty clear what the xctoolchain bundles are doing, but is that just the wrong level of abstraction?

Should we expect to see xctoolchain bundles without the platform SDKs in future and install the SDKs separately instead? I'm thinking Wasm and Android specifically here

1 Like

Thanks for the feedback!

Even though it's a pitch and not a formal proposal, I tried to follow the Swift Evolution Proposal template as closely as possible, which does require specifying at least some of the implementation details.

Can you elaborate what specifically in the proposal felt irrelevant? As for the naming, it's inherited from SE-0305, but I'm willing to consider alternatives if any are proposed.

Yes, sdkRootDir is the sysroot containing libraries and headers. It's a path for compile-time dependencies, while runtimeDir is a place where we'd put dependencies and tools to be used at run time. runtimeDir is not used in the implementation in any way right now, but it's mentioned as an extension for future directions. For example, it could contain a debugger or deployment helpers that would make swift run and swift test work with the target platform.

I'm not sure what you're referring to with resourceDir? It wasn't in the previous version of the schema in the first place. It's also not mentioned anywhere in the SwiftPM codebase.

A custom toolchain is just a single component that may or may not be needed for cross-compilation. If the top-level toolchain provides all of the support needed for cross-compiling, the custom toolchain becomes redundant, and the only thing you need is an SDK for your platform. This is already almost the case for Ubuntu Linux, where essentially the only thing we need in the .xctoolchain directory is lld. What's missing in your invocation, is the -sdk flag, and the proposed solution abstracts over all that. We'd prefer users who don't care about low-level details also not to care about toolchains and SDKs, but to operate on the higher level of destination bundles.

What would be the benefit of having a toolchain that's not functional without a corresponding SDK?

1 Like

Great work @Max_Desiatov! It's a big step for Swift toward being a cross compile friendly language!
The proposal and the direction overall look reasonable to me. Let me put some of my concerns here.

The most critical missing piece is: How does the swiftc driver know where the target-specific resource directory is placed?

As @Geordie_J mentioned, the proposal is missing resource directory concept, which is typically named lib/swift or lib/swift_static and passed as -resource-dir <path> to the driver and the frontend.

Resource directory is a directory containing:

  • Standard library modules and pre-compiled libraries:
    e.g. libswiftCore.so, Swift.swiftmodule, ...
  • Core library headers/modules and pre-compiled libraries:
    e.g. libFoundation.so, Foundation.swiftmodule, CoreFoundation/module.modulemap, ...
  • Clang compiler-rt libraries
  • swiftrt.o, which contains image constructor for metadata registration
  • ...
Current resource directory layout
β”œβ”€β”€ Block
β”‚   └── Block.h
β”œβ”€β”€ CFURLSessionInterface
β”‚   β”œβ”€β”€ CFURLSessionInterface.h
β”‚   └── module.map
β”œβ”€β”€ CFXMLInterface
β”‚   β”œβ”€β”€ CFXMLInterface.h
β”‚   └── module.map
β”œβ”€β”€ CoreFoundation
β”‚   β”œβ”€β”€ CFArray.h
β”‚   β”œβ”€β”€ CFAttributedString.h
β”‚   β”œβ”€β”€ CFAvailability.h
β”‚   β”œβ”€β”€ CFBag.h
β”‚   β”œβ”€β”€ CFBase.h
β”‚   β”œβ”€β”€ CFBinaryHeap.h
β”‚   β”œβ”€β”€ CFBitVector.h
β”‚   β”œβ”€β”€ CFBundle.h
β”‚   β”œβ”€β”€ CFBundlePriv.h
β”‚   β”œβ”€β”€ CFByteOrder.h
β”‚   β”œβ”€β”€ CFCalendar.h
β”‚   β”œβ”€β”€ CFCalendar_Internal.h
β”‚   β”œβ”€β”€ CFCharacterSet.h
β”‚   β”œβ”€β”€ CFCharacterSetPriv.h
β”‚   β”œβ”€β”€ CFData.h
β”‚   β”œβ”€β”€ CFDate.h
β”‚   β”œβ”€β”€ CFDateComponents.h
β”‚   β”œβ”€β”€ CFDateFormatter.h
β”‚   β”œβ”€β”€ CFDateInterval.h
β”‚   β”œβ”€β”€ CFDateIntervalFormatter.h
β”‚   β”œβ”€β”€ CFDictionary.h
β”‚   β”œβ”€β”€ CFError.h
β”‚   β”œβ”€β”€ CFKnownLocations.h
β”‚   β”œβ”€β”€ CFLocale.h
β”‚   β”œβ”€β”€ CFLocaleInternal.h
β”‚   β”œβ”€β”€ CFLocking.h
β”‚   β”œβ”€β”€ CFLogUtilities.h
β”‚   β”œβ”€β”€ CFMachPort.h
β”‚   β”œβ”€β”€ CFMessagePort.h
β”‚   β”œβ”€β”€ CFNotificationCenter.h
β”‚   β”œβ”€β”€ CFNumber.h
β”‚   β”œβ”€β”€ CFNumberFormatter.h
β”‚   β”œβ”€β”€ CFPlugIn.h
β”‚   β”œβ”€β”€ CFPlugInCOM.h
β”‚   β”œβ”€β”€ CFPreferences.h
β”‚   β”œβ”€β”€ CFPriv.h
β”‚   β”œβ”€β”€ CFPropertyList.h
β”‚   β”œβ”€β”€ CFRegularExpression.h
β”‚   β”œβ”€β”€ CFRunArray.h
β”‚   β”œβ”€β”€ CFRunLoop.h
β”‚   β”œβ”€β”€ CFRuntime.h
β”‚   β”œβ”€β”€ CFSet.h
β”‚   β”œβ”€β”€ CFSocket.h
β”‚   β”œβ”€β”€ CFStream.h
β”‚   β”œβ”€β”€ CFStreamPriv.h
β”‚   β”œβ”€β”€ CFString.h
β”‚   β”œβ”€β”€ CFStringEncodingConverter.h
β”‚   β”œβ”€β”€ CFStringEncodingConverterExt.h
β”‚   β”œβ”€β”€ CFStringEncodingExt.h
β”‚   β”œβ”€β”€ CFTimeZone.h
β”‚   β”œβ”€β”€ CFTree.h
β”‚   β”œβ”€β”€ CFURL.h
β”‚   β”œβ”€β”€ CFURLAccess.h
β”‚   β”œβ”€β”€ CFURLComponents.h
β”‚   β”œβ”€β”€ CFURLPriv.h
β”‚   β”œβ”€β”€ CFURLSessionInterface.h
β”‚   β”œβ”€β”€ CFUUID.h
β”‚   β”œβ”€β”€ CFUserNotification.h
β”‚   β”œβ”€β”€ CFUtilities.h
β”‚   β”œβ”€β”€ CoreFoundation.h
β”‚   β”œβ”€β”€ ForFoundationOnly.h
β”‚   β”œβ”€β”€ ForSwiftFoundationOnly.h
β”‚   β”œβ”€β”€ TargetConditionals.h
β”‚   └── module.map
β”œβ”€β”€ FrameworkABIBaseline
β”‚   β”œβ”€β”€ README
β”‚   β”œβ”€β”€ Swift
β”‚   β”‚   β”œβ”€β”€ ABI
β”‚   β”‚   └── API
β”‚   β”œβ”€β”€ _Concurrency
β”‚   β”‚   └── ABI
β”‚   └── nil.json
β”œβ”€β”€ _InternalSwiftScan
β”‚   β”œβ”€β”€ DependencyScan.h
β”‚   β”œβ”€β”€ DependencyScanMacros.h
β”‚   └── module.modulemap
β”œβ”€β”€ _InternalSwiftStaticMirror
β”‚   β”œβ”€β”€ BinaryScan.h
β”‚   β”œβ”€β”€ StaticMirrorMacros.h
β”‚   └── module.modulemap
β”œβ”€β”€ clang -> ../clang/13.0.0
β”œβ”€β”€ dispatch
β”‚   β”œβ”€β”€ base.h
β”‚   β”œβ”€β”€ block.h
β”‚   β”œβ”€β”€ data.h
β”‚   β”œβ”€β”€ dispatch.h
β”‚   β”œβ”€β”€ group.h
β”‚   β”œβ”€β”€ introspection.h
β”‚   β”œβ”€β”€ io.h
β”‚   β”œβ”€β”€ module.modulemap
β”‚   β”œβ”€β”€ object.h
β”‚   β”œβ”€β”€ once.h
β”‚   β”œβ”€β”€ queue.h
β”‚   β”œβ”€β”€ semaphore.h
β”‚   β”œβ”€β”€ source.h
β”‚   └── time.h
β”œβ”€β”€ linux
β”‚   β”œβ”€β”€ Cxx.swiftmodule
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftdoc
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftinterface
β”‚   β”‚   └── x86_64-unknown-linux-gnu.swiftmodule
β”‚   β”œβ”€β”€ Distributed.swiftmodule
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftdoc
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftinterface
β”‚   β”‚   └── x86_64-unknown-linux-gnu.swiftmodule
β”‚   β”œβ”€β”€ Glibc.swiftmodule
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftdoc
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftinterface
β”‚   β”‚   └── x86_64-unknown-linux-gnu.swiftmodule
β”‚   β”œβ”€β”€ RegexBuilder.swiftmodule
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftdoc
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftinterface
β”‚   β”‚   └── x86_64-unknown-linux-gnu.swiftmodule
β”‚   β”œβ”€β”€ Swift.swiftmodule
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftdoc
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftinterface
β”‚   β”‚   └── x86_64-unknown-linux-gnu.swiftmodule
β”‚   β”œβ”€β”€ SwiftOnoneSupport.swiftmodule
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftdoc
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftinterface
β”‚   β”‚   └── x86_64-unknown-linux-gnu.swiftmodule
β”‚   β”œβ”€β”€ _Concurrency.swiftmodule
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftdoc
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftinterface
β”‚   β”‚   └── x86_64-unknown-linux-gnu.swiftmodule
β”‚   β”œβ”€β”€ _Differentiation.swiftmodule
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftdoc
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftinterface
β”‚   β”‚   └── x86_64-unknown-linux-gnu.swiftmodule
β”‚   β”œβ”€β”€ _RegexParser.swiftmodule
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftdoc
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftinterface
β”‚   β”‚   └── x86_64-unknown-linux-gnu.swiftmodule
β”‚   β”œβ”€β”€ _StringProcessing.swiftmodule
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftdoc
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftinterface
β”‚   β”‚   └── x86_64-unknown-linux-gnu.swiftmodule
β”‚   β”œβ”€β”€ libBlocksRuntime.so
β”‚   β”œβ”€β”€ libFoundation.so
β”‚   β”œβ”€β”€ libFoundationNetworking.so
β”‚   β”œβ”€β”€ libFoundationXML.so
β”‚   β”œβ”€β”€ libXCTest.so
β”‚   β”œβ”€β”€ lib_InternalSwiftScan.so
β”‚   β”œβ”€β”€ lib_InternalSwiftStaticMirror.so
β”‚   β”œβ”€β”€ libdispatch.so
β”‚   β”œβ”€β”€ libicudataswift.so -> libicudataswift.so.65.1
β”‚   β”œβ”€β”€ libicudataswift.so.65 -> libicudataswift.so.65.1
β”‚   β”œβ”€β”€ libicudataswift.so.65.1
β”‚   β”œβ”€β”€ libicui18nswift.so -> libicui18nswift.so.65.1
β”‚   β”œβ”€β”€ libicui18nswift.so.65 -> libicui18nswift.so.65.1
β”‚   β”œβ”€β”€ libicui18nswift.so.65.1
β”‚   β”œβ”€β”€ libicuucswift.so -> libicuucswift.so.65.1
β”‚   β”œβ”€β”€ libicuucswift.so.65 -> libicuucswift.so.65.1
β”‚   β”œβ”€β”€ libicuucswift.so.65.1
β”‚   β”œβ”€β”€ libswiftCore.so
β”‚   β”œβ”€β”€ libswiftCxx.so
β”‚   β”œβ”€β”€ libswiftDispatch.so
β”‚   β”œβ”€β”€ libswiftDistributed.so
β”‚   β”œβ”€β”€ libswiftGlibc.so
β”‚   β”œβ”€β”€ libswiftRegexBuilder.so
β”‚   β”œβ”€β”€ libswiftRemoteMirror.so
β”‚   β”œβ”€β”€ libswiftSwiftOnoneSupport.so
β”‚   β”œβ”€β”€ libswift_Concurrency.so
β”‚   β”œβ”€β”€ libswift_Differentiation.so
β”‚   β”œβ”€β”€ libswift_RegexParser.so
β”‚   β”œβ”€β”€ libswift_StringProcessing.so
β”‚   β”œβ”€β”€ libswiftstd.so
β”‚   β”œβ”€β”€ std.swiftmodule
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftdoc
β”‚   β”‚   β”œβ”€β”€ x86_64-unknown-linux-gnu.swiftinterface
β”‚   β”‚   └── x86_64-unknown-linux-gnu.swiftmodule
β”‚   └── x86_64
β”‚       β”œβ”€β”€ Dispatch.swiftdoc
β”‚       β”œβ”€β”€ Dispatch.swiftmodule
β”‚       β”œβ”€β”€ Foundation.swiftdoc
β”‚       β”œβ”€β”€ Foundation.swiftmodule
β”‚       β”œβ”€β”€ FoundationNetworking.swiftdoc
β”‚       β”œβ”€β”€ FoundationNetworking.swiftmodule
β”‚       β”œβ”€β”€ FoundationXML.swiftdoc
β”‚       β”œβ”€β”€ FoundationXML.swiftmodule
β”‚       β”œβ”€β”€ SwiftGlibc.h
β”‚       β”œβ”€β”€ XCTest.swiftdoc
β”‚       β”œβ”€β”€ XCTest.swiftmodule
β”‚       β”œβ”€β”€ glibc.modulemap
β”‚       β”œβ”€β”€ libcxxshim.h
β”‚       β”œβ”€β”€ libcxxshim.modulemap
β”‚       β”œβ”€β”€ libstdcxx.h
β”‚       β”œβ”€β”€ libstdcxx.modulemap
β”‚       └── swiftrt.o
β”œβ”€β”€ migrator
β”‚   β”œβ”€β”€ ios4.json
β”‚   β”œβ”€β”€ ios42.json
β”‚   β”œβ”€β”€ macos4.json
β”‚   β”œβ”€β”€ macos42.json
β”‚   β”œβ”€β”€ overlay4.json
β”‚   β”œβ”€β”€ overlay42.json
β”‚   β”œβ”€β”€ tvos4.json
β”‚   β”œβ”€β”€ tvos42.json
β”‚   β”œβ”€β”€ watchos4.json
β”‚   └── watchos42.json
β”œβ”€β”€ os
β”‚   β”œβ”€β”€ generic_unix_base.h
β”‚   β”œβ”€β”€ generic_win_base.h
β”‚   └── object.h
β”œβ”€β”€ pm
β”‚   β”œβ”€β”€ ManifestAPI
β”‚   β”‚   β”œβ”€β”€ PackageDescription.swiftdoc
β”‚   β”‚   β”œβ”€β”€ PackageDescription.swiftmodule
β”‚   β”‚   └── libPackageDescription.so
β”‚   β”œβ”€β”€ PluginAPI
β”‚   β”‚   β”œβ”€β”€ PackagePlugin.swiftdoc
β”‚   β”‚   β”œβ”€β”€ PackagePlugin.swiftmodule
β”‚   β”‚   └── libPackagePlugin.so
β”‚   └── llbuild
β”‚       β”œβ”€β”€ libllbuild.so
β”‚       └── libllbuildSwift.so
β”œβ”€β”€ shims
β”‚   β”œβ”€β”€ AssertionReporting.h
β”‚   β”œβ”€β”€ CoreFoundationShims.h
β”‚   β”œβ”€β”€ FoundationShims.h
β”‚   β”œβ”€β”€ GlobalObjects.h
β”‚   β”œβ”€β”€ HeapObject.h
β”‚   β”œβ”€β”€ KeyPath.h
β”‚   β”œβ”€β”€ LibcOverlayShims.h
β”‚   β”œβ”€β”€ LibcShims.h
β”‚   β”œβ”€β”€ MetadataSections.h
β”‚   β”œβ”€β”€ Random.h
β”‚   β”œβ”€β”€ RefCount.h
β”‚   β”œβ”€β”€ Reflection.h
β”‚   β”œβ”€β”€ RuntimeShims.h
β”‚   β”œβ”€β”€ RuntimeStubs.h
β”‚   β”œβ”€β”€ SwiftStdbool.h
β”‚   β”œβ”€β”€ SwiftStddef.h
β”‚   β”œβ”€β”€ SwiftStdint.h
β”‚   β”œβ”€β”€ System.h
β”‚   β”œβ”€β”€ Target.h
β”‚   β”œβ”€β”€ ThreadLocalStorage.h
β”‚   β”œβ”€β”€ UnicodeData.h
β”‚   β”œβ”€β”€ Visibility.h
β”‚   β”œβ”€β”€ _SwiftConcurrency.h
β”‚   β”œβ”€β”€ _SwiftDistributed.h
β”‚   └── module.modulemap
└── swiftToCxx
    β”œβ”€β”€ _SwiftCxxInteroperability.h
    β”œβ”€β”€ _SwiftStdlibCxxOverlay.h
    └── experimental-interoperability-version.json

The resource directory path is currently determined by the compiler driver by the following rule:

  1. If -resource-dir is specified, respect the given path.
  2. Otherwise, if compiling for static executable or -static-stdlib, use compiler-relative ../lib/swift_static.
  3. Otherwise, use compiler-relative ../lib/swift.

The current proposal requires putting the target-specific resource directories into host toolchain or users need to pass -resouce-dir manually.
And also, the resource directory should be propagated to the clang driver at link-time to resolve some clang resources like libclang_rt.builtins-x86_64.a, but those resources are placed in destination bundles and clang executable is in host toolchain. (Sidenote: Currently, resource-dir is always resolved as clang compiler-relative path because swiftc driver does not propagate it for now.)

How should resource directory be resolved considering destination bundle and -static/-static-executable/-static-stdlib options in swiftc driver? Do we need to introduce destination-bundle-relative concept into the swiftc driver?

3 Likes

It's assumed by the toolchain to be in /usr/lib path relative to sdkRootDir.

I don't think that's the case. This directory path is inferred from sdkRootDir. In my understanding, it was previously inferred from the -sdk option, and specifying sdkRootDir is equivalent to passing a path with -sdk.

I don't think we need to, this already worked with the older destination.json schema, which didn't require specifying -resource-dir separately.

Okay, so you mean "SDK" should contain both platform system things and platform-specific Swift things.
I remember there are several discussions around here before. I think it's a good time to formalize "SDK" term in Swift.

Correct, I think the best way to describe it as a sysroot. I'm not sure though if the conventional meaning of "sysroot" also includes all build tools participating in cross-compilation? If so, that's the only difference in our case, since some build tools can be supplied by the top-level toolchain.