[Second review] SE-0387: Cross-Compilation Swift SDKs (previously Destination Bundles)

The second review of SE-0387: Cross-Compilation Destination Bundles begins now and runs through May 5th, 2023.

This proposal has already been accepted in principle . We would like to keep this review focused on Cross-Compilation Triples Nomenclature:

Here is the proposal from @Max_Desiatov:

Cross-Compilation Triples Nomenclature

Alternatives Considered So Far

In the original text of SE-0387 proposal we used “build-time triple” and “run-time triple” nomenclature instead of “host triple” and “target triple”, as was previously established by LLVM/Clang and ad-hoc cross-compilation support in Swift. Main motivation for that was to avoid possible confusion, due to “target” having a different meaning in the context of SwiftPM and build systems in general.

Specifically in SwiftPM, we’d like to keep open a future direction where users can specify a triple they’re building for in each build system target declaration. Suppose one would want to build a compiler with SwiftPM this way:

targets: [
  .target(
    name: "clang",
    triple: .host
  ),
  .target(
    name: "compiler-rt",
    triple: .target
  ),
  // ...
]

Notice that .target would have different meaning in each context, either a build system target passed in an array as a targets argument to Package initializer, or .target as a way to specify a triple of the machine this target is built for. We’d like to avoid ambiguity both in the package manifests and in colloquial use of the nomenclature when talking about cross-compilation in SwiftPM.

During SE-0387 review some aslo argued that “build-time triple” and “run-time triple” were unclear and a break from established convention was unexpected. On the other hand, we also received feedback on inconsistency between the convention used CMake/LLVM/Clang (“host”/“target”) and a convention used by GCC, Autotools, and subsequently picked up by the Meson build system (“build”/“host“).

Nomenclature Comparison for Basic Cross-Compilation

To address this feedback we also considered alternative naming that breaks away from these conventions and also avoids possible confusion with the “target” term used by SwiftPM already in Package.swift manifests. In this alternative nomenclature:

  • a builder is a machine that builds code
  • an executor is a on which code generated by the builder is executed.

One downside of this approach is that “executor” is ambiguous with previously proposed actor executors in Swift. On the other hand, Swift Concurrency and cross-compilation support in SwiftPM are different enough contexts, so possible ambiguity in “executor” usage is less likely than with the “target” term.

For a basic cross-compilation scenario that SE-0387 is focused on, a comparative nomenclature table looks like this:

GCC/Autotools/Meson LLVM/Clang/CMake Original Proposal Text Alternative
build host build-time triple builder
host target run-time triple executor

Canadian Cross

Another aspect of feedback received during SE-0387 review was related to a so called Canadian Cross scenario. In this case an intermediate platform is used for cross-compilation when the original platform is too slow to build the compiler itself.

For example instead of building on Raspberry Pi a Swift compiler that can cross-compile to x86 Linux, one may prefer to build that compiler on M2 Mac much more quickly than building it on the Raspberry Pi itself.

In this example M2 Mac building the compiler would be called a build platform and Raspberry Pi running the compiler would be called a host platform, by both GCC and LLVM/Clang.

The alternative nomenclature of builder and executor platforms could be extended with a runnerplatform, which is an intermediate platform running the compiler. Here’s the comparative nomenclature table updated for Canadian Cross:

GCC/Autotools/Meson LLVM/Clang/CMake Original Proposal Text Alternative
build build build-time triple builder
host host N/A runner
target target run-time triple executor

Summary

The original SE-0387 proposal text did not consider more complex scenarios, but we’d like to make sure that the nomenclature we use is consistent. We feel that formalizing cross-compilation in SwiftPM is a good opportunity to make it less ambiguous. From that perspective builder/runner/executor matches that criteria and is proposed to the community for additional review.

We’d also like to gather more feedback on each of these variants, taking into account all the pros and cons:

  • GCC/Autotools nomenclature is quite widely established on platforms where these are the default build tools, although they may be less familiar to developers targeting Darwin platforms. In Canadian Cross scenarios the appearance of “target platform” introduces possible confusion with SwiftPM package manifest targets.
  • LLVM/Clang/CMake nomenclature is what Swift developers may already be used to, with the downside of “target” term used more frequently, which means that there’s a bigger risk of ambiguity. On the other hand, sticking to this nomenclature means less churn in our documentation and code. Additionally, consistency with LLVM, Clang, and CMake are important factors too.
  • “Build-time triple” and “run-time triple” terms introduced in the original proposal text seem a bit more verbose, and they did receive some negative feedback. We also haven’t found a good counterpart term for an intermediate platform in Canadian Cross scenarios.
  • “Builder”, “runner”, and “executor” are completely new terms we’re considering here. While this may make them more easily searchable, it also means we’re not following existing conventions. The fact that Swift Concurrency also uses “executors” introduces its own ambiguity, which we think is less harmful than the ambiguity in the use of the term “target”.
4 Likes

I don’t see any advantage to novel terminology. “Runner” sounds like where I run my program, not where I run the build tools.

I know from recent experience that this is an inherently complicated situation, and consistency with other build systems is valuable. Especially since Swift is intrinsically tied with LLVM and Clang, it should use the same terminology.

6 Likes

I was thinking just the same thing. What makes this even more confusing is that there’s a swift run command.

(Edit: Removed part of the quote I didn’t mean to quote.)

2 Likes

The main advantage comes from disambiguating the code snippet shared in the post. I can see that proposed names may take some time to get used to, but if you'd like to stick to Clang/LLVM convention, the need for disambiguation remains.

Would "builder" work better than "runner"? It certainly looks less confusing

targets: [
  .target(
    name: "clang",
    triple: .builder
  ),
  // ...
]

than this

targets: [
  .target(
    name: "clang",
    triple: .target
  ),
  // ...
]

The section about the Canadian Cross already uses the terms “builder”, “runner”, and “executor”. So changing “runner” to “builder” isn’t really possible without finding another term for the original “builder”.

What we actually want to express here are that one platform builds the compiler, one runs the compiler, and one runs the build product, right? So an unambiguous alternative would be:

  • compilerBuilder
  • compilerRunner
  • buildProductRunner

But I guess this isn’t desirable because the split of compilerBuilder and compilerRunner isn’t necessary in the basic cross-compilation case where there’s no third platform involved.

(Also: I should mention that I don’t have any experience with cross compilation whatsoever and only wanted to point out the potential for confusion between runner and swift run :slightly_smiling_face:)

I'd anticipate the need to explicitly specify either of these two outside of a Canadian Cross setup would be small. In simple cross-compilation scenarios one would only refer to the latter two of the three, and compilerRunner is quite unambiguous.

I like how specific these are, but buildProductRunner seems slightly too verbose, maybe productRunner would be ok?

That is, I'm probably ok with these as they gain clarity at a relatively small cost of increased length:

  • compilerBuilder
  • compilerRunner
  • productRunner

Or

  • toolchainBuilder
  • toolchainRunner
  • productRunner

since there are many more tools involved other than just a compiler.

I understand the thought process, but I am a little surprised by this comment.

"Runner" is a pretty common term in the CI/CD space to signify a non-local machine that runs some specific process (often building your application is a main one).

GitLab and Github Actions, for example, both use this term.

That being said, it may also be a reason not to use this shorthand term of "runner", and signify someway that it is the platform of the runner you are describing.

Each phase of cross-compilation runs a compiler that produces an executable that also runs somewhere. “Runner” is too vague of a term.

1 Like

I haven't had time to look at this in detail, but I'd like to ask one thing: if this ends up being named "Swift SDKs" or something like that, could we also update the proposal's filename and title in the repo?

It can be difficult to find things in the evolution repo when the accepted names have diverged greatly from the originally proposed names.

2 Likes

I am strongly opposed to Swift inventing entirely novel terms for this. If we consider this to be an "API" for cross-compilation, then we should embrace the philosophies of the Swift API Design Guidelines:

  • Don’t surprise an expert: anyone already familiar with the term will be surprised and probably angered if we appear to have invented a new meaning for it.
  • Don’t confuse a beginner: anyone trying to learn the term is likely to do a web search and find its traditional meaning.
  • Embrace precedent. Don’t optimize terms for the total beginner at the expense of conformance to existing culture.

Inventing new names fails all three of these checks.

It could be argued that "precedent" here is fuzzy because different build systems have done different things. Even in that case, inventing a whole new standard isn't the right answer. We need to break the tie, and the best way to do that is to acknowledge that Swift is an LLVM- and Clang-based toolchain and that the least confusing option is to adopt the same terminology used by the compiler's underlying technology. That's "host" for the platform where the compiler runs and "target" for the platform where the compiled code will run.

I don't find the argument about target being confusable between a SwiftPM target and a target triple to be compelling for a few reasons:

First, every build system I've interacted with in my career uses "target" to refer to a library or binary that can be built. Almost every build system I've interacted with refers to a "target platform/configuration/environment" as well. These are used in completely different contexts and rarely confused by users.

This is an "expert"-level feature, for some definition of "expert". That is, it is not something your average day-one Swift developer is going to stumble on. I think it's fair that folks doing cross-compilation should be expected to have a deep enough understanding of their environment that they aren't confused by target in the context of a platform and target in the context of a thing being built. If we invent entirely new terms, they will immediately be less discoverable than using existing terms of art, and users of this feature who are used to other tools will be confused about why the terms don't align. This is especially true if they are doing advanced stuff with Clang and have to switch between different terms in the same build.

Lastly, I would expect triple: .target to be the default configuration for a SwiftPM target, and users would only need to specify that argument when they want to reference the host (or some other future alternative platforms).

IMO, the misnamed thing here in SwiftPM's existing target(...), which should be named something like module(...) instead. target in this usage is too vague, given that there are also executableTargets and testTargets—there's nothing that explains to users what a plain old target is. That's out of scope for this proposal, but given that SwiftPM maintainers have expressed a desire to move away from the target suffix, maybe there's a future where target itself is renamed as well, and then we have no need to worry about a conflict with the target triple.

16 Likes

I agree with this and also I don't think that the new terms are reducing confusion even if we consider coming up with new terms a goal.

Agreed, we have previously discussed module or library for this, e.g. this pre-pitch proposed Library. I think it would be fair to mention getting rid of .target() a future direction that could alleviate the concern that lead to inventing new terminology.

4 Likes

I tend agree with the sentiment expressed by @NeoNacho and @allevato, we should try to avoid new terminology where possible

I also agree that the right solution is to change SwiftPM use of the term target for module/library since it is confusing / misleading regardless of the CC proposal

3 Likes

As one of those who do cross-compiling in Swift, I'm happy to see the proposal is very close to being accepted!

First, let me put some comments on CMake things:

  1. The CMake document "Cross Compiling With CMake" says and consistently uses the following definition:

    The system used to build the software will be called the “build host,” and the system for which the software is built will be called the “target system” or “target platform.”

    Even though the variable describing the build host is somehow CMAKE_HOST_SYSTEM, I think both Autotools and CMake agree that the system used to build the software should be called the “build host,”

    So the CMake column in the table of Basic Cross-Compilation would be:

    GCC/Autotools/Meson LLVM/Clang/CMake Original Proposal Text Alternative
    build build build-time triple builder
    host target run-time triple executor
  2. As far as I understand, CMake doesn't provide Canadian Cross support natively. It's usually done by external tools through two-stage Basic Cross-Compilation. So to be precise, the "host platform" for describing the system running the compiler is not defined in CMake.

    GCC/Autotools/Meson LLVM/Clang CMake Original Proposal Text Alternative
    build build build build-time triple builder
    host host N/A N/A runner
    target target target run-time triple executor
  3. Note that CMake uses "target" as a core concept of build work unit as well as SwiftPM.

Having considered the above discussion, I object to introducing Canadian Cross support and new terminologies for the following reasons:

  1. I think Canadian Cross support in SwiftPM is too specific to the compiler. It will be rarely used except for Swift compiler. Autotools uses build/host/target but target is rarely used according to the document. Also it can be done by two stage basic cross-compilation and it would be much simpler.
  2. If we support only basic cross-compilation (it means we only have "build" / "target"), users don't need to specify the compilation triple in Package.swift at all because SwiftPM can automatically distinguish "build" or "target" just by checking whether it's macro or plugin. So there won't be "target of target" in user facing interface.
    (If we support Canadian Cross, users have to specify which one should be compiled for "host" and "target", so we have to expose such interface in that case.)
  3. The CMake build system already has "ambiguity" in terms of "target", but I guess the two "target"s are well far.

For anyone who wants some background on the “build”, “host”, and “target” terms, the GCC documentation has some history. These terms are several decades old at this point.

2 Likes

An approach to stay true to well-established meanings while resolving SPM ambiguity without introducing a whole new terminology would be to add a System suffix, i.e. use buildSystem and targetSystem (as well as hostSystem if we want Canadian Cross support).

1 Like

Thanks, that's a great point, I've updated the proposal text but kept the name of the file as it was. I don't think we have a nice way to add redirects from the old file name if we rename it and I don't want existing links to break.

1 Like

they aren't quite the same thing though? a target identifier can contain special characters (+, |, etc) but a module identifier must be a c99 identifier.

A .target represents a module in SPM's build model; when you build one, that's the output. The fact that the name might be transformed if it's not a valid module identifier doesn't change that.

when parsing package metadata from SPM, you usually get the target identifier, not the module identifier, and you don't want to accidentally plug a target string into some other tooling API that expects a module identifier string, because 99% of the time they are the same but that 1% of the time is a bug. so i have found it helpful to encode the distinction in the type system.

it's fair to ask if this is something that should be surfaced in the PackageDescription API. but the distinction is not completely useless.

Strong Agree