[PITCH] Support for binary dependencies

Hello,

this draft introduces support for binary dependencies in SwiftPM. It will allow vendors to upload artifacts for all supported platforms which will be resolved, downloaded and linked by SwiftPM. There are still some open things left, which can be found at the bottom, but we wanted to get community feedback before we continue with this.

https://github.com/FranzBusch/swift-evolution/blob/baf4f09dac450c91ece59dee45b1288b18e02ba5/proposals/0000-swiftpm-binary-dependencies.md

Binary dependencies

Introduction

SwiftPM currently supports source-only packages for several languages, and with
a very proscriptive build model which considerably limits exactly how the
compilation of the source can be performed. While this makes packages consistent
and to some extent "simple", it limits their use in several important cases:

  • Software vendors who wish to provide easy integration with the package
    manager, but do not deliver source code, cannot integrate.
  • Existing code bases which would like to integrate "simply" with SwiftPM, but
    require more complicated build processes, have no recourse.

For example, consider these use cases:

  • Someone want's to create a Swift package for
    generating LLVM code. However, LLVM's build process is
    far more complex than can be currently fit into SwiftPM's build model. This
    makes building an easy to use package difficult.
  • A third-party wants to provide a Swift SDK for easily integrating their
    service with server-side Swift applications. The SDK itself relies on
    substantial amounts of internal infrastructure the company does not want to
    make available as open source.
  • A large company has an internal team which wants to deliver a Swift package
    for use in their iOS applications, but for security reasons cannot publish
    the source code.

This proposal defines a new SwiftPM feature to allow SwiftPM to accept some
forms of "binary packages". This proposal is intentionally written to be a
address the above use cases explicitly, it does not define a general
purpose "binary artifact" mechanism intended to address other use cases (such as
accelerating build performance). The motivations for this are discussed in more
detail below.

Swift-evolution thread: Discussion thread topic for that proposal

Pitch Thread: SPM Support for Binaries Distribution - #24 by hartbit

Motivation

SwiftPM has a large appeal to certain developer communities, like the iOS
ecosystem, where it is currently very common to rely on closed source
dependencies such as Firebase, GoogleAnalytics, Adjust and many more. Existing
package managers like Cocoapods support these use cases. By adding such support
to SwiftPM, we will unblock substantially more adoption of SwiftPM within those
communities.

Prior to Swift 5.1, the Swift compiler itself did not expose all of the features
(like ABI compatibility) required to build a workable solution. Now that those
features are present, it makes sense to reevaluate the role of binary packages.

The goal of this proposal is to make consumption of binary packages as
described above easy, intuitive, safe, and consistent. This proposal
does not attempt to provide any affordances for the creation of the binary
package itself. The overall intent of this proposal is to allow consumption of
binary packages where necessary, but not to encourage their use or faciliate a
transition from the existing source-based ecosystem to a binary one.

This proposal is also focused at packages which come exclusively in binary form,
it explicitly does not introduce a mechanism which allows a package to be
present in either source or binary form. See alternatives considered for more
information on this choice.

Proposed solution

To enable binary dependencies we have to make two changes in the Package.swift manifest file, one for the vendor and one for the consumer. First, we propose to add a new target type which describes a binary target. An example of such a package can be seen below:

 let package = Package(
     name: "LibPNG",
     products: [
         .library(name: "LibPNG", targets: ["LibPNG"])
     ],
     dependencies: [
         .package(url: "http://example.com.com/ExamplePackage/ExamplePackage", from: "1.2.3")
     ],
     targets: [
         .target(
             name: "LibPNG",
             dependencies: [
                 "CLibPng"
             ]),
         .binaryTarget(
             name: "CLibPng",
             artifacts: [
             		.artifact(
             			source: .url("https://github.com/firebase/firebase-ios-sdk/releases/download/6.2.0/Firebase-6.2.0.zip"),
             			.when(platforms: [.macOS, .iOS])
         			),
         			.artifact(
             			source: .url("https://github.com/firebase/firebase-ios-sdk/releases/download/6.2.0/Firebase-6.2.0.zip")
             			.when(platforms: [.linux], architectures: [.x86])
             		],
         		dependencies: [
         			"ExamplePackage"
         		])
     ]
 )

Secondly to use such a binary dependency in another package we have to explicitly opt-in as showcased below.

     let package = Package(
         name: "Paper",
         products: [...],
         dependencies: [
             .package(url: "http://example.com.com/ExamplePackage/ExamplePackage", from: "1.2.3"),
             .package(url: "http://some/other/lib", .exact("1.2.3"), allowsBinary: true)
         ],
         targets: [...]
     )

Packages are allowed to contain a mix of binary and source targets. This is
useful when, for example, providing a pre-built or closed source C library
alongside an open source set of Swift bindings for the library.

To be built, a package which has binary targets must be either the root
package, or must be included via a .package declaration that includes the
allowsBinary: true attribute. Similarly, any package must follow the same
requirements to itself use the allowsBinary: true. This ensures that any areas
in a packages transitive graph which might add a dependency on a binary package
are explicitly declared. This is intended to prevent binary artifacts from being
transparently introduced without explicit consenst up the entire dependency
chain.

When a package is built that depends upon any product with a binary target, the
package manager will search the artifacts declaration list to find an artifact
which matches the current build target (platform, architecture, etc.). This list
will be searched in order, and the first matching artifact will be used. It is
the job of the package author/published to provide an appropriate set of
artifacts for the use cases the package wishes to support.

Detailed design

The design consists of the following key points:

  • New PackageDescription API for defining a binary target.
  • New PackageDescription conditional APIs (as used in BuildSettingCondition)
    for describing a specific artifact with an appropriate level of granularity.
  • New parameter on a package dependency declaration to allow use of binary artifacts.
  • A new convention-based platform-specific layout for a binary target artifact.
  • New requirements for the Package.resolved file when using binary packages.
  • A new mechanism for downloading binary target artifacts.

Terminology:

  • Technically, a target is binary or not. However, we anticipate that often a
    single package will consist of either exclusively source or binary targets. We
    will use the term binary package to refer to any package which contains at
    least one binary product. Similarly, a binary product is one which contains
    at least one binary target.

Our design attempts to optimize for the following goals:

  • Ease of use for clients
  • Ease of implementation in existing SwiftPM
  • Ease of maintenance in the face of an evolving SwiftPM
  • Understandable composition with current and upcoming SwiftPM features
  • Support existing well-known occurrences of binary artifacts in the existing
    (often iOS focused) target developer market.

while keeping the following as non-goals:

  • Ease of production of binary packages
  • Simplicity of binary artifact distribution mechanism
  • Widespread use of binary packages

New PackageDescription API

BinaryTarget

Since, a binary target is different compared to a source only target, we propose to introduce a new struct Artifact. This struct defines a targets associated artifacts.

public struct Artifact {
    public enum Source {
        case url(String)
    }

    public let source: Source
    public let condition: ArtifactCondition
}

Furthermore, we propose to add a new artifacts: [Artifacts]? property to the Target, as well as extend the initlizer with this paramter and create a new static method called .binaryTarget(). Lastly, we propose to exten the TargetType enum with a new case called binary.

ArtifactCondition

To describe for what platform and architecture any given artifact is, we propose to create a new ArtifactCondition, similar to the BuildSettingCondition.

/// Represents an architecture that usually corresponds to a processor architecture such as
/// x86 or ARM.
public struct Architecture {

    /// The name of the platform.
    fileprivate let name: String

    private init(name: String) {
        self.name = name
    }

    public static let x86: Platform = Platform(name: "x86")
    public static let arm: Platform = Platform(name: "ARM")

}

public struct ArtifactCondition: Encodable {

    private let platforms: [Platform]
    private let architectures: [Architecture]?

    private init(platforms: [Platform], architecture: [Architecture]?) {
        self.platforms = platforms
        self.architectures = architectures
    }

    /// Create an artifact condition.
    ///
    /// - Parameters:
    ///   - platforms: The platforms for which this condition will be applied.
    ///   - architectures: The architectures for which this condition will be applied.
    public static func when(
        platforms: [Platform],
        architectures: [Architecture]? = nil
        ) -> ArtifactCondition {
        return ArtifactCondition(platforms: platforms, architecture: architectures)
    }
}

PackageDescription

To include binary packages it is required to opt-in. For this we propose to modify the Dependency struct and add a new property allowsBinary.

      public class Dependency: Encodable {
        public enum Requirement {
            ...
        }

        /// The url of the dependency.
        public let url: String

        /// The dependency requirement.
        public let requirement: Requirement

        public let allowsBinary: Bool

        /// Create a dependency.
        init(url: String, requirement: Requirement, allowsBinary: Bool = false) {
            self.url = url
            self.requirement = requirement
            self.allowsBinary = allowsBinary
        }
    }

New Package.resolved Behavior

  • FIXME

Resolution

Package resolution and dependency expression will not be impacted by this change (except where explicitly noted).

Binary Target Artifact Format

SwiftPM supports various platforms and for each of them we need to find a format for the artifacts. Below is a list with a convention for the artifiacts that we expect for each platform.

Dynamic Static Executables
Apple (Swift) XCFramework XCFramework bin
Apple (C) XCFramework XCFramework bin
"POSIX" (Swift) module.swiftmodule/architecture.swiftmodule module.swiftmodule/architecture.swiftinterface module.swiftmodule/architecture.swiftinterface lib/libTargetName.so lib/libTargetName.a bin
"POSIX" (C) lib/libTargetName.so headers lib/libTargetName.a bin

Security

Since binary only dependencies are not inspectable and one has to extend a certain trust to the third party it should be an opt-in feature. This means when declaring a dependency one has to explicitly allow the usage of binary frameworks. Furthermore, the hash of the binary should also be stored in the package resolved to avoid that the vendor changes the artifact behind a version without anyone noticing.

Impact on existing packages

No current package should be affected by this change since this is only an additive in enabling SwiftPM to use binary dependencies.

Alternatives considered

General Approach

There are three popular use cases for binary packages (terminology courtesy
of
Tommaso Piazza). They
are all related, but for the purposes of this proposal we will distinguish them:

  1. "Vendored binaries" (no source available, or cannot be built from source)
  2. "Artifact cache" (pre-built version of packages which are available in source form)
  3. "Published & tagged binaries" (the package manager heavily depends on
    published and tagged binary artifacts)

In the first case, binary packages are used because there is no other viable
alternative. In the second case, binary artifacts are used to either accelerate
development (by eliminating existing build or analysis steps), or to simplify
cognitive load (e.g. by removing uninteresting sources from display in an IDE
with package integration). In the third case, the very mechanism the package
manager uses to resolve dependencies is deeply integrated with the publishing of
a binary artifact. While the third approach is popular in certain ecosystems and
package managers like Maven, we consider it out of scope given SwiftPM's current
decentralized architecture, and we will ignore it for the remained of this
proposal.

The proposal explicit sets out to solve the first use case; a natural question
is should the second use case be supported by the same feature. In this
proposal, we chose not to go that route, for the following reasons:

  • When used as a build or space optimization, artifact caching is a general
    purpose strategy which can be applied to any package. SwiftPM was explicitly
    designed in order to allow the eventual implementation of performant,
    scalable, and even distributed caches for package artifacts. Artifact caching
    is something we would like to "just work" in order to give the best possible
    user experience.

    In particular, when artifact is employed "manually" to achieve the above
    goals, it often introduces certain amounts of ambiguity or risk. From the
    user's perspective, when the source of a package is available, then one would
    typically like to think of the artifact cache as a perfect reproduction of
    "what would have been built, if I built it myself". However, leveraging a
    binary package like mechanism instead of explicit tool support for this often
    means:

    • There is almost no enforcement that the consumed binary artifact matches the
      source. The above presumption of equivalence makes such artifact caches a
      ripe opportunity for embedding malware into an ecosystem.

    • The consumer does not always have control over the artifact production. This
      interacts adversely with potential future SwiftPM features which would allow
      the build of a package to be more dependent on its consumer (e.g. allowing
      compile-time configuration "knobs & switches").

    • The artifact cache "optimization" may not apply to all packages, or may
      require substantial manual effort to maintain.

  • When used as a workflow improvement (e.g. to reduce the scope of searches),
    our position is that the user would ultimately have a better user experience
    by explicitly enumarting and designing features (either in SwiftPM, or in
    related tools) to address these use cases. When analyzed, it may become clear
    that there is more nuance to the solution than an artifact caching scheme
    could reasonably support.

  • The choice to support both source and binary packages in the same mechanism
    imposes certain requirements on the design, which makes it more complex than the
    existing proposal. In particular, it means that the metadata about how the
    source and artifacts are mapped must be kept somewhere adjacent to but
    distinct from the package description (since a source package needs to define
    its source layout). However, such a mechanism must also be defined in a way
    that works when no source layout is present to support binary only packages.

    Finally, since it would be a feature with user-authored metadata, such a
    mechanism would need to be updated when any other SwiftPM enhancement
    introduces or changes the nature of the source layout specification.

Taken together, the above points led us to focus on a proposal focused at
"vendored binaries", while our hope is that artifact caching eventually becames
a built-in and automatic feature of the package manager which applies to all
packages.

Binary Signatures

We considered adding signature checks during the checkout of binary dependencies but when these have transitive dependencies it gets complicated expressing that in the Package.swift.

     let package = Package(
         name: "Paper",
         products: [...],
         dependencies: [
             .package(url: "http://some/other/lib", .exact("1.2.3"), binarySignature: .any),
             .package(url: "http://some/other/lib", .exact("1.2.3"), binarySignature: .gpg("XXX")"),
         ],
         targets: [...]
     )

Support for various artifact stores

Initially, we considered the various artifact stores on the market and how we can integrate with them. We decided to support a URL based artifact definition for the first implementation since the various providers require each their own method of authentication. However, we wanted to keep the possiblity for future additions of providers open; therefore, we made the source of an artifiact an enum which can be extended.

Possible artifact stores we considered:

  • Github releases
  • Github packages
  • Gitlab
  • Bitbucket
  • Artifactory, Nexus etc.

TODO

  • FIXME: Add information on integration with any resources proposal (XFrameworks support them right, how about linux though?)
  • FIXME: Add information on dSYMs (XCFrameworks support them out of the box right?)
  • FIXME: More on security
  • FIXME: Goals (easy for consumers)
  • FIXME: Transitive behavior
  • FIXME: Discuss concern with explosion of artifacts (consequence of putting at
    the target level).
26 Likes

Thanks for this pitch, it seems like an interesting idea! I do have some concerns about the pitch as it relates to non-Apple platforms, though.

Firstly, and I appreciate that this is a problem SwiftPM already has: Linux is not a platform that one can meaningfully target. Linux is not a homogeneous collection of environments that differ only in GUI toolkits, but a number of wildly different environments that share a common kernel. The only code that truly targets Linux is kernel extensions or anything making raw syscalls (libc, Go). All other code actually targets a complex matrix of dependency versions.

We can get away with this vagueness about the "Linux" platform when distributing source, as we can resolve the dependencies using compile time macros and package managers, but when distributing binaries this is no longer true. This is because, unlike Apple's platforms and Windows, there is simply no such thing as "the Linux ABI" (again, excluding making literal raw syscalls). A given Linux distribution may define a stable ABI within a single release (e.g. Ubuntu 18.04), but even there it is rare for that ABI to be stable when moving forward to a new release: only when staying within that specific release is there any hope that the ABI of the system will remain stable.

Additionally, these ABIs are not consistent with one another. As a general rule, binaries compiled on a single Linux distribution are not portable to another, as the set of libraries required for the binary to run likely were compiled with different options, and often have a non-overlapping set of versions available.

As an additional note, not only is "Linux" not an appropriate platform specifier, but "ARM" is not an appropriate architecture specifier either. In fact, neither is "x86"! At the very least we should note that Swift does support 32-bit environments, and so at the very least we need a 32bit/64bit split (there are no fat binaries on Linux). But even that isn't sufficient: see Swift Linux layout considerations (aka Linux is difficult, lets go shopping) for a more comprehensive list of what it means to have an actual description of an ARM ABI.

For the moment, then, I would suggest that we forbid distributing binary packages on Linux until we come up with a more holistic solution for how we plan to do that. I've spun off a separate thread for that discussion: Binary dependencies on Linux.

7 Likes

Thanks Cory for your reply! We discussed this as well and we are aware that supporting linux with binaries is very complicated and we are not sure if we can support that from the get-go. All your points lay out how complicated the linux environment can be. If the community is okay with having binary dependencies only for Apple-platforms in the beginning, I would be completely fine with this. The only thing we should make sure is that our artifact conditions are able to support linux binaries in the future if we ever want that.

Thanks for creating the other post, let's continue the discussion there and focus here on Apple platforms :)

5 Likes

In spite of these challenges, many projects do successfully distributed precompiled Linux archives.

The intent here is that the condition language would grow large enough that someone who wanted to could vend a number of Linux archives (enough to support interested parties).

You are right that this doesn’t compose very well with the spirit of this proposal in that it doesn’t support a package being either a binary or source, but I still believe it has considerable value to Linux.

I agree, but I think it may be better to consider the Linux support as a separate step, rather than try to do both, or to ship a partially working Linux version we have to break in the future to get it to be more expressive. Essentially, let’s follow the Python community and make it work in the easy case first, then generalise to the harder one.

1 Like

What would the concrete concern be here? "Getting more expressive" as I imagine it would involve extending the artifact condition API, which would most likely not be a breaking change. I don't feel that Linux support will require significant extra initial implementation burden, so I would prefer to start with something which provides "some" value, so that we can begin learning how it needs extension.

While it is not the primary intent of this proposal, it does provide a "release valve"/escape hatch for projects that people want to tinker with as SwiftPM dependencies, but are too complex to build in SwiftPM. I believe this is an important enough use case to motivate having Linux support, even if incomplete.

How do you propose to extend it? Right now it calls "Linux" the platform, without any guidance as to what the actual target is or what the dependencies are. Would the platform have sub-levels, requiring associated data or function calls? Would it be versioned, such that there was a "Linux2" platform if we raised the minimum required libc ABI? Would we ever want to support a musl-based version of Linux? What about using libstdc++ instead of libc++?

Right now the thing that ships is unusable, in my opinion, because it underspecifies the ABI. The result is that we are defining an implicit ABI. This implicit ABI causes all kinds of problems:

  1. It's impossible to know ahead of time whether your specific Linux project can be built to target this ABI, because you don't know what it is.
  2. It guarantees bug reports of "your package causes linker errors" that will be a nightmare to debug because they result from ABI incompatibilities.
  3. That ABI is lowest-common-denominator: ARM cannot have hardware floating point because it's not clear if it's supported, as one example, or a bunch of glibc functionality may not be usable.
  4. That ABI is also implicitly too specific: in the absence of certainty it may be reasonably assumed that the supported ABI is "whatever the oldest Swift.org supported toolchain is", which means "Ubuntu 14.04". That may mean that community Swift support has to deal with a number of "SwiftPM packages" that do not install on their system, making them distinctly second class.

Do we have concrete examples? Why is the current model of "have a system dependency" not a suitable solution? What dependencies are they expecting to have? It's not clear to me if the proposal that exists today would solve their problem, and it seems to me to be vastly less pressing than the issue of getting Apple platforms to work.

I'm not motivated by the "learn how it needs to be extended" argument. This problem is well understood. We are not the first group of people to have to face the question of what it means to build something that will run on a wide variety of Linuxes. I'm proposing we take the time to actually solve that problem in a way that allows SwiftPM to work well for the foreseeable future.

1 Like

I really don't think this counts as security. At best the "opt-in" mechanism is a user annoyance and does nothing to prevent the malicious usage of binary resources. Users will merely copy-paste the Package.swift line from the installation instructions, not noticing the difference at all. So I don't think further complicated SPM's usage will do much to address whatever issue this is supposed to address.

I've read about the security concerns of binary and other resources a few times during the discussion SPM's support, but I have yet to see a real breakdown of the issues. Any proposal that adds such support should both clearly list the concerns as well as the mechanisms being being employed to mitigate them.

It also seems likely that ArtifactCondition should be a separately proposed mechanism for a generialized conditional support capability, rather than the partial mechanism proposed here.

1 Like

I don't think the allowsBinary opt-in is sufficient in establishing the ability to vet which binaries exist in your dependency tree. As it stands right now, one would be opting in a whole subtree of one's dependency graph into using any number of binary dependencies. I think we need explicit opt-in of the specific binaries being used.

2 Likes

I will freely admit to not having deeply considered this question (not that I am unaware of it).

Here is what I had in mind though:

  • "Linux" would be a static platform, which I agree is underspecified, but it matches Swift's #if os(Linux).
  • We would have other "condition" types to represent various bits of nuance. The idea would be that by combining various specifiers you could end up with an accurate representation.
  • To completely spitball, we might have conditions like osVariant(is: .ubuntu14_04) or libc(is: .glibc26) or supportsHardwareFloatingPoint etc etc
  • The idea is that by representing these as additional condition types, the API is extensible.

I agree with your concerns in general, but this seems more like a hypothetical problem than one that would occur very much in practice in today's world, where swift.org publishes support for relatively few Linux variants (and there are non-trivial hurdles to supporting more). I think if the initial set of conditions was enough to discriminate between various Ubuntu variants, for example, that >90% of use cases would be covered.

I'm not motivated by the "learn how it needs to be extended" argument. This problem is well understood. We are not the first group of people to have to face the question of what it means to build something that will run on a wide variety of Linuxes. I'm proposing we take the time to actually solve that problem in a way that allows SwiftPM to work well for the foreseeable future.

Do you have any thoughts to what that solution would look like? My idea was to have various condition types, which is why I felt an initial foray which was an extensible API would be acceptable.

I'd love references to how other people have solved this problem, but I do think there is some uniqueness to Swift's approach simply because of how constraining getting the language/PM itself to work on the platform are.

1 Like

That sounds extremely annoying for something like Firebase Analytics, which pulls in a variety of binary dependencies from Google. What issue is this sort of opt-in mechanism trying to address, and why is it important?

Do you have any preferred suggestions for how to accomplish that?

1 Like

Not any fully formed ones, I think the main question is how to define identity for binaries. Since they're referenced by URL only, their identity would need to be based on that as well.

A simple way of doing it could be an URL prefix match, one could opt-in to a specific release by specifying the whole URL, a specific project (in the GitHub sense) by something like "GitHub - firebase/firebase-ios-sdk: Firebase iOS SDK" and into a whole org by "Firebase · GitHub". This might address @Jon_Shier's concern a little bit as well, as someone using Firebase could opt-in to allowing any binaries from the Firebase org.

4 Likes

In any dependency system, depending on a package is implicitly extending trust to the owner of that Git URL to only vend appropriate source code to you (and Package.resolved then lets you trust that you got what you thought, if relied upon).

The intent of the allowsBinary flag is merely to enforce that one does not unintentionally start getting binaries from a package which wasn't previously assumed to have them. The problem it is solving is little more than that binaries are harder to inspect than source, and also expected to be much rarer. The hope is that packages would limit use of this flag only to very trusted sources.

I do agree the mechanism is rather weak, and could easily just be copied pasted without much thought. However, note that the existing system is very weakly protected. It isn't clear to me that binary packages significantly change the status quo, relative to the current system; the proposal is written to expect binary packages to be rare, so the risks w.r.t. source packages are generally far higher than binary packages, even if binary packages are harder to validate.

I like this idea, it is one we didn't consider (we talked about things like vetting signatures, or pinning to very specific artifacts, neither of those seemed to work well at least in the current environment).

My worry with this line of thinking is that it is self-fulfilling. If we make design decisions based on the fact that swift.org supports only one Linux platform and so we only need to optimise for that platform, we make it incrementally harder for other Linux platforms to obtain Swift support. The repeated application of this pattern leads to the conclusion that Swift doesn't support Linux, it supports Ubuntu, which is not quite the same thing.

I am not the expert on getting Swift to build on weird Linuxes (I defer to @kevints in that regard) but I don't believe that Swift is particularly special compared to, say, Python or Ruby. My motivation here is to not build a world where you are disincentivized from getting Swift to build on, say, Arch Linux, because once you do a chunk of the package ecosystem cannot be used on your platform without modifying the package manager (and thereby going through the Swift Evolution process). This seems like an unnecessary punishment for Linux developers, and an unnecessary brake on the evolution of Swift on the Server.

My proposal would be to broadly emulate the model used by Python. That means:

  1. We define a subset of the truly ABI-stable libraries available on Linux, pick a (fairly old) ABI version, and allow you to dynamically link those dependencies. Anything not present there must be statically linked (conveniently, we can potentially use SwiftPM to express this quite nicely with binary package support).
  2. We require you to specify the architecture whose ABI you're targeting. This requires specificity at the level of a target triple, with the advantage that one third of that triple (Linux) is implied, and another third can be inferred from the final element. This triple can thus be rebuilt for making linking decisions at build time.
  3. We version this, such that we can respond to changes in major Linux distribution patterns (e.g. what do we do if the C++ runtime available on Linuxes starts to bifurcate?)
  4. We provide Docker images that contain a build environment that perfectly matches our requirements. If your code builds in that, then it is safe to distribute. Any dependency you install into that docker builder must come along with your binary target.

This seems to me to cover the basic case. It doesn't require multiplication of switches, it works with currently unsupported Linuxes, and it allows us to build tooling to make it easy to build, distribute, and validate these packages.

As a next level, if we wanted to we could also version these things with a Swift version, which would implicitly also version the copies of Foundation, Dispatch, and other support libraries that we ship as part of the Swift distribution. This could then be used to provide those libraries to the external build process if we wanted. I'd be inclined to do this later though: beginning with just gating the ABI is probably the best choice.

5 Likes

While I'm here, I'll note that the actual current tip of the Python community thinking is PEP 600.

2 Likes

My worry with this line of thinking is that it is self-fulfilling. If we make design decisions based on the fact that swift.org supports only one Linux platform and so we only need to optimise for that platform, we make it incrementally harder for other Linux platforms to obtain Swift support. The repeated application of this pattern leads to the conclusion that Swift doesn't support Linux, it supports Ubuntu, which is not quite the same thing.

I definitely agree with this concern, makes sense.

I'll need to think more and follow up on some of Python's experience here...

I like this idea the most so far if we go beyond just a flag to allow the use of binaries.

One thing that came up in an offline discussion between @lukasa and me was this:

This proposal was written to heavily prefer only using the binary package for cases where the source is "unavailable", although it is recognize that it also serves as an "escape hatch" for things that can't be built with SwiftPM today. That latter part is only intended to be a short term concession though, not the desired end state of the ecosystem.

Thus, for example, I would not want to see "hard to build" libraries being supplied as binary packages, that is not intended by the spirit of the proposal. As @lukasa noted, the current draft could do with making this much clearer.

3 Likes