# \[PITCH\] Support for binary dependencies

**URL:** <https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620>\
**Category:** Package Manager\
**Created:** [August 5, 2019, 7:33am UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620 "2019-08-05T07:33:00Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![FranzBusch](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/franzbusch/32/19447_2.png) [@FranzBusch](https://forums.swift.org/u/FranzBusch)\
**Post date:** [August 5, 2019, 7:33am UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/1 "2019-08-05T07:33:00Z")

</div>

Hello,

this draft introduces support for binary dependencies in SwiftPM. It will allow vendors to upload artifacts for all supported platforms which will be resolved, downloaded and linked by SwiftPM. There are still some open things left, which can be found at the bottom, but we wanted to get community feedback before we continue with this.

> [https://github.com/FranzBusch/swift-evolution/blob/baf4f09dac450c91ece59dee45b1288b18e02ba5/proposals/0000-swiftpm-binary-dependencies.md](https://github.com/FranzBusch/swift-evolution/blob/baf4f09dac450c91ece59dee45b1288b18e02ba5/proposals/0000-swiftpm-binary-dependencies.md)

# Binary dependencies

- Proposal: SE-NNNN
- Authors: [Braden Scothern](https://github.com/bscothern), [Daniel Dunbar](https://github.com/ddunbar), [Franz Busch](https://github.com/FranzBusch)
- Review Manager: TBD
- Status: **WIP**

## Introduction

SwiftPM currently supports source-only packages for several languages, and with  
a very proscriptive build model which considerably limits exactly how the  
compilation of the source can be performed. While this makes packages consistent  
and to some extent "simple", it limits their use in several important cases:

- Software vendors who wish to provide easy integration with the package  
manager, but do not deliver source code, cannot integrate.
- Existing code bases which would like to integrate "simply" with SwiftPM, but  
require more complicated build processes, have no recourse.

For example, consider these use cases:

- Someone want's to create a Swift package for  
generating [LLVM](https://llvm.org) code. However, LLVM's build process is  
far more complex than can be currently fit into SwiftPM's build model. This  
makes building an _easy to use_ package difficult.
- A third-party wants to provide a Swift SDK for easily integrating their  
service with server-side Swift applications. The SDK itself relies on  
substantial amounts of internal infrastructure the company does not want to  
make available as open source.
- A large company has an internal team which wants to deliver a Swift package  
for use in their iOS applications, but for security reasons cannot publish  
the source code.

This proposal defines a new SwiftPM feature to allow SwiftPM to accept some  
forms of "binary packages". This proposal is intentionally written to be a  
address the above use cases _explicitly_, it **does not** define a general  
purpose "binary artifact" mechanism intended to address other use cases (such as  
accelerating build performance). The motivations for this are discussed in more  
detail below.

Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/)

Pitch Thread: [SPM Support for Binaries Distribution - #24 by hartbit](https://forums.swift.org/t/spm-support-for-binaries-distribution/25549/24)

## Motivation

SwiftPM has a large appeal to certain developer communities, like the iOS  
ecosystem, where it is currently very common to rely on closed source  
dependencies such as Firebase, GoogleAnalytics, Adjust and many more. Existing  
package managers like Cocoapods support these use cases. By adding such support  
to SwiftPM, we will unblock substantially more adoption of SwiftPM within those  
communities.

Prior to Swift 5.1, the Swift compiler itself did not expose all of the features  
(like ABI compatibility) required to build a workable solution. Now that those  
features are present, it makes sense to reevaluate the role of binary packages.

The goal of this proposal is to make _consumption_ of binary packages as  
described above _easy_, _intuitive_, _safe_, and _consistent_. This proposal  
**does not** attempt to provide any affordances for the creation of the binary  
package itself. The overall intent of this proposal is to allow consumption of  
binary packages _where necessary_, but not to encourage their use or faciliate a  
transition from the existing source-based ecosystem to a binary one.

This proposal is also focused at packages which come exclusively in binary form,  
it explicitly **does not** introduce a mechanism which allows a package to be  
present in either source or binary form. See alternatives considered for more  
information on this choice.

## Proposed solution

To enable binary dependencies we have to make two changes in the `Package.swift` manifest file, one for the vendor and one for the consumer. First, we propose to add a new target type which describes a binary target. An example of such a package can be seen below:

```swift
 let package = Package(
     name: "LibPNG",
     products: [
         .library(name: "LibPNG", targets: ["LibPNG"])
     ],
     dependencies: [
         .package(url: "http://example.com.com/ExamplePackage/ExamplePackage", from: "1.2.3")
     ],
     targets: [
         .target(
             name: "LibPNG",
             dependencies: [
                 "CLibPng"
             ]),
         .binaryTarget(
             name: "CLibPng",
             artifacts: [
             		.artifact(
             			source: .url("https://github.com/firebase/firebase-ios-sdk/releases/download/6.2.0/Firebase-6.2.0.zip"),
             			.when(platforms: [.macOS, .iOS])
         			),
         			.artifact(
             			source: .url("https://github.com/firebase/firebase-ios-sdk/releases/download/6.2.0/Firebase-6.2.0.zip")
             			.when(platforms: [.linux], architectures: [.x86])
             		],
         		dependencies: [
         			"ExamplePackage"
         		])
     ]
 )

```

Secondly to use such a binary dependency in another package we have to explicitly opt-in as showcased below.

```swift
     let package = Package(
         name: "Paper",
         products: [...],
         dependencies: [
             .package(url: "http://example.com.com/ExamplePackage/ExamplePackage", from: "1.2.3"),
             .package(url: "http://some/other/lib", .exact("1.2.3"), allowsBinary: true)
         ],
         targets: [...]
     )

```

Packages are allowed to contain a mix of binary and source targets. This is  
useful when, for example, providing a pre-built or closed source C library  
alongside an open source set of Swift bindings for the library.

To be built, a package which has binary targets _must_ be either the root  
package, or must be included via a `.package` declaration that includes the  
`allowsBinary: true` attribute. Similarly, any package _must_ follow the same  
requirements to itself use the `allowsBinary: true`. This ensures that any areas  
in a packages transitive graph which might add a dependency on a binary package  
are explicitly declared. This is intended to prevent binary artifacts from being  
transparently introduced without explicit consenst up the entire dependency  
chain.

When a package is built that depends upon any product with a binary target, the  
package manager will search the `artifacts` declaration list to find an artifact  
which matches the current build target (platform, architecture, etc.). This list  
will be searched in order, and the first matching artifact will be used. It is  
the job of the package author/published to provide an appropriate set of  
artifacts for the use cases the package wishes to support.

## Detailed design

The design consists of the following key points:

- New `PackageDescription` API for defining a binary target.
- New `PackageDescription` conditional APIs (as used in `BuildSettingCondition`)  
for describing a specific artifact with an appropriate level of granularity.
- New parameter on a package dependency declaration to allow use of binary artifacts.
- A new convention-based platform-specific layout for a binary target artifact.
- New requirements for the `Package.resolved` file when using binary packages.
- A new mechanism for downloading binary target artifacts.

Terminology:

- Technically, a _target_ is binary or not. However, we anticipate that often a  
single package will consist of either exclusively source or binary targets. We  
will use the term _binary package_ to refer to any package which contains at  
least one binary product. Similarly, a _binary product_ is one which contains  
at least one binary target.

Our design attempts to optimize for the following goals:

- Ease of use for clients
- Ease of implementation in existing SwiftPM
- Ease of maintenance in the face of an evolving SwiftPM
- Understandable composition with current and upcoming SwiftPM features
- Support existing well-known occurrences of binary artifacts in the existing  
(often iOS focused) target developer market.

while keeping the following as non-goals:

- Ease of production of binary packages
- Simplicity of binary artifact distribution mechanism
- Widespread use of binary packages

## New `PackageDescription` API

### BinaryTarget

Since, a binary target is different compared to a source only target, we propose to introduce a new struct `Artifact`. This struct defines a targets associated artifacts.

```swift
public struct Artifact {
    public enum Source {
        case url(String)
    }

    public let source: Source
    public let condition: ArtifactCondition
}

```

Furthermore, we propose to add a new `artifacts: [Artifacts]?` property to the `Target`, as well as extend the initlizer with this paramter and create a new static method called `.binaryTarget()`. Lastly, we propose to exten the `TargetType` enum with a new case called `binary`.

### ArtifactCondition

To describe for what platform and architecture any given artifact is, we propose to create a new `ArtifactCondition`, similar to the `BuildSettingCondition`.

```swift
/// Represents an architecture that usually corresponds to a processor architecture such as
/// x86 or ARM.
public struct Architecture {

    /// The name of the platform.
    fileprivate let name: String

    private init(name: String) {
        self.name = name
    }

    public static let x86: Platform = Platform(name: "x86")
    public static let arm: Platform = Platform(name: "ARM")

}

public struct ArtifactCondition: Encodable {

    private let platforms: [Platform]
    private let architectures: [Architecture]?

    private init(platforms: [Platform], architecture: [Architecture]?) {
        self.platforms = platforms
        self.architectures = architectures
    }

    /// Create an artifact condition.
    ///
    /// - Parameters:
    /// - platforms: The platforms for which this condition will be applied.
    /// - architectures: The architectures for which this condition will be applied.
    public static func when(
        platforms: [Platform],
        architectures: [Architecture]? = nil
        ) -> ArtifactCondition {
        return ArtifactCondition(platforms: platforms, architecture: architectures)
    }
}

```

### PackageDescription

To include binary packages it is required to opt-in. For this we propose to modify the `Dependency` struct and add a new property `allowsBinary`.

```swift
      public class Dependency: Encodable {
        public enum Requirement {
            ...
        }

        /// The url of the dependency.
        public let url: String

        /// The dependency requirement.
        public let requirement: Requirement

        public let allowsBinary: Bool

        /// Create a dependency.
        init(url: String, requirement: Requirement, allowsBinary: Bool = false) {
            self.url = url
            self.requirement = requirement
            self.allowsBinary = allowsBinary
        }
    }

```

## New `Package.resolved` Behavior

- FIXME

### Resolution

Package resolution and dependency expression will not be impacted by this change (except where explicitly noted).

## Binary Target Artifact Format

SwiftPM supports various platforms and for each of them we need to find a format for the artifacts. Below is a list with a convention for the artifiacts that we expect for each platform.

| | Dynamic | Static | Executables | |
| --- | --- | --- | --- | --- |
| Apple (Swift) | XCFramework | XCFramework | bin | |
| Apple (C) | XCFramework | XCFramework | bin | |
| "POSIX" (Swift) | module.swiftmodule/architecture.swiftmodule module.swiftmodule/architecture.swiftinterface module.swiftmodule/architecture.swiftinterface lib/libTargetName.so | lib/libTargetName.a | bin | |
| "POSIX" (C) | lib/libTargetName.so headers | lib/libTargetName.a | bin | |
| | | | | |

## Security

Since binary only dependencies are not inspectable and one has to extend a certain trust to the third party it should be an opt-in feature. This means when declaring a dependency one has to explicitly allow the usage of binary frameworks. Furthermore, the hash of the binary should also be stored in the package resolved to avoid that the vendor changes the artifact behind a version without anyone noticing.

## Impact on existing packages

No current package should be affected by this change since this is only an additive in enabling SwiftPM to use binary dependencies.

## Alternatives considered

### General Approach

There are three popular use cases for binary packages (terminology courtesy  
of  
[Tommaso Piazza](https://forums.swift.org/t/spm-support-for-binaries-distribution/25549/32)). They  
are all related, but for the purposes of this proposal we will distinguish them:

1. "Vendored binaries" (no source available, or cannot be built from source)
2. "Artifact cache" (pre-built version of packages which are available in source form)
3. "Published & tagged binaries" (the package manager heavily depends on  
published and tagged binary artifacts)

In the first case, binary packages are used because there is no other viable  
alternative. In the second case, binary artifacts are used to either accelerate  
development (by eliminating existing build or analysis steps), or to simplify  
cognitive load (e.g. by removing uninteresting sources from display in an IDE  
with package integration). In the third case, the very mechanism the package  
manager uses to resolve dependencies is deeply integrated with the publishing of  
a binary artifact. While the third approach is popular in certain ecosystems and  
package managers like Maven, we consider it out of scope given SwiftPM's current  
decentralized architecture, and we will ignore it for the remained of this  
proposal.

The proposal explicit sets out to solve the first use case; a natural question  
is should the second use case be supported by the same feature. In this  
proposal, we chose not to go that route, for the following reasons:

- When used as a build or space optimization, artifact caching is a general  
purpose strategy which can be applied to _any_ package. SwiftPM was explicitly  
designed in order to allow the eventual implementation of performant,  
scalable, and even distributed caches for package artifacts. Artifact caching  
is something we would like to "just work" in order to give the best possible  
user experience.

- When used as a workflow improvement (e.g. to reduce the scope of searches),  
our position is that the user would ultimately have a better user experience  
by explicitly enumarting and designing features (either in SwiftPM, or in  
related tools) to address these use cases. When analyzed, it may become clear  
that there is more nuance to the solution than an artifact caching scheme  
could reasonably support.

- The choice to support both source and binary packages in the same mechanism  
imposes certain requirements on the design, which makes it more complex than the  
existing proposal. In particular, it means that the metadata about how the  
source and artifacts are mapped must be kept somewhere adjacent to but  
distinct from the package description (since a source package needs to define  
its source layout). However, such a mechanism must also be defined in a way  
that works when no source layout is present to support binary only packages.

Taken together, the above points led us to focus on a proposal focused at  
"vendored binaries", while our hope is that artifact caching eventually becames  
a built-in and automatic feature of the package manager which applies to all  
packages.

### Binary Signatures

We considered adding signature checks during the checkout of binary dependencies but when these have transitive dependencies it gets complicated expressing that in the `Package.swift`.

```swift
     let package = Package(
         name: "Paper",
         products: [...],
         dependencies: [
             .package(url: "http://some/other/lib", .exact("1.2.3"), binarySignature: .any),
             .package(url: "http://some/other/lib", .exact("1.2.3"), binarySignature: .gpg("XXX")"),
         ],
         targets: [...]
     )

```

### Support for various artifact stores

Initially, we considered the various artifact stores on the market and how we can integrate with them. We decided to support a URL based artifact definition for the first implementation since the various providers require each their own method of authentication. However, we wanted to keep the possiblity for future additions of providers open; therefore, we made the source of an artifiact an enum which can be extended.

Possible artifact stores we considered:

- Github releases
- Github packages
- Gitlab
- Bitbucket
- Artifactory, Nexus etc.

## TODO

- FIXME: Add information on integration with any resources proposal (XFrameworks support them right, how about linux though?)
- FIXME: Add information on dSYMs (XCFrameworks support them out of the box right?)
- FIXME: More on security
- FIXME: Goals (easy for consumers)
- FIXME: Transitive behavior
- FIXME: Discuss concern with explosion of artifacts (consequence of putting at  
the target level).

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [August 5, 2019, 9:15am UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/2 "2019-08-05T09:15:55Z")

</div>

Thanks for this pitch, it seems like an interesting idea! I do have some concerns about the pitch as it relates to non-Apple platforms, though.

Firstly, and I appreciate that this is a problem SwiftPM already has: Linux is not a platform that one can meaningfully target. Linux is not a homogeneous collection of environments that differ only in GUI toolkits, but a number of wildly different environments that share a common kernel. The only code that truly targets _Linux_ is kernel extensions or anything making raw syscalls (libc, Go). All other code actually targets a complex matrix of dependency versions.

We can get away with this vagueness about the "Linux" platform when distributing source, as we can resolve the dependencies using compile time macros and package managers, but when distributing binaries this is no longer true. This is because, unlike Apple's platforms and Windows, there is simply no such thing as "the Linux ABI" (again, excluding making literal raw syscalls). A given Linux distribution may define a stable ABI within a single release (e.g. Ubuntu 18.04), but even there it is rare for that ABI to be stable when moving forward to a new release: only when staying within that specific release is there any hope that the ABI of the system will remain stable.

Additionally, these ABIs are not consistent with one another. As a general rule, binaries compiled on a single Linux distribution are not portable to another, as the set of libraries required for the binary to run likely were compiled with different options, and often have a non-overlapping set of versions available.

As an additional note, not only is "Linux" not an appropriate platform specifier, but "ARM" is not an appropriate architecture specifier either. In fact, neither is "x86"! At the very least we should note that Swift does support 32-bit environments, and so at the very least we need a 32bit/64bit split (there are no fat binaries on Linux). But even that isn't sufficient: see [Swift Linux layout considerations (aka Linux is difficult, lets go shopping)](https://forums.swift.org/t/swift-linux-layout-considerations-aka-linux-is-difficult-lets-go-shopping/27400) for a more comprehensive list of what it means to have an actual description of an ARM ABI.

For the moment, then, I would suggest that we forbid distributing binary packages on Linux until we come up with a more holistic solution for how we plan to do that. I've spun off a separate thread for that discussion: [Binary dependencies on Linux](https://forums.swift.org/t/binary-dependencies-on-linux/27623).

---

<div class="post-metadata">

**Author:** ![FranzBusch](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/franzbusch/32/19447_2.png) [@FranzBusch](https://forums.swift.org/u/FranzBusch)\
**Post date:** [August 5, 2019, 9:26am UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/3 "2019-08-05T09:26:47Z")

</div>

Thanks Cory for your reply! We discussed this as well and we are aware that supporting linux with binaries is very complicated and we are not sure if we can support that from the get-go. All your points lay out how complicated the linux environment can be. If the community is okay with having binary dependencies only for Apple-platforms in the beginning, I would be completely fine with this. The only thing we should make sure is that our artifact conditions are able to support linux binaries in the future if we ever want that.

Thanks for creating the other post, let's continue the discussion there and focus here on Apple platforms :)

---

<div class="post-metadata">

**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\
**Post date:** [August 5, 2019, 3:11pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/4 "2019-08-05T15:11:12Z")

</div>

In spite of these challenges, many projects do successfully distributed precompiled Linux archives.

The intent here is that the condition language would grow large enough that someone who wanted to could vend a number of Linux archives (enough to support interested parties).

You are right that this doesn’t compose very well with the spirit of this proposal in that it doesn’t support a package being either a binary or source, but I still believe it has considerable value to Linux.

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [August 5, 2019, 4:10pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/5 "2019-08-05T16:10:36Z")

</div>

I agree, but I think it may be better to consider the Linux support as a separate step, rather than try to do both, or to ship a partially working Linux version we have to break in the future to get it to be more expressive. Essentially, let’s follow the Python community and make it work in the easy case first, then generalise to the harder one.

---

<div class="post-metadata">

**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\
**Post date:** [August 5, 2019, 4:32pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/6 "2019-08-05T16:32:08Z")

</div>

> [@lukasa](#):
>
> ... ship a partially working Linux version we have to break in the future to get it to be more expressive. ...

What would the concrete concern be here? "Getting more expressive" as I imagine it would involve extending the artifact condition API, which would most likely not be a breaking change. I don't feel that Linux support will require significant extra initial implementation burden, so I would prefer to start with something which provides "some" value, so that we can begin learning how it needs extension.

While it is not the primary intent of this proposal, it does provide a "release valve"/escape hatch for projects that people want to tinker with as SwiftPM dependencies, but are too complex to build in SwiftPM. I believe this is an important enough use case to motivate having Linux support, even if incomplete.

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [August 5, 2019, 4:59pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/7 "2019-08-05T16:59:16Z")

</div>

> [@ddunbar](#):
>
> What would the concrete concern be here? "Getting more expressive" as I imagine it would involve extending the artifact condition API, which would most likely not be a breaking change.

How do you propose to extend it? Right now it calls "Linux" the platform, without any guidance as to what the actual target is or what the dependencies are. Would the platform have sub-levels, requiring associated data or function calls? Would it be versioned, such that there was a "Linux2" platform if we raised the minimum required libc ABI? Would we ever want to support a musl-based version of Linux? What about using libstdc++ instead of libc++?

Right now the thing that ships is unusable, in my opinion, because it underspecifies the ABI. The result is that we are defining an implicit ABI. This implicit ABI causes all kinds of problems:

1. It's impossible to know ahead of time whether your specific Linux project can be built to target this ABI, because you don't know what it is.
2. It guarantees bug reports of "your package causes linker errors" that will be a nightmare to debug because they result from ABI incompatibilities.
3. That ABI is lowest-common-denominator: ARM cannot have hardware floating point because it's not clear if it's supported, as one example, or a bunch of glibc functionality may not be usable.
4. That ABI is also implicitly too specific: in the absence of certainty it may be reasonably assumed that the supported ABI is "whatever the oldest [Swift.org](http://Swift.org) supported toolchain is", which means "Ubuntu 14.04". That may mean that community Swift support has to deal with a number of "SwiftPM packages" that do not install on their system, making them distinctly second class.

> [@ddunbar](#):
>
> While it is not the primary intent of this proposal, it does provide a "release valve"/escape hatch for projects that people want to tinker with as SwiftPM dependencies, but are too complex to build in SwiftPM. I believe this is an important enough use case to motivate having Linux support, even if incomplete.

Do we have concrete examples? Why is the current model of "have a system dependency" not a suitable solution? What dependencies are they expecting to have? It's not clear to me if the proposal that exists today would solve their problem, and it seems to me to be vastly less pressing than the issue of getting Apple platforms to work.

I'm not motivated by the "learn how it needs to be extended" argument. This problem is well understood. We are not the first group of people to have to face the question of what it means to build something that will run on a wide variety of Linuxes. I'm proposing we take the time to actually solve that problem in a way that allows SwiftPM to work well for the foreseeable future.

---

<div class="post-metadata">

**Author:** ![Jon\_Shier](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/jon_shier/32/921_2.png) [@Jon\_Shier](https://forums.swift.org/u/Jon_Shier)\
**Post date:** [August 5, 2019, 5:08pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/8 "2019-08-05T17:08:58Z")

</div>

> [@FranzBusch](#):
>
> ## Security
> 
> Since binary only dependencies are not inspectable and one has to extend a certain trust to the third party it should be an opt-in feature. This means when declaring a dependency one has to explicitly allow the usage of binary frameworks. Furthermore, the hash of the binary should also be stored in the package resolved to avoid that the vendor changes the artifact behind a version without anyone noticing.

I really don't think this counts as security. At best the "opt-in" mechanism is a user annoyance and does nothing to prevent the malicious usage of binary resources. Users will merely copy-paste the `Package.swift` line from the installation instructions, not noticing the difference at all. So I don't think further complicated SPM's usage will do much to address whatever issue this is supposed to address.

I've read about the security concerns of binary and other resources a few times during the discussion SPM's support, but I have yet to see a real breakdown of the issues. Any proposal that adds such support should both clearly list the concerns as well as the mechanisms being being employed to mitigate them.

It also seems likely that `ArtifactCondition` should be a separately proposed mechanism for a generialized conditional support capability, rather than the partial mechanism proposed here.

---

<div class="post-metadata">

**Author:** ![NeoNacho](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/neonacho/32/1368_2.png) [@NeoNacho](https://forums.swift.org/u/NeoNacho)\
**Post date:** [August 5, 2019, 5:14pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/9 "2019-08-05T17:14:23Z")

</div>

I don't think the `allowsBinary` opt-in is sufficient in establishing the ability to vet which binaries exist in your dependency tree. As it stands right now, one would be opting in a whole subtree of one's dependency graph into using any number of binary dependencies. I think we need explicit opt-in of the specific binaries being used.

---

<div class="post-metadata">

**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\
**Post date:** [August 5, 2019, 5:15pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/10 "2019-08-05T17:15:37Z")

</div>

I will freely admit to not having deeply considered this question (not that I am unaware of it).

Here is what I had in mind though:

- "Linux" would be a static platform, which I agree is underspecified, but it matches Swift's `#if os(Linux)`.
- We would have other "condition" types to represent various bits of nuance. The idea would be that by combining various specifiers you could end up with an accurate representation.
- To completely spitball, we might have conditions like `osVariant(is: .ubuntu14_04)` or `libc(is: .glibc26)` or `supportsHardwareFloatingPoint` etc etc
- The idea is that by representing these as additional condition types, the API is extensible.

I agree with your concerns in general, but this seems more like a hypothetical problem than one that would occur very much in practice in today's world, where [swift.org](http://swift.org) publishes support for relatively few Linux variants (and there are non-trivial hurdles to supporting more). I think if the initial set of conditions was enough to discriminate between various Ubuntu variants, for example, that \>90% of use cases would be covered.

> I'm not motivated by the "learn how it needs to be extended" argument. This problem is well understood. We are not the first group of people to have to face the question of what it means to build something that will run on a wide variety of Linuxes. I'm proposing we take the time to actually solve that problem in a way that allows SwiftPM to work well for the foreseeable future.

Do you have any thoughts to what that solution would look like? My idea was to have various condition types, which is why I felt an initial foray which was an extensible API would be acceptable.

I'd love references to how other people have solved this problem, but I do think there is some uniqueness to Swift's approach simply because of how constraining getting the language/PM itself to work on the platform are.

---

<div class="post-metadata">

**Author:** ![Jon\_Shier](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/jon_shier/32/921_2.png) [@Jon\_Shier](https://forums.swift.org/u/Jon_Shier)\
**Post date:** [August 5, 2019, 5:16pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/11 "2019-08-05T17:16:21Z")

</div>

That sounds extremely annoying for something like Firebase Analytics, which pulls in a variety of binary dependencies from Google. What issue is this sort of opt-in mechanism trying to address, and why is it important?

---

<div class="post-metadata">

**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\
**Post date:** [August 5, 2019, 5:16pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/12 "2019-08-05T17:16:47Z")

</div>

Do you have any preferred suggestions for how to accomplish that?

---

<div class="post-metadata">

**Author:** ![NeoNacho](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/neonacho/32/1368_2.png) [@NeoNacho](https://forums.swift.org/u/NeoNacho)\
**Post date:** [August 5, 2019, 5:21pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/13 "2019-08-05T17:21:51Z")

</div>

Not any fully formed ones, I think the main question is how to define identity for binaries. Since they're referenced by URL only, their identity would need to be based on that as well.

A simple way of doing it could be an URL prefix match, one could opt-in to a specific release by specifying the whole URL, a specific project (in the GitHub sense) by something like "[GitHub - firebase/firebase-ios-sdk: Firebase iOS SDK](https://github.com/firebase/firebase-ios-sdk/)" and into a whole org by "[Firebase · GitHub](https://github.com/firebase/)". This might address @Jon_Shier's concern a little bit as well, as someone using Firebase could opt-in to allowing any binaries from the Firebase org.

---

<div class="post-metadata">

**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\
**Post date:** [August 5, 2019, 5:25pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/14 "2019-08-05T17:25:23Z")

</div>

In any dependency system, depending on a package is implicitly extending trust to the owner of that Git URL to only vend appropriate source code to you (and Package.resolved then lets you trust that you got what you thought, if relied upon).

The intent of the `allowsBinary` flag is merely to enforce that one does not _unintentionally_ start getting binaries from a package which wasn't previously assumed to have them. The problem it is solving is little more than that binaries are harder to inspect than source, and also expected to be much rarer. The hope is that packages would limit use of this flag only to very trusted sources.

I do agree the mechanism is rather weak, and could easily just be copied pasted without much thought. However, note that the existing system is very weakly protected. It isn't clear to me that binary packages significantly change the status quo, relative to the current system; the proposal is written to expect binary packages to be rare, so the risks w.r.t. source packages are generally far higher than binary packages, even if binary packages are harder to validate.

---

<div class="post-metadata">

**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\
**Post date:** [August 5, 2019, 5:26pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/15 "2019-08-05T17:26:30Z")

</div>

I like this idea, it is one we didn't consider (we talked about things like vetting signatures, or pinning to very specific artifacts, neither of those seemed to work well at least in the current environment).

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [August 5, 2019, 5:30pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/16 "2019-08-05T17:30:46Z")

</div>

> [@ddunbar](#):
>
> I agree with your concerns in general, but this seems more like a hypothetical problem than one that would occur very much in practice in today's world, where [swift.org](http://swift.org) publishes support for relatively few Linux variants (and there are non-trivial hurdles to supporting more)

My worry with this line of thinking is that it is self-fulfilling. If we make design decisions based on the fact that [swift.org](http://swift.org) supports only one Linux platform and so we only need to optimise for that platform, we make it incrementally harder for other Linux platforms to obtain Swift support. The repeated application of this pattern leads to the conclusion that Swift doesn't support Linux, it supports Ubuntu, which is not quite the same thing.

> [@ddunbar](#):
>
> I'd love references to how other people have solved this problem, but I do think there is some uniqueness to Swift's approach simply because of how constraining getting the language/PM itself to work on the platform are.

I am not the expert on getting Swift to build on weird Linuxes (I defer to @kevints in that regard) but I don't believe that Swift is particularly special compared to, say, Python or Ruby. My motivation here is to not build a world where you are disincentivized from getting Swift to build on, say, Arch Linux, because once you do a chunk of the package ecosystem cannot be used on your platform without modifying the package manager (and thereby going through the Swift Evolution process). This seems like an unnecessary punishment for Linux developers, and an unnecessary brake on the evolution of Swift on the Server.

> [@ddunbar](#):
>
> Do you have any thoughts to what that solution would look like?

My proposal would be to broadly emulate the model used by Python. That means:

1. We define a subset of the truly ABI-stable libraries available on Linux, pick a (fairly old) ABI version, and allow you to dynamically link those dependencies. Anything not present there must be statically linked (conveniently, we can potentially use SwiftPM to express this quite nicely with binary package support).
2. We require you to specify the architecture whose ABI you're targeting. This requires specificity at the level of a target triple, with the advantage that one third of that triple (Linux) is implied, and another third can be inferred from the final element. This triple can thus be rebuilt for making linking decisions at build time.
3. We version this, such that we can respond to changes in major Linux distribution patterns (e.g. what do we do if the C++ runtime available on Linuxes starts to bifurcate?)
4. We provide Docker images that contain a build environment that perfectly matches our requirements. If your code builds in that, then it is safe to distribute. Any dependency you install into that docker builder must come along with your binary target.

This seems to me to cover the basic case. It doesn't require multiplication of switches, it works with currently unsupported Linuxes, and it allows us to build tooling to make it easy to build, distribute, and validate these packages.

As a next level, if we wanted to we could also version these things with a _Swift_ version, which would implicitly also version the copies of Foundation, Dispatch, and other support libraries that we ship as part of the Swift distribution. This could then be used to provide those libraries to the external build process if we wanted. I'd be inclined to do this later though: beginning with just gating the ABI is probably the best choice.

---

<div class="post-metadata">

**Author:** ![lukasa](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/lukasa/32/1232_2.png) [@lukasa](https://forums.swift.org/u/lukasa)\
**Post date:** [August 5, 2019, 5:35pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/17 "2019-08-05T17:35:33Z")

</div>

While I'm here, I'll note that the actual current tip of the Python community thinking is [PEP 600](https://www.python.org/dev/peps/pep-0600/).

---

<div class="post-metadata">

**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\
**Post date:** [August 5, 2019, 5:37pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/18 "2019-08-05T17:37:05Z")

</div>

> My worry with this line of thinking is that it is self-fulfilling. If we make design decisions based on the fact that [swift.org](http://swift.org) supports only one Linux platform and so we only need to optimise for that platform, we make it incrementally harder for other Linux platforms to obtain Swift support. The repeated application of this pattern leads to the conclusion that Swift doesn't support Linux, it supports Ubuntu, which is not quite the same thing.

I definitely agree with this concern, makes sense.

I'll need to think more and follow up on some of Python's experience here...

---

<div class="post-metadata">

**Author:** ![Braden\_Scothern](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/braden_scothern/32/1580_2.png) [@Braden\_Scothern](https://forums.swift.org/u/Braden_Scothern)\
**Post date:** [August 5, 2019, 5:59pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/19 "2019-08-05T17:59:27Z")

</div>

I like this idea the most so far if we go beyond just a flag to allow the use of binaries.

---

<div class="post-metadata">

**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\
**Post date:** [August 5, 2019, 6:01pm UTC](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620/20 "2019-08-05T18:01:57Z")

</div>

One thing that came up in an offline discussion between @lukasa and me was this:

This proposal was written to heavily prefer _only_ using the binary package for cases where the source is "unavailable", although it is recognize that it also serves as an "escape hatch" for things that can't be built with SwiftPM today. That latter part is only intended to be a short term concession though, not the desired end state of the ecosystem.

Thus, for example, I would not want to see "hard to build" libraries being supplied as binary packages, that is not intended by the spirit of the proposal. As @lukasa noted, the current draft could do with making this much clearer.

[Next page](https://forums.swift.org/t/pitch-support-for-binary-dependencies/27620.md?page=2)