[PITCH] Support for binary dependencies

Development Package Manager
41

+1 on supporting binary dependencies. I'm a little torn on requiring opt-in, though I strongly dislike allowBinary as the mechanism and would vastly prefer the suggested binary signature check.

     dependencies: [
         .package(url: "http://some/other/lib", .exact("1.2.3"), binarySignature: .any),
         .package(url: "http://some/other/lib", .exact("1.2.3"), binarySignature: .gpg("XXX")"),
     ],

By passing in binarySignature, this is your opt-in approach. While it's true that you don't validate dependencies, we don't really do that today. Also, it could be extended to the future to a flat list of dependencies and signatures (or this list could be shared in the package), eg:

.package(url: "http://some/other/lib", .exact("1.2.3"), binarySignature: .gpg("XXX"), dependencySignatures: [
    .package(.url: "http://some/dep1", binarySignature: .gpg("AAA")),
    .package(.url: "http://some/dep2", binarySignature: .gpg("BBB"))
], .dependencySignaturesRequired: .all/.some)

If you really want to validate transitive dependencies, you'll need something like this anyway. The dependencySignaturesRequired could be used to allow for a partial dependency graph being listed.

42

Regarding, the kind of framework we would distribute for Apple platforms, in my opinion XCFrameworks are the only reasonable choice to take here. They support static and dynamic variants, resources and multiple architectures which are stripped automatically by Xcode. In the future, we can still extend it to support different types of binaries but I think we have a solid foundation with XCFrameworks. I personally would like to see support for static libraries as well since I know that for iOS some vendors exists that distribute them.

I think that is an interesting point and I think we should definitely cover that in the proposal.

1 Like
43

I'm not sure what you mean with "by Xcode". In what phase in this happening? Upon archiving?

What about CI/CD processes that don't use Xcode or ApplicationLoader to submit but use other ways like fastlane ?

44

When linking and bundling an iOS application that depends on a precompiled XCFramework, Xcode will pick the correct .framework bundle from within the XCFramework and use that.

Remember, an XCFramework structure is something like this (very loosely defined):

Example.xcframework/
Example.xcframework/arm64/
Example.xcframework/arm64/Example.framework
Example.xcframework/arm32/
Example.xcframework/arm32/Example.framework

so there are multiple copies of the .framework bundle at play, built for different architectures. Xcode will pick whichever is appropriate for the target platform.

This is not part of the proposal, this is how XCFrameworks are implemented in Xcode. The proposal is just saying that this format is the future for binary distributable from Xcode, so it makes sense to support it from the POV of usability.

But that brings an interesting point, if XCFrameworks are to be supported as binary dependencies, will SPM duplicate the logic to link against/bundle them in whatever the output format is?

Perhaps XCFrameworks should have another thread to discuss how it's implemented and what's the internal layout. Are there public docs on it @harlanhaskins?

45

XCFrameworks are publicly documented here: https://help.apple.com/xcode/mac/11.0/index.html?localePath=en.lproj#/dev6f6ac218b.

2 Likes
46

I think a got a little spooked/confused by that comment. I believe it's still the case that SPM can't build iOS/Mac/Watch/tv applications so nothing to worry about there.

However for cli programs...

if XCFrameworks are to be supported as binary dependencies, will SPM duplicate the logic to link against/bundle them in whatever the output format is?

this is a valid question

47

I withdraw my pessimism. :-)

1 Like
48

I imagine we will determine at build time the target architecture and reach into the XCFramework to link agains the Framework inside of it.

49

I am strongly in favour of restricting the scope of this proposal to cover only distributing packages that cannot be distributed in the form of source code. I'd like to move to a world where we can provide the ability to download binary versions of build packages, but in my view we are not ready for that, not least because it requires both the ability for SwiftPM to build more complex projects, and for Swift on Linux to have some notion of a stable ABI. Until that time is reached, I think there is value in addressing this single specific stop-gap of installing packages for which source is not available.

I share @millenomi's concerns about the actual utility of this distribution mechanism on Linux, but I am not strongly opposed to keeping it: my suspicion is simply that no-one will use it.

2 Likes
50

To add my two cents to the XCFramework subthread and address @sergiocampamaā€™s questions:

First of all, the pitch needs a lot more details on this area, so Iā€™d somewhat suggest pending discussion until we have it written (I think we are also very open to collaborators).

Second: no matter the mechanism, the artifact archive is likely to need some kind of manifest file or convention based format so that the PM can locate the artifacts inside. Leveraging XCFramework on Apple platforms if it is an existing ā€œstandardā€ is preferable to doing something from scratch (and might simplify life for producers). If that doesnā€™t make sense for some reasons (likely discovered during filling out the pitch), we would discuss other alternatives.

Third: open question in my mind: should the artifact manifest format be defined in such a way that multiple ā€œvariantsā€ can exist inside of one artifact. This would be useful in allowing manifests to be simpler when supporting many variants, but might cause unnecessary download or a more complex download protocol. It would also likely require replicating some of the artifact condition expression language into the artifacts manifest. Iā€™m interested if anyone has thoughts on this part.

Fourth: with regard to how the artifacts actually are defined ā€” the idea of this proposal is to make the artifacts look very very similar to what SwiftPM itself ā€œwould haveā€ built had it built the source, in order to make the PMā€™s job easier and define how things would evolve as the PM evolved (i.e. the expected format would continue to track how the PM built targets). What that means right now is that an artifact might look a lot like a collection of .o files that get linked in. Dylibs are as @sergiocampama mentioned not going to be useful because the PM itself doesnā€™t have any concepts of how to package or relocate them. The idea of this proposal is to treat that as orthogonal, and something the PM should fix for general use (source packages), and then the binary support would follow suit. I realize this is not a very satisfactory answer, it is just a way to decompose the two problems.

2 Likes
51

Thanks for the summary @ddunbar. The title of this pitch does seem to imply a general solution for binary support, even thought there might be missing pieces to achieve that goal. It sounds like there are different discussions that need to happen to fully resolve this particular design. Taking from the list of things you mentioned:

  1. API for Package.swift: This seems to be the topic that is unblocked from being defined implemented and where discussions should focus right now, based on the other considerations listed below.
    Open questions here:

    1. How to deal with security so that clients can be sure that the artifact they trusted continues to be trusted as the dependency evolve. Another part of this question is the transitive trust for indirect dependencies.
    2. Should the binary targets be defined in the client or in the provider? What I mean by this is whether CompanyA exposes a Package.swift with .binaryTarget pointing to the artifact, or a client developer adding a binary dependency similar to .package, removing the need for CompanyA to provide a Package.swift file at all, at the cost of SPM understanding how to interpret that binary dependency, this is tied to Format for the artifact/artifact manifest

  2. Format for the artifact/artifact manifest: I think this is what Binary Target Artifact Format aims to describe in the pitch, though it might need more elaboration.
    Open questions here are:

    1. @lukasa's question on how to handle Linux ABI across different distributions/platforms. I have no insight here.
    2. Should we choose a format that allows downloading only what's required for the target platform, as opposed to the full artifact with all the binaries, potentially an expensive action if the artifact supports many platforms. Given the push for XCFrameworks as the standard for Apple platforms, this might mean a central service that understands the structure of the XCFrameworks, or a full download of XCFrameworks. My preference here is to download the full artifact for now, as that simplifies current work but still leaves this part open for future discussion and solution

  3. Support for runtime dependencies: This is what I was initially asking about with dynamic frameworks, and it also considers generic resources as well. This presupposes the existence of a bundling solution supported in Linux and macOS. IMO before we consider implementing support for dynamic binary artifacts, this needs to be resolved. It sounds like this particular aspect should be discussed in Swift PM, Bundles and Resources - #36 by nacho4d. Note: this is not a requirement for implementing static artifact support.

In my opinion, these are 3 different and major aspects for support for binary dependencies, and each should have a separate pitch, otherwise we'll just talk over each other and following the discussion will become hard (potentially discouraging other members of the community from participating).

Note that the designs/pitches for API for Package.swift and Format for the artifact/artifact manifest can happen in parallel, but before the implementation phase can start, a solution for runtime dependencies needs to happen first.

1 Like
52
  1. Support for runtime dependencies : This is what I was initially asking about with dynamic frameworks, and it also considers generic resources as well. This presupposes the existence of a bundling solution supported in Linux and macOS. IMO before we consider implementing support for dynamic binary artifacts, this needs to be resolved. It sounds like this particular aspect should be discussed in Swift PM, Bundles and Resources. Note: this is not a requirement for implementing static artifact support.

I agree with your assessment; ideally that would happen concurrently. If not, we should end up with some understanding of binary packages would support the dynamic bundling, once available.

In my opinion, these are 3 different and major aspects for support for binary dependencies, and each should have a separate pitch, otherwise we'll just talk over each other and following the discussion will become hard (potentially discouraging other members of the community from participating).

I agree it is very useful to detangle the separate concerns. Not yet sure we need to have a separate thread for each, but I'm more than happy to do so if others think it is helpful.

2 Likes
53

Thanks folks for putting this pitch together! We (the Firebase team) are excited as this would unblock us from fully supporting SwiftPM for all our SDKs. When it comes time for implementation, we'll ensure this works as expected for Firebase SDKs. A few thoughts and comments as it relates to supporting Firebase's use case:

API for Package.swift

No large concerns here. Having allowsBinary is fine, although I imagine most folks will copy paste from a getting started guide without much thought so I'm not sure how effective it will be for the goal of being more secure.

I believe the API would also allow for our existing scenario of a binary target depending on a source based target (i.e. FirebaseAnalytics depending on FirebaseCore) so that's great!

Format of the artifact/artifact manifest:

We'll likely have to ship XCFrameworks anyways (and potentially shift to only XCFrameworks when Xcode 11 is a requirement to submit to the App Store in order to simplify our packaging), so this isn't a concern for us.

(EDIT: Not Needed) Multiple XCFrameworks per binaryTarget

One thing that could be helpful though is providing a zip file with 1 or multiple XCFrameworks instead of limiting to just one. This would match our existing strategy with CocoaPods (example in the FirebaseAnalytics podspec).

Our current approach on CocoaPods allows us to modularize our builds and SDKs without having to publish multiple podspecs for a single product SDK. It's not a dealbreaker but would prevent us from having individual .binaryTargets for XCFrameworks that wouldn't make sense to expose to developers on their own.

Edit: As pointed out by @NeoNacho, targets aren't visible to end users and we can package multiple .binaryTargets in a product to achieve this. A single xcframework works for us.

Security

Re: @NeoNacho's idea:

A simple way of doing it could be an URL prefix match, one could opt-in to a specific release by specifying the whole URL, a specific project (in the GitHub sense) by something like "GitHub - firebase/firebase-ios-sdk: Firebase SDK for Apple App Development" and into a whole org by "Firebase Ā· GitHub". This might address @Jon_Shier's concern a little bit as well, as someone using Firebase could opt-in to allowing any binaries from the Firebase org.

This is interesting and could work well for us (although our main binary distribution mechanism isn't via GitHub but our own servers, https://dl.google.com/firebase/sdk/ios). This could easily be documented in our getting started guide but I could see it being missed as some of our other instructions are missed regularly.

One issue I foresee is communication when a new dependency is added: if FirebaseAnalytics or another SDK has a new binary dependency on another Google package that lives outside of the Firebase URL scheme, the error messages would be important and include as much detail as possible to help the developer. On the Firebase side we're extremely careful about adding dependencies but we have on occasion restructured our frameworks to be easier to build, modularize, etc and this could be a support burden when doing this internal work.

Overall we'd be fine without any additional security added since that's the current state of other package managers but would support something like URL prefix matching.

Linux Support

No comments here, not enough context and it doesn't currently apply to Firebase, if ever.

3 Likes
54

Thanks for your input!

Only products are exposed to developers, not targets, my assumption is that the introduction of binary targets won't change anything here. So you could still choose to expose multiple XCFrameworks as a single product to your users.

55

Whoops, you're totally right. Thanks for pointing that out! I'll make an edit in my original reply.

56

Apologies for the off-topic post: being able to run server side Swift on something like Google Cloud Run with support for Firebase (in other words Firebase admin natively in Swift) is among my wildest dreams, so Iā€™m keeping my fingers crossed for the ā€˜if everā€™ part. :slight_smile:
Btw I really, really love Firebase - you guys are doing some amazing work!

1 Like
57

Thanks for all the replies here. I want to respond to some points that were brought up.

  1. Support for Linux
    I think supporting Linux is very valuable on its own since it is a supported platform of Swift. Nevertheless, I agree with the feedback from multiple people here that supporting Linux is a hard obstacle and will make this proposal more complicated. From my point of view it would be totally fine if we start with Apple platform and later expand it to Linux if ever.

  2. Security
    Some feedback was that allowsBinarys is too weak and wouldn't really solve any security concern since oftentimes it will just get copy-pasted from some installation instruction. While I agree with that point I think the same can be made with source only packages. I really like the idea from @NeoNacho with the whitelist of domains and will put it into the proposal as well with some scenarios.

  3. Multiple frameworks (Firebase case)
    First, @Ryan_Wilson thanks for chiming in here. We took Firebase as an example multiple times when coming up with that proposal since you have a pretty sophisticated setup :) From my understand your use-cases would work out of the box with the proposal and you can distribute as many binaries under one package as you like.

  4. Linking of XCFrameworks
    This is the most interesting point that is still open where I would appreciate feedback/help to go forward. For the case that the package is included in >Xcode11 I guess Apple will take care of linking the framework in properly. However, for the swift executable case build via swift-build we need to come up with a linking solution. Right now I am a bit in the dark here since I haven't started implementing the linking but focused on the PackageDescription, resolution and downloading of the artifacts. If anybody has some pointers or ideas around how to link it, please feel free to let me know. (This focuses mostly around dynamic frameworks)

To move forward with this integration I think it is important we make some progress on the following points:

  • Do we want to support Linux from the beginning?
  • How should security look like? (I will expand the proposal with @NeoNacho idea)
  • How should the package description look like?

I think these are for now the most important details to clarify, afterwards we can tackle the others. Furthermore, if anybody is interested in joining this effort please reach out to us wether here or in the SwiftPM slack :)

58

@tonyarnold brought up this use concern this morning in the SwiftPM slack:

One use-case that doesnā€™t seem to be covered in the conversation to date that Iā€™d really like to see is support for linking binary dependencies on a per-configuration basis. I work on Reveal (https://revealapp.com), and would love to distribute our binary SDK via SPM in future - but we need to ensure that users donā€™t ship release builds of their apps with the Reveal SDK in them.

I understand this is a fairly unique use case, but I imagine other vendors may have need of per-configuration binary dependencies, too?

59

I think that is a very valid point but I do not know if that is something related to binary dependencies or rather its own feature. We already have conditional syntax for some parameters which was introduced by this proposal. Maybe we can elevate that conditions to the dependency declaration. This way we would enable conditional linkage for source and binary dependencies. @Aciid what do you think about that?

1 Like
60

Others in the Swift on Server community could speak up - but I'm fine with not supporting Linux in the beginning (as long as the design moves forward as if it was), as it doesn't really change the status quo for us or unlock extremely common and widely used situations.

I'm in +1 support of the suggestion from @NeoNacho, with a future direction of trying to verify binaries later once we have some more real world experience.

For the most part, the API for Artifact is fine, but I agree with others on the difficulty of explaining architectures.

The biggest change I would make is instead of .when, I'd have it be .for or .on as it more accurately indicates that the binary artifact is intended to be used with.

It could be argued from both perspectives, since .when is saying "Use this artifact when these conditions are met", but since we're specifying several artifacts and conditions constraining which environment they are intended to be used, I think for and on more accurately cover that case.