SPM Support for Binaries Distribution

Development Package Manager
1

Hello!

It's very exciting to finally see SPM being integrated in Xcode 11. I've seen mentioned multiple times over the last few days that currently SPM packages only support source code and unit tests. Support for assets and binaries need to be explored by the community. Taking inspiration from some ongoing Twitter discussions (here and here), I've decided to start this forum thread in order to gather some ideas on what it would take to support binaries distribution in SPM.

Adding binaries support to SPM would definitely increase adoption from bigger projects that may contain many dependencies and could be affected longer build times for example.

What do you think? What are the dependencies and restrictions of adding binaries distribution to SPM?

22 Likes
2

There were a few of us at try! Swift on Friday that started talking through a proposal and its behaviors. We will hopefully have a pitch for this very soon.

(We are currently getting it up on Github so we can work on it now that we aren't in person)

3 Likes
3

There are two orthogonal areas that we need to tackle for "binary support" in SwiftPM:

  1. Local/Remote build caching for your package dependencies
  2. Distributing closed-source prebuilt binaries as Swift packages

I think @Braden_Scothern is talking about 2 where as @BalestraPatrick is talking about 1?

1 Like
4

Ya we started #2 and Iā€™ve had some thoughts about #1 but am not actively working on it.

Thanks for helping clarify @Aciid

1 Like
5

I'm fairly sure @BalestraPatrick is talking about #2 also. @Braden_Scothern do you have the github pitch available to look at?

6

Hi all,

Could someone take the time to explain to me the difference between #1 and #2 and how they are orthogonal?

7

Hi,

I might not give you the best answer here, but imho:
#1 talks about building and caching the open source package, similar to what Carthage does, in order to gain productivity by not rebuilding the packages after each project cleanup.
#2 talks about providing support for closed source binaries in SwiftPM - this is essential for the integrators, same as for vendors, since supporting the binaries will cut out the need of having 3rd party dependency manager - eg Carthage or Cocoapods.

2 Likes
8

I have been maintaining Carthage and a distributed cache manager for Carthage (GitHub - tmspzz/Rome: Carthage cache for S3, Minio, Ceph, Google Storage, Artifactory and many others) and I don't see any difference between the two cases.

Whether a binary is vendored or produced from source, it shouldn't make any difference. The goal is to never ever build again the same stuff over and over and over again. For example, once Alamofire is built by the CI running on Alamofire's repo, that's pretty much it. No one in the world should have to rebuild that commit ever again.

Ideally whether it's a fresh clone of a project (think CI) or a simple re-run, what is actually built should be the minimum possible subset.

At this level of granularity I therefore don't see a difference between vendored binaries and others. If a binary is available, grab it and be done with it. If not, build form source. If no source, fail.

7 Likes
9

I agree with the above,

yet the main distinction here is that Swift Packages != xcframeworks thus require different handling/distribution/caching strategy. There are also some constraints, eg xcframework cant depend on swift package, etc.
Im happy for SwiftPM engineers and the guys from trySwift! to bring clarity to the two points mentioned earlier.

Best,
Boris.

10

As @tmpz explained, I am talking about both actually. I don't see a big technical difference between the two. Sure, open-source package binaries would need to be hosted somewhere in the open internet (GitHub, Artifactory, you name it), but we should also support closed-source binaries that could be stored on any arbitrary storage. This would allow to build a package only once from source and store it forever as a binary (with related artifacts and metadata if needed) under the form of XCFramework for example.

1 Like
11

The difference is that #2 can be done first, before #1 and that without #2, users who depend on dependencies solely vendored as pre-built binaries can't replace Cocoapods/Carthage with SwiftPM.

3 Likes
12

Thanks for the explanation. I would consider #2 the base for #1 then, definitely not orthogonal problems imho.

13

These might look the same problem on the surface but they come with different sets of problems. Let me try to give some context:

  1. Local/Remote build caching for your package dependencies

This basically means sources of dependencies is available to clients but they don't want to rebuild them after performing dependency resolution. One solution could be: SwiftPM builds and store the build products of the dependencies in some shared location and it shares the build products across all projects (when possible) that use this dependency on the system. SwiftPM would be responsible for things like building with the right client-specific specialization and cache eviction. This is mostly doesn't even require manifest API additions.

Alamofire is a bit of a trivial example as it doesn't have further dependencies. Imagine a package Foo that depends on package Bar from 1.0.0 and going up to 2.0.0 (.upToNextMajor(from: "1.0.0")). You don't really know what package Bar is going to resolve at for your clients and since SemVer means API stability but not ABI stability, it's not safe to cache build products of Bar at a particular version in that range. This is a good thing. Package authors who provide source shouldn't really need to concerns themselves with ABI stability. The package manager can handle all these things for you and cache the build products accordingly.

  1. Distributing closed-source prebuilt binaries as Swift packages

In this case, we have different problems to solve. We need a manifest API to declare how binary artifacts are laid out and where to fetch them from. The build system can't do much at the client side. The package author is (mostly) responsible for making sure that the right artifacts are provided. Another problem is if these packages vendor or declare dependency on another package, this effectively means SwiftPM can only resolve to a single version if the other package appear multiple times in the graph (because of the same ABI issue). Or, you could be in a worse situation if the prebuilt binary statically links a package that you're also using in your app. Sure, you could use this for build caching but it'll be much better to have a separate feature for that. There is also a security aspect since it's difficult to inspect binaries than sources.

4 Likes
14

I just got the github link and will post it later today (Iā€™m not the owner of it on github).

15

I am hoping to have time to work on #1 after finishing #2. It will depend on other proposals that I am also starting since try! swift that I want to share soon as well.

16

Thanks for the detailed explanation! I have a few more questions if you don't mind

You don't really know what package Bar is going to resolve at for your clients

Why is that? Once the dependency graph is resolved, this question is answered.

and since SemVer means API stability but not ABI stability, it's not safe to cache build products of Bar at a particular version in that range.

Can you elaborate on this? I think it's perfectly fine to cache build product per toolchain version in case the binary is not ABI stable. In case it is, then no issue there.

In this case, we have different problems to solve. We need a manifest API to declare how binary artifacts are laid out and where to fetch them from.

Why not use the same format to describe the cache? I would not assume the cache is system global or even lives on the same system where swiftPM is running. I would have a 2 level cache system, remote & local (system global). This is particularly important to achieve what in my opinion is the most important goal, never build a binary again if it has been built somewhere in the world.

Another problem is if these packages vendor or declare dependency on another package, this effectively means SwiftPM can only resolve to a single version if the other package appear multiple times in the graph

To my understanding this is the case already even for source.

Or, you could be in a worse situation if the prebuilt binary statically links a package that you're also using in your app.

I see the problem here. Wouldn't it work to just distribute as relocatable object file without linking performed?

Sure, you could use this for build caching but it'll be much better to have a separate feature for that.

I would rather prefer to use the same building blocks and all information made explicit and accessible. Any piece of information that is not in the manifest but is private knowledge of swiftPM will effectively prevent others from building tools that supplement swiftPM.

17

Regarding #2 at try swift San JosƩ @ddunbar, @Braden_Scothern and I started an initial draft for binary dependencies. So far we have laid out some syntax for defining a binary target and using a binary dependencies and addressed some security concerns. The proposal still needs some work before we can pitch it but we hope to get it to a presentable level soon.

2 Likes
18

I donā€™t think that should ever become a general assumption about how packages will be used.

I understand that it is useful to speed up developmentā€time tasks, so it would probably be a good idea in --configuration debug.

Ultimately though, a packageā€™s products will be executed many more times than they are built or linked. I would much rather that --configuration release always built the entire dependency tree from scratch and applied an asā€yet hypothetical whole package optimization. For release builds the compiler could optimize across module and even package boundaries. Inlining could be done all the way through to the private methods of a dependency. Generic functions could be specialized once and provided even for types declared in a subā€subā€subā€client. Dependencies could be dead stripped even to the point of removing entire subā€dependenciesā€”even dropping dynamic librarie products that would go unused. While no work has gone into this yet (that I know of), this direction has been hinted at since the earliest days of the package manager. And all these wonderful improvements would be blocked by a design that assumes a model of preā€built dependencies.

(Closedā€source is closedā€source. None of what I said really applies to option 2.)

3 Likes
19

Great ideas. I totally agree that caching should be opt-in

20

This is a bit of a tangent, but note that how we typically would approach this in LLVM is that the "build" of the initial package would just produce an intermediate form which still could be cached (and "never ever built again") that would cache much of the slow compilation work, while still allowing full program optimization to be done at link time. See also things like ThinLTO

1 Like