SPM Support for Binaries Distribution

I was only talking about this:

If you have separate source and binary targets would the person importing the package have to choose which they want to use? I can imagine us wanting the choice of source/binary dependency being hidden from the user.

But I haven't thought much about this whole space, and it seems like there are a lot of design directions worth exploring.

If you want make binaries available but also allow users to build from source, then the user needs to have the choice. In what cases don't you want to leave the choice?

I want to avoid this scenario:

  1. Developer adds a 3rd party SPM dependency "A" to their Package.swift.
  2. Package resolution occurs and pulls in A's dependency: Package "B".
  3. Package B offers both source and binary targets. A has chosen to depend on the binary target.
  4. But the developer wants to support multiple platforms - they have a problem. They don't control A so they can't switch it from the binary to source target. Ideally they would have some kind of fallback available - "build from source if you can't find a suitable binary target".
1 Like

I would like to summarize 3 scenarios that are, in my opinion, important to support:

  • Vendored binaries (no source, referred as in the other posts #2)
  • Artifact cache (local/remote/both) for packages only available as source (referred as #1 in the other posts)
  • Packages providing both source and binary

2/3 are already supported by other package managers in the Cocoa community, and with addition of other tools also caching is covered.

Did I forget anything?

This seems like a fairly rare scenario, no? Just taking a step back, but from my experience using Cocoapods, binary frameworks are quite rare, and when they do exist, the source is not available: they are specifically used in cases where the vendor wants to hide the source. So how likely are developers going to want to provide both source and binary options. What is the use case?

2 Likes

On the other hands, offering both binary and source quite common in Carthage. Consumers can chose to directly depend on a binary via binary dependency or let Carthage check for binary release on github. The second scenario can be controlled via --no-use-binaries.

Quite literally this is the most useful feature of Carthage.

7 Likes

Support for binaries installed in the system would also be nice. I'm trying to implement Swift for ROS 2 (Robot operating system) and the build system requires some thought. ROS currently supports Python and C++ as official languages, but Swift would be a really powerful alternative that provides the power of C++ and the ease of use of Python. I'd like to be able to compile different swift libraries to binaries and link other swift packages to those binaries without rebuilding dependencies every time. ROS 2 has a buildfarm that builds binaries for apt, brew, and chocolatey, and it would be great to be able to install swift libraries with a system package manager and use them in projects. I'd also like to be able to link against compiled swift libraries in a local workspace. While some might consider the first case outside the scope of the swift package manager, I think this kind of versatility is important if swift is to gain better adoption, both for ROS and for many other projects.

2 Likes

Hi all,

once we gain support for distributing the binaries, would it be possible to also define the sub-dependencies of that binary(relation binary to binary)?

Thanks.

1 Like

Just wanted to check, has anyone posted a proposal to GitHub yet? At Firebase we have a mix of open source and closed source SDKs and are very interested in helping out where we can. :slightly_smiling_face:

9 Likes

I think it's important to support this. Imagine a package which makes binaries available for a certain platform (e.g. ARM) but wants to allow the user to build from source on other platforms (e.g. Linux x86-64).

Can you elaborate on why you feel this is an important use case?

On the other hands, offering both binary and source quite common in Carthage. Consumers can chose to directly depend on a binary via binary dependency or let Carthage check for binary release on github. The second scenario can be controlled via --no-use-binaries .

Quite literally this is the most useful feature of Carthage.

My impression is that this is primarily used in order for the workflow implications of using a pre-built binary, however. I feel very strongly that all of the workflow benefits of having binary packages (build time reduction, cognitive reduction in clutter, etc.) should be built in to the package manager universally in a way that is secure and flexible.

To put it another way, the way I see it is that binary packaging is often used as a "workaround" for missing features. I think that implementing those missing features directly will ultimately provide a smoother, safer, and more consistent developer experience, rather than allowing it to be shoe-horned into binary package distribution (which has other downsides).

My proposed approach is more complicated though, and will take more effort to achieve, I very freely admit that.

I'll try to get more of my thoughts here codified into a concrete document, to hopefully make things more clear. I really appreciate the discussion though, it helps clarify what needs to be spelled out clearly.

1 Like

To put it another way, the way I see it is that binary packaging is often used as a "workaround" for missing features. I think that implementing those missing features directly will ultimately provide a smoother, safer, and more consistent developer experience, rather than allowing it to be shoe-horned into binary package distribution (which has other downsides).

Agreed from a consumer perspective. However, there's still a need to support binary distribution to package manager users from other build systems and for libraries that are not able to make their source public.

1 Like

Indeed, that is the focus of the proposal a couple of us are working on and will hopefully have posted before too long.

1 Like

I think you are mixing up several features in Carthage. The binary feature in Carthage is for binary, vendored frameworks only. The GitHub pre-built release functionality (that can be turned off with —no-use-binaries) is separate and depends on source being available. There really isn’t a decent way to provide both at the same time.

Essentially, I would say that Carthage handles this very similarly to what @ddunbar and @Aciid have proposed.

1 Like

Echoing what I think others are saying, would love to see a solution where if we have a binary that matches the SemVersion either remote or local, and it is built then we could use that, otherwise if we have the source it would see that it does not have the right version prebuilt and build that version. If we have it cached we don't pay the cost of building it. If there was no SemVersion associated, like a branch, or in edit mode, then it would require building every time but once it is locked down under a SemVersion then it can be thought of as a version of the code that can be cached. This would be a big time saver only building the code under active revision.

My motivation is to make sure Swift packages are as cross-platform as possible.

Today, a problem many Swift packages have is they are Darwin only - for example because they use some part of Foundation that is not yet implemented on Linux.

If we add support for binary targets I think it is quite likely that many packages will only publish binaries for Darwin. Then, if another package depends on that binary target it becomes a Darwin-only package. It would only take one binary Darwin target in a dependency tree to make the whole tree Darwin-only.

I would much prefer a model where binary targets are used if they are available and compatible, but otherwise packages continue to build from source automatically.

I think this has the best chance of growing the cross-platform package ecosystem.

9 Likes

I just wanted to reiterate the desire for supporting this case - someone mentioned they thought binary only frameworks were rare, but if you do any enterprise programming they are not rare at all - from special printer / device libraries, to things like Adobe Analytics there are a lot of libraries that I would love to use Swift Package Manager to load...

I am working for a company now that is also distributing a binary only framework because of some proprietary things we are doing inside - The framework we build now works well for distribution, but would love to see that in SPM as I really see that as the future of integration for third party libraries, especially with the Xcode 11 integration of SPM to easily select libraries to include in a project.

Personally I think this support is so helpful I'd love to see even something crude to start with (can't be any cruder than Cartahge/Cocoapods support) that evolved as time went on, just to be able to get this working for third party binary projects today. Maybe to start with it would only support frameworks and not more complex binary distributions. I totally understand the concerns about distributing binary only frameworks in a wider cross platform world, but perhaps something like a clear definition of what binaries would be included in what was being distributed would at least make it clear who could make use of a framework and who could not based on targets they chose to build for.

I look forward to seeing and commenting on future specs that people have been working on...

13 Likes

I’m the maintainer of a fork of Carthage (nsoperations/carthage) which independently fixed a lot of bugs and added a lot of binary caching features. Just wanted to give some insights into the experiences with this from my point of view.
I work at a company where we have more than 50 internal frameworks. Rebuilding everything every time is simply not an option. However, relying on binaries, if they don’t use the module evolution feature, is not a breeze with the current state of Swift. The problem is that a change in any of the versions of the transitive dependencies have to invalidate the cache for that dependency because the Swift compiler can link to private symbols. So even if a resolved version of a dependency is the same and the swift version is exactly the same it is not guaranteed that a cached version can be used, unless the complete set of transitive dependencies is also exactly the same as was used at build time for the cached dependency. We learned this the hard way with crashes at run time because private symbols of some transitive dependencies could not be found.
The work around we currently implemented in our Carthage fork is that we calculate a hash of the complete set of transitive dependencies and this hash needs to match for the cache to be valid. So basically we store a version in the cache for every permutation of transitive dependencies.

I want to add support for marking a dependency as stable (i.e. built with module evolution enabled) such that we can bypass the above for that dependency. We cannot simply turn it on for our whole code base, because some features of module evolution break backwards compatibility in particular for @objc annotated code.

By the way the original Carthage simply ignores the above and is therefore completely broken for binary caching.

13 Likes

@Braden_Scothern, @ddunbar, and @FranzBusch have had their proposal accepted for Package Manager Binary Dependencies. It is currently implemented in Swift 5.3. Here is the original proposal, if anyone is curious, SE-0272

1 Like
Terms of Service

Privacy Policy

Cookie Policy