SPM Support for Binaries Distribution

Great ideas. I totally agree that caching should be opt-in

This is a bit of a tangent, but note that how we typically would approach this in LLVM is that the "build" of the initial package would just produce an intermediate form which still could be cached (and "never ever built again") that would cache much of the slow compilation work, while still allowing full program optimization to be done at link time. See also things like ThinLTO

1 Like

I largely echo @Aciid's reply here, but let me frame it a little differently.

First, I absolutely agree with you that #1 & #2 "can be" tackled as two sides of the same coin. However, I very intentionally believe that we should not do that.

Here are my main reasons for wanting to treat them as orthogonal:

  • I have largely been working towards a world in which #1 is automatic, safe, secure, and transparent to users. I want high performance builds with excellent caching to be something that "just works" and users don't need to think about, to the extent possible.
  • From a design space perspective, what a lot of it comes down to is whether a single package can be both source and binary. If that is allowed, it alters the design space for #2.
    • As a concrete example, the proposal @FranzBusch @Braden_Scothern and I are hashing out introduces a new target type for binary targets. That works, because it is orthogonal to existing targets. If we wanted to support both dual representations, we would have to have a different (and I believe more complex) API.

The combination of these two points means that I believe we will get the best user experience by treating these as separate.

How is this related to also being able to ship other binary files that may not be libraries or executable? Something like icons or design documents.

It is largely separate, that is covered in other threads about resources. If/when SwiftPM gets resource support this feature ideally should compose with it well, but otherwise they are separate

I think we can safely assume that targets can be either source or binary. In cases where library maintainers want to support both cases, they can always have a source target and a binary target (and associated source and binary products) and has some script build the source into the binary folder on each release.

I think it's important to support this. Imagine a package which makes binaries available for a certain platform (e.g. ARM) but wants to allow the user to build from source on other platforms (e.g. Linux x86-64).

4 Likes

Do you think my solution is sufficient?

I'm sorry, I haven't seen your solution. Is there a link to a markdown file on github?

I was only talking about this:

If you have separate source and binary targets would the person importing the package have to choose which they want to use? I can imagine us wanting the choice of source/binary dependency being hidden from the user.

But I haven't thought much about this whole space, and it seems like there are a lot of design directions worth exploring.

If you want make binaries available but also allow users to build from source, then the user needs to have the choice. In what cases don't you want to leave the choice?

I want to avoid this scenario:

  1. Developer adds a 3rd party SPM dependency "A" to their Package.swift.
  2. Package resolution occurs and pulls in A's dependency: Package "B".
  3. Package B offers both source and binary targets. A has chosen to depend on the binary target.
  4. But the developer wants to support multiple platforms - they have a problem. They don't control A so they can't switch it from the binary to source target. Ideally they would have some kind of fallback available - "build from source if you can't find a suitable binary target".
1 Like

I would like to summarize 3 scenarios that are, in my opinion, important to support:

  • Vendored binaries (no source, referred as in the other posts #2)
  • Artifact cache (local/remote/both) for packages only available as source (referred as #1 in the other posts)
  • Packages providing both source and binary

2/3 are already supported by other package managers in the Cocoa community, and with addition of other tools also caching is covered.

Did I forget anything?

This seems like a fairly rare scenario, no? Just taking a step back, but from my experience using Cocoapods, binary frameworks are quite rare, and when they do exist, the source is not available: they are specifically used in cases where the vendor wants to hide the source. So how likely are developers going to want to provide both source and binary options. What is the use case?

1 Like

On the other hands, offering both binary and source quite common in Carthage. Consumers can chose to directly depend on a binary via binary dependency or let Carthage check for binary release on github. The second scenario can be controlled via --no-use-binaries.

Quite literally this is the most useful feature of Carthage.

5 Likes

Support for binaries installed in the system would also be nice. I'm trying to implement Swift for ROS 2 (Robot operating system) and the build system requires some thought. ROS currently supports Python and C++ as official languages, but Swift would be a really powerful alternative that provides the power of C++ and the ease of use of Python. I'd like to be able to compile different swift libraries to binaries and link other swift packages to those binaries without rebuilding dependencies every time. ROS 2 has a buildfarm that builds binaries for apt, brew, and chocolatey, and it would be great to be able to install swift libraries with a system package manager and use them in projects. I'd also like to be able to link against compiled swift libraries in a local workspace. While some might consider the first case outside the scope of the swift package manager, I think this kind of versatility is important if swift is to gain better adoption, both for ROS and for many other projects.

2 Likes

Hi all,

once we gain support for distributing the binaries, would it be possible to also define the sub-dependencies of that binary(relation binary to binary)?

Thanks.

1 Like

Just wanted to check, has anyone posted a proposal to GitHub yet? At Firebase we have a mix of open source and closed source SDKs and are very interested in helping out where we can. :slightly_smiling_face:

6 Likes

I think it's important to support this. Imagine a package which makes binaries available for a certain platform (e.g. ARM) but wants to allow the user to build from source on other platforms (e.g. Linux x86-64).

Can you elaborate on why you feel this is an important use case?

Terms of Service

Privacy Policy

Cookie Policy