SPM Support for Binaries Distribution

Thanks for the detailed explanation! I have a few more questions if you don't mind

You don't really know what package Bar is going to resolve at for your clients

Why is that? Once the dependency graph is resolved, this question is answered.

and since SemVer means API stability but not ABI stability, it's not safe to cache build products of Bar at a particular version in that range.

Can you elaborate on this? I think it's perfectly fine to cache build product per toolchain version in case the binary is not ABI stable. In case it is, then no issue there.

In this case, we have different problems to solve. We need a manifest API to declare how binary artifacts are laid out and where to fetch them from.

Why not use the same format to describe the cache? I would not assume the cache is system global or even lives on the same system where swiftPM is running. I would have a 2 level cache system, remote & local (system global). This is particularly important to achieve what in my opinion is the most important goal, never build a binary again if it has been built somewhere in the world.

Another problem is if these packages vendor or declare dependency on another package, this effectively means SwiftPM can only resolve to a single version if the other package appear multiple times in the graph

To my understanding this is the case already even for source.

Or, you could be in a worse situation if the prebuilt binary statically links a package that you're also using in your app.

I see the problem here. Wouldn't it work to just distribute as relocatable object file without linking performed?

Sure, you could use this for build caching but it'll be much better to have a separate feature for that.

I would rather prefer to use the same building blocks and all information made explicit and accessible. Any piece of information that is not in the manifest but is private knowledge of swiftPM will effectively prevent others from building tools that supplement swiftPM.

Regarding #2 at try swift San José @ddunbar, @Braden_Scothern and I started an initial draft for binary dependencies. So far we have laid out some syntax for defining a binary target and using a binary dependencies and addressed some security concerns. The proposal still needs some work before we can pitch it but we hope to get it to a presentable level soon.

2 Likes

I don’t think that should ever become a general assumption about how packages will be used.

I understand that it is useful to speed up development‐time tasks, so it would probably be a good idea in --configuration debug.

Ultimately though, a package’s products will be executed many more times than they are built or linked. I would much rather that --configuration release always built the entire dependency tree from scratch and applied an as‐yet hypothetical whole package optimization. For release builds the compiler could optimize across module and even package boundaries. Inlining could be done all the way through to the private methods of a dependency. Generic functions could be specialized once and provided even for types declared in a sub‐sub‐sub‐client. Dependencies could be dead stripped even to the point of removing entire sub‐dependencies—even dropping dynamic librarie products that would go unused. While no work has gone into this yet (that I know of), this direction has been hinted at since the earliest days of the package manager. And all these wonderful improvements would be blocked by a design that assumes a model of pre‐built dependencies.

(Closed‐source is closed‐source. None of what I said really applies to option 2.)

2 Likes

Great ideas. I totally agree that caching should be opt-in

This is a bit of a tangent, but note that how we typically would approach this in LLVM is that the "build" of the initial package would just produce an intermediate form which still could be cached (and "never ever built again") that would cache much of the slow compilation work, while still allowing full program optimization to be done at link time. See also things like ThinLTO

1 Like

I largely echo @Aciid's reply here, but let me frame it a little differently.

First, I absolutely agree with you that #1 & #2 "can be" tackled as two sides of the same coin. However, I very intentionally believe that we should not do that.

Here are my main reasons for wanting to treat them as orthogonal:

  • I have largely been working towards a world in which #1 is automatic, safe, secure, and transparent to users. I want high performance builds with excellent caching to be something that "just works" and users don't need to think about, to the extent possible.
  • From a design space perspective, what a lot of it comes down to is whether a single package can be both source and binary. If that is allowed, it alters the design space for #2.
    • As a concrete example, the proposal @FranzBusch @Braden_Scothern and I are hashing out introduces a new target type for binary targets. That works, because it is orthogonal to existing targets. If we wanted to support both dual representations, we would have to have a different (and I believe more complex) API.

The combination of these two points means that I believe we will get the best user experience by treating these as separate.

How is this related to also being able to ship other binary files that may not be libraries or executable? Something like icons or design documents.

It is largely separate, that is covered in other threads about resources. If/when SwiftPM gets resource support this feature ideally should compose with it well, but otherwise they are separate

I think we can safely assume that targets can be either source or binary. In cases where library maintainers want to support both cases, they can always have a source target and a binary target (and associated source and binary products) and has some script build the source into the binary folder on each release.

I think it's important to support this. Imagine a package which makes binaries available for a certain platform (e.g. ARM) but wants to allow the user to build from source on other platforms (e.g. Linux x86-64).

5 Likes

Do you think my solution is sufficient?

I'm sorry, I haven't seen your solution. Is there a link to a markdown file on github?

I was only talking about this:

If you have separate source and binary targets would the person importing the package have to choose which they want to use? I can imagine us wanting the choice of source/binary dependency being hidden from the user.

But I haven't thought much about this whole space, and it seems like there are a lot of design directions worth exploring.

If you want make binaries available but also allow users to build from source, then the user needs to have the choice. In what cases don't you want to leave the choice?

I want to avoid this scenario:

  1. Developer adds a 3rd party SPM dependency "A" to their Package.swift.
  2. Package resolution occurs and pulls in A's dependency: Package "B".
  3. Package B offers both source and binary targets. A has chosen to depend on the binary target.
  4. But the developer wants to support multiple platforms - they have a problem. They don't control A so they can't switch it from the binary to source target. Ideally they would have some kind of fallback available - "build from source if you can't find a suitable binary target".
1 Like

I would like to summarize 3 scenarios that are, in my opinion, important to support:

  • Vendored binaries (no source, referred as in the other posts #2)
  • Artifact cache (local/remote/both) for packages only available as source (referred as #1 in the other posts)
  • Packages providing both source and binary

2/3 are already supported by other package managers in the Cocoa community, and with addition of other tools also caching is covered.

Did I forget anything?

This seems like a fairly rare scenario, no? Just taking a step back, but from my experience using Cocoapods, binary frameworks are quite rare, and when they do exist, the source is not available: they are specifically used in cases where the vendor wants to hide the source. So how likely are developers going to want to provide both source and binary options. What is the use case?

2 Likes

On the other hands, offering both binary and source quite common in Carthage. Consumers can chose to directly depend on a binary via binary dependency or let Carthage check for binary release on github. The second scenario can be controlled via --no-use-binaries.

Quite literally this is the most useful feature of Carthage.

6 Likes

Support for binaries installed in the system would also be nice. I'm trying to implement Swift for ROS 2 (Robot operating system) and the build system requires some thought. ROS currently supports Python and C++ as official languages, but Swift would be a really powerful alternative that provides the power of C++ and the ease of use of Python. I'd like to be able to compile different swift libraries to binaries and link other swift packages to those binaries without rebuilding dependencies every time. ROS 2 has a buildfarm that builds binaries for apt, brew, and chocolatey, and it would be great to be able to install swift libraries with a system package manager and use them in projects. I'd also like to be able to link against compiled swift libraries in a local workspace. While some might consider the first case outside the scope of the swift package manager, I think this kind of versatility is important if swift is to gain better adoption, both for ROS and for many other projects.

2 Likes