SPM Support for Binaries Distribution

To put it another way, the way I see it is that binary packaging is often used as a "workaround" for missing features. I think that implementing those missing features directly will ultimately provide a smoother, safer, and more consistent developer experience, rather than allowing it to be shoe-horned into binary package distribution (which has other downsides).

Agreed from a consumer perspective. However, there's still a need to support binary distribution to package manager users from other build systems and for libraries that are not able to make their source public.

1 Like

Indeed, that is the focus of the proposal a couple of us are working on and will hopefully have posted before too long.

1 Like

I think you are mixing up several features in Carthage. The binary feature in Carthage is for binary, vendored frameworks only. The GitHub pre-built release functionality (that can be turned off with —no-use-binaries) is separate and depends on source being available. There really isn’t a decent way to provide both at the same time.

Essentially, I would say that Carthage handles this very similarly to what @ddunbar and @Aciid have proposed.

1 Like

Echoing what I think others are saying, would love to see a solution where if we have a binary that matches the SemVersion either remote or local, and it is built then we could use that, otherwise if we have the source it would see that it does not have the right version prebuilt and build that version. If we have it cached we don't pay the cost of building it. If there was no SemVersion associated, like a branch, or in edit mode, then it would require building every time but once it is locked down under a SemVersion then it can be thought of as a version of the code that can be cached. This would be a big time saver only building the code under active revision.

My motivation is to make sure Swift packages are as cross-platform as possible.

Today, a problem many Swift packages have is they are Darwin only - for example because they use some part of Foundation that is not yet implemented on Linux.

If we add support for binary targets I think it is quite likely that many packages will only publish binaries for Darwin. Then, if another package depends on that binary target it becomes a Darwin-only package. It would only take one binary Darwin target in a dependency tree to make the whole tree Darwin-only.

I would much prefer a model where binary targets are used if they are available and compatible, but otherwise packages continue to build from source automatically.

I think this has the best chance of growing the cross-platform package ecosystem.

8 Likes

I just wanted to reiterate the desire for supporting this case - someone mentioned they thought binary only frameworks were rare, but if you do any enterprise programming they are not rare at all - from special printer / device libraries, to things like Adobe Analytics there are a lot of libraries that I would love to use Swift Package Manager to load...

I am working for a company now that is also distributing a binary only framework because of some proprietary things we are doing inside - The framework we build now works well for distribution, but would love to see that in SPM as I really see that as the future of integration for third party libraries, especially with the Xcode 11 integration of SPM to easily select libraries to include in a project.

Personally I think this support is so helpful I'd love to see even something crude to start with (can't be any cruder than Cartahge/Cocoapods support) that evolved as time went on, just to be able to get this working for third party binary projects today. Maybe to start with it would only support frameworks and not more complex binary distributions. I totally understand the concerns about distributing binary only frameworks in a wider cross platform world, but perhaps something like a clear definition of what binaries would be included in what was being distributed would at least make it clear who could make use of a framework and who could not based on targets they chose to build for.

I look forward to seeing and commenting on future specs that people have been working on...

12 Likes

I’m the maintainer of a fork of Carthage (nsoperations/carthage) which independently fixed a lot of bugs and added a lot of binary caching features. Just wanted to give some insights into the experiences with this from my point of view.
I work at a company where we have more than 50 internal frameworks. Rebuilding everything every time is simply not an option. However, relying on binaries, if they don’t use the module evolution feature, is not a breeze with the current state of Swift. The problem is that a change in any of the versions of the transitive dependencies have to invalidate the cache for that dependency because the Swift compiler can link to private symbols. So even if a resolved version of a dependency is the same and the swift version is exactly the same it is not guaranteed that a cached version can be used, unless the complete set of transitive dependencies is also exactly the same as was used at build time for the cached dependency. We learned this the hard way with crashes at run time because private symbols of some transitive dependencies could not be found.
The work around we currently implemented in our Carthage fork is that we calculate a hash of the complete set of transitive dependencies and this hash needs to match for the cache to be valid. So basically we store a version in the cache for every permutation of transitive dependencies.

I want to add support for marking a dependency as stable (i.e. built with module evolution enabled) such that we can bypass the above for that dependency. We cannot simply turn it on for our whole code base, because some features of module evolution break backwards compatibility in particular for @objc annotated code.

By the way the original Carthage simply ignores the above and is therefore completely broken for binary caching.

13 Likes

@Braden_Scothern, @ddunbar, and @FranzBusch have had their proposal accepted for Package Manager Binary Dependencies. It is currently implemented in Swift 5.3. Here is the original proposal, if anyone is curious, SE-0272

3 Likes

I am looking for a way to use dynamic libraries in Swift packages that are compiled from the same code for the platforms macOS/Intel, macOS/ARM, Red Hat 7/x86, and Windows/x86. The library is compiled from C++ code, but this is done via cmake in a complex manner (e.g. some C++ files created by cmake), so it cannot by compiled inside a Swift package. I have a binding for this library to Swift that is fully functional in an Xcode project.

So the library is built for specific platforms and has nothing to do with Swift up to that point. Why shouldn't it be possible for this library to be used within a Swift project on such a platform? From https://github.com/apple/swift-evolution/blob/master/proposals/0272-swiftpm-binary-dependencies.md:

Non-Apple platforms provide non-trivial challenges since they are not always giving guarantees of the ABI of the platform

I think this does only apply to binaries built from Swift code, or am I missing something? So it should not be a problem for binaries built from C++ code? The platforms above should be specific enough. So why shouldn't there be an easy possibility to integrate such a library in a Swift package? Is there any "trick" currently to make this possible?

Thanks in advance for any answer.

MLIRSwift does this for MLIR (a C++ library with a complex CMake build flow). The "trick" I use is to build the library and install a pkg-config file to /usr/local/lib/pkgconfig, and then import it as a system library (Package.swift and Tools/build-dependencies are interesting files to look at). It does seem like SPM is moving in the direction of supporting this is a more straightforward manner, but I am unaware of a better way to accomplish this at this time.

1 Like

Thanks for the answer. This could solve it for macOS, and on Linux, the library could also be installed I think, but on Windows I am out of luck then, as putting own libraries to C:\Windows is rightfully considered as a very bad idea [UPDATE: see here how it should be done on Windows via regsvr32, but still unwanted in most cases]... And even on macOS installing a library does not sound that good... And as I said, it should not be so difficult in the described case. Are there any current plans or even a timeframe for a possible realization? I could not find anything about it.

SE-0305 lays the foundation for that, and binary library support is listed in future directions.

I’m waiting for the support too. This would allow more thing to be done with SwiftPM alone on Windows.

1 Like

This is still a problem for binaries built from C++ code. The problem you have is that SwiftPM manages binary artifacts by using the target triple. This is shared across all Linux distributions, and does not uniquely identify them. This means whatever you build must work across all Linux distributions, or there’ll be frustrating and mysterious linker errors.

This is a bugbear of mine, and I absolutely want to get SwiftPM to solve it. Mostly it’s a tooling problem: it’s entirely possible to build ABI-stable C/C++ libraries that support all major Linux distributions, but if you don’t know what you’re doing it’s very hard to do right. I’d like us to build some tooling that makes it much easier to do this correctly.

3 Likes

I already read about that in the discussions. My answer is: I do have binaries that are specific enough (see above: I do not have a binary for "Linux", but more precisely for "Red Hat 7/x86"). Then we just need a possibility to be enough specific in a Swift package, i.e. to be able to differentiate down to that level which binary to choose. I might be a problem with some inner workings of the current SPM (sorry, I am not an expert here), but then this is something that "should" be changed. As I said, the idea is "simple enough" (sorry, I know it might still be a lot of work).

...A common use case seems to be a dynamic library built via cmake. Of course, it would be great if cmake could be called from the Swift package to produce the binary. In many cases, this would be an even better solution than having to provide binaries for different platforms.

Note that the projects building those libraries are very often projects by other authors, in practice you have what you get (and which is usually OK). You do not want to have to change the build processes for those libraries.

Yes, I understand what you have. I'm explaining the problem with that, which is that your package now supports only Red Hat 7: it supports no other Linux distributions. That's fine, there's nothing inherently wrong with that, but SwiftPM does not have any awareness of Linux distributions and so it can't provide your users with helpful warnings. It is not capable of saying "This package is not going to work because you're running RHEL 8, or Ubuntu 20.04, or whatever".

Additionally, if we were to add such support to SwiftPM, we would be greatly increasing the workload required to on-board new Linux distributions to Swift, as SwiftPM would be required to know about all of them in order to disambiguate between them. Additionally, package authors would need to re-evaluate what they need to build for each time Swift is updated, as Linux distributions may have been added or removed.

So what I'm saying is that SwiftPM should not support that use-case, but instead should target the wider use-case of building a binary that will work on all Linux distributions built on a given libc. This is entirely do-able: Python calls the idea manylinux and it is working successfully in that community.

1 Like

OK, this should be the better way, but as long as this is difficult and/or too far away from realization, other – less good – options should maybe [UPDATE: also] be considered? (Also see my above note about the libraries stemming from other authors.)

Yes, this is unperfect, but emitting a warning for such packages could be fine, you are actually never sure if a new version of e.g. a package breaks something, so the world isn't perfect anyway... [UPDATE: Sorry about the sarcasm here, I completely understand your point of view, this was inappropriate.]

This could maybe be a compromise?

[UPDATE: This would be in-line with the usual Swift way: Make it safe by default, but let people do dangerous things if they really want to, but make those dangerous things explicit, in this case also inform the user of such a package about the dangerous thing, that the package might not work anymore with another distribution of the operating system or on another operating system, maybe even giving a list of platforms currently supported by the package. This already should be the case for Swift packages using the XCFramework, which are usable only on Apple platforms.]

...So we actually have two topics here that are a actually a little different:

  1. Support for distribution of Swift packages that could be distributed in source code form, but where the authors decide to distribute them in binary form.
  2. Support for distribution of binaries stemming from other sources (not Swift), that cannot be easily compiled within a Swift package.

If you consider 1., I think @lukasa has a totally valid point here (but I do not know enough about it technically). Concerning 2., I think it would be helpful to let people do more "dangerous" things (with warnings also for the users of a package!), e.g. possibly not supporting a wide range of Linux distributions.