Are there any plans to allow binary libraries to be packaged and distributed through SPM?
I have seen this asked about a few times in the past and the answer was "no current plans, source based libraries only" but the most recent answer I could find was ~2 years ago. Is this still the case?
Reason for asking:
I work on Firebase iOS and we have a large number of frameworks with a mix of open source SDKs on GitHub and closed source SDKs that we package as binaries. We ship them via CocoaPods, Carthage, and as a zip file for those who don't want to use a package manager.
We've had requests already, but I anticipate a large influx of requests once there are deeper ties to iOS and Xcode with SPM projects. At the moment it doesn't look like there's a way we'd be able to ship the closed source frameworks to customers.
I am afraid the answer is still the same. We need a proposal for supporting binary packages. One of the major concern with binary packages is how do we ensure security since the dependency consumer can't really inspect the source code.
Even if the framework is open source, shipping a pre-compiled binary could greatly improve compilation speed. This could be great for Vapor since long compilation times are an issue at the moment.
Would build caching allow for debug builds of the SPM package to use release builds of the framework in question? One issue we have currently is not being able to provide pure-Swift implementations for things like BCrypt since the debug-mode builds are unusably slow.
When we get mixedâconfiguration package graphs, it would be nice to have an additional assertion level:
assert: If this fails, my own package made a mistake. Optimize it out whenever building for a client package.
(doesnât exist yet): If this fails, client code made a mistake. Perform the check when built for a debugâconfigured client to help them find the mistake faster, but optimize it out when built for a releaseâconfigured client.
precondition: Perform the check even if everything is built for release. Optimize it only if the client waives it with -Ounchecked.
It probably means a third configuration: DEBUG/DEPENDENCY_OF_DEBUG/RELEASE.
Maybe precondition does this job well in the context of an OS library, but it is unnecessarily heavy in a package setting.
Right now, while the package manager builds the entire graph in the same configuration, assert fills this niche nicely, firing when clients debug and disappearing when they release. If the package manager starts building dependencies in release mode even when clients debug (which is reasonable), assert will no longer work for that, moving instead to role comparable to the standard libraryâs _internalInvariant. There will be nothing to fill the resulting functionality gap.
But we are going way off topic. Thatâs my fault, so I apologize.
Officially, the difference between precondition and fatalError is only that fatalError survives -Ounchecked. (An additional hidden difference in practice is that precondition loses its accompanying message in release builds, but it still performs the check and stops execution.)
No worries! IMO, it isn't entirely off-topic. The intended purpose for precondition is to perform safety checks for preconditions when getting called from code you don't control. For a resilient binary, such as one that ships with an OS distribution or as binary middleware, "code you don't control" is pretty much all of your clients; in the more typical swiftpm source package case, although a package author doesn't control their clients' code, the clients do ultimately control all of the input code that goes into their binary, and in principle they could audit their own code and their source dependencies to the point that turning off precondition checks is an acceptable cost/benefit tradeoff. It could certainly be worth discussing whether precondition ought to have more fine-grained behavior control in the source package case as compared to the resilient binary case.
Could you expand on this? The problem canât be closed source in itself, since most of Appleâs frameworks like UIKit and AppKit are closed source, and Foundation on Darwin is closed source except for the overlays.
I imagine one concern is that a package manager downloads content from arbitrary internet addresses. If a repository or the connection to it is compromised in some way, building from source at least gives you the opportunity to inspect those sources before they're compiled. If you download binaries and link directly to those, there's not really a way to inspect the library.
There are of course ways to improve verification (like having dependency declarations include a SHA), and source-based dependencies aren't impossible to compromise either (e.g., Node's event-stream vulnerability from last year), but supporting binary dependencies needs a bit of extra care that isn't just as straightforward as "download this and link it."
Right verification is one aspect. Another concern is how do I trust my package dependencies without inspecting the source code. Someone could publish a binary package that has malicious code. Maybe I wouldn't be adding a dependency unless I trust it but what if the package is coming somewhere deep in my package graph. One solution could be limiting binary packages to top-level packages.
Any security that relies on users taking some action is doomed to fail, so the inspectability seems rather irrelevant here. Allowing SPM to download and link binary dependencies puts it on the same footing as every other community package manager, so unless there's something else going on here, I'm not sure what the problem is.
Of course, that's not to say there couldn't be improvements. A package index that allows authenticated publishing of dependencies and tracks their sources and hashes which SPM could then verify would be a great value for the community. It would prevent surreptitious altering of package content (which is probably the least popular method of package compromise, as it's one of the most difficult), but would still do little to end the ability of malicious actors to engineer their way into a libraries's account and publish a compromised version. A 2FA solution for publishing may help there but ultimately there's nothing that can totally prevent these sorts of issues.
I do suspect that one the reasons there is no SPM index from Apple is Apple legal's fear of liability in such cases. If that's the case, I wish Apple would just come out and say it and let the community get on with producing a real index, rather than fearing any attempt at an index could be sherlocked by Apple at any time. Perhaps this is where a separate foundation could help, to shield Apple and the rest of the community from liability in these matters?
Few people inspect the source code of every dependency they use; people just don't have that kind of time or expertise. The community will have these needs met whether SPM does it or not, so if there's interest in support such a feature at all, getting started now and adding security over time seems better than delaying the feature and forcing everyone to use different tools that don't offer such security anyway.
+1. I don't see anything to be concerned about as far as security goes. It seems to be predicated on the idea that source code is easier to manually inspect than binary code, which (aside from the fact that almost nobody does this), is only true for the most novice, obvious forms of malware. Just because you can see the source doesn't mean you can fully understand it in any reasonable amount of time, or why it's written that way.
Authenticating packages is interesting, but as you say it's more of an index feature and not really anything to do with source/binary packages.
Also, I don't see any principled reason to limit binary dependencies to top-level packages.
I find that I use 3rd-party dependencies precisely when I need an implementation that I don't have the time or knowledge to build myself. I would like to echo the sentiment that I am not going to audit such code for malicious behavior when I am unlikely to understand the non-malicious behavior in a reasonable amount of time.
(Frameworks for connecting to services are different, of course, but are still difficult for someone without knowledge of the service's API to trivially audit.)