Until we have an ABI and library evolution story that is forward-compatible on Linux, I need to be a hard -1; swift-corelibs-foundation is explicitly not ABI compatible at the moment between versions. As the work to finally reach parity continues, several changes have been made that would not have been allowed in a world where ABI compatibility is tantamount, and there is still a window where such changes may be necessary before we can 'declare victory' and start only making ABI-preserving changes.
This isn't a comment on the content or the intent. Once the above is settled, this looks like a fine area for proposals.
Can you elaborate on why this necessarily is a showstopper? In my mind, this just means that one of the artifact conditions on Linux would have to be the corelibs versions things were built against.
Currently, packages only version on the version of the tools their Package.swift supports (modulo the explicit version spec in the targets, IIRC), meaning they can still support a range of Swift releases. Binary artifact availability already has to provide for a range of OSes, OS families (like Linux) and platforms; do we also want to multiply that by the number of supported Swift releases?
Not necessarily, but the motivation of this proposal is to give users who currently have no other recourse options. The production of binary artifacts as envisioned by this proposal is expected to be something producers would automate.
I'm very much -1 on the idea of discouraging the use of binary packages. I don't feel this is adequately explained in the pitch; it feels overly-conservative, like the pitch authors are afraid of pitching it for some reason rather than it being a reasoned decision.
In general, and to be frank: I have a very deep dislike for how strict and opinionated SwiftPM is; it's extremely awkward to use and I feel it is self-defeating WRT the goal of encouraging a richer, package-based ecosystem.
For this pitch, I would rather binary packages be allowed by default. Users who are picky about it can have a flag on swift build to only allow source packages.
As far as the hash embedding in Package.resolved is concerned, we should of course still allow updates, possibly with some kind of SSH-like prompt:
The authenticity of host 'blah.blah.blah (10.10.10.10)' can't be established.
RSA key fingerprint is a4:d9:a4:d9:a4:d9a4:d9:a4:d9a4:d9a4:d9a4:d9a4:d9a4:d9.
Are you sure you want to continue connecting (yes/no)?
Agreed, I think this feature has been an ask for a while because there is this necessity, especially at an organization level, to require both the benefits of modular projects but closed encapsulation with vendored binaries for a whole host of reason. I think it's totally fair to say that vending binaries is not ideal and to encourage using source when possible, but I can't think of other package managers that go out of their way to prefer one or the other.
Comments on the pitch though:
General +1, I think the binaryTarget is expressive enough, although the when feels a bit awkward.
I think allowsBinary is fine, but I'm almost wondering if something other than .package would fit more so it's clear whether it's being vended or compiled
I would be okay with opt-in by default as that seems to be the general case for most package managers (correct me if I've just got a bad sample size)
binarySignature would be a nice to have, although I don't see it as required for the initial pitch. Would be interesting to do it Bazel style: if you're pulling in a vendored dependency, you have to do it at the top WORKSPACE level, so even if your transitive dependencies are trying to pull in something, you have to make sure that graph can be fulfilled
A non-goal isnât at all to do with discouragement; it is merely scoping this proposal. It is, after all, the very first one to address this problem.
As the proposal details, vending binaries hasnât been feasible until module stability in Swift 5.1. Given that Swift 5.1 itself hasnât even shipped yet, this proposal is pretty timely.
Great write up, thanks for taking the time (and for crediting me for something I didn't expect to be credited for )
I have a few comments, starting with this observation:
However, we anticipate that often a single package will consist of either exclusively source or binary targets.
I would not make this assumption. It is already not the case for many projects out there that attach binary releases to Github tags. I would rather treat this as a normal scenario. I misunderstood. Seems less critical.
About adding .binaryTartget, I wonder if this is semantically correct. Maybe it's just me but .target reminds me of the association with something like build product. I wonder if this could be expressed by separating binaries from the targets lists.
Similarly, any package must follow the same requirements to itself use the allowsBinary: true.
I would like to be able to opt-out of this behavior.
Furthermore, the hash of the binary should also be stored in the package resolved to avoid that the vendor changes the artifact behind a version without anyone noticing.
I would like to be able to opt-out of this behavior.
SwiftPM was explicitly designed in order to allow the eventual implementation of performant, scalable, and even distributed caches for package artifacts.
I see some challenges in this. How would SwiftPM know what to re-build and what to "accept" as coming from the cache? IMHO SwiftPM needs to have some level of awareness that artifacts can be cached (e.g. .version files in carthage)
As far as Apple platforms are concerned, this seems to be covering the same ground as XCFrameworks in Xcode 11, but in an incompatible way. This will put maintainers of binary packages in the position of choosing whether to support both, or only one or the other, and quite likely lead to apps needing to have both kinds of dependency.
I recognize the challenge of syncing open-source Swift evolution with closed-source Xcode development, but this doesnât seem like a good outcome.
What makes you say that? Our imagined artifact representation for Apple platforms would probably be an XCFramework, but the details havenât been worked through yet.
XCFrameworks are what Apple is encouraging developers to use for binary distribution rather than fat frameworks. Since that is the direction Apple is headed we felt that it was best to have this proposal embrace that direction.
This was a very conscious decision. We wanted to keep certain aspects simple because there is a lot to this problem no matter how many or few bundle types we support. We know that many popular binary only frameworks will need to migrate to XCFrameworks or the functionality will need to be expanded to support more types.
Very much this. Starting with the what we believe is the common case* while keeping it simple but extensible is what we wanted when putting together this pitch.
I want to support binary package distribution and I even want to support installation of packages. The reason they are not part of this pitch is that either of those would drastically increase the size and complexity of the pitch.
Last year I started researching the requirements to build, install, and consume binaries. There is a mountain of complexity with the build or install components and both of those features are useless if you can't consume them.
* In my experience, there are fewer developers that create closed source libraries for 3rd parties than consume them. Most of my current and previous co-workers have never created such a library.
I think I'd generally support requiring xcframeworks only given that if you don't properly strip simulator architectures App Store submission isn't allowed, and it wouldn't be evident to a consumer of a binary framework which architectures are necessarily included in a binary; unless SPM takes on appropriate architecture stripping it would require the consumer of these fat frameworks to have some custom tooling outside of the Xcode SPM integration like carthage-copy handles to properly submit, right?
edit: misphrased this, meant that SPM's binary support should require xcframeworks as it's the only way without external tooling to get a properly-formed binary for modules
XCFrameworks don't use fat binaries, and Xcode will pull only the required architectures out of the XCFramework upon submission to the App Store. There is no need to manually remove architectures from a binary XCFramework.
One thing I'm missing from the proposal is more elaboration on how dynamic binaries (dylib/so) are going to be handled. AFAIK, SPM does not output bundles of binaries to be distributed into environments that do not build the binaries (e.g. uploading a binary to be run into a Linux server).
I can see swift run working since the dylib and binary can be placed into the same .build location directory, but the moment you need to move the binary to another location, it becomes non-trivial with dynamic dependencies. It'd be interesting to see SPM set/conform to a standard for bundled distribution that works well on Linux and other platforms (does such a standard already exist in Linux? I'm not aware of one)
On the Apple side, iOS will be fine since Xcode will take care of the bundling for you, and with XCFrameworks it will only bundle the required architectures (as @harlanhaskins mentions), but if you want to build macOS Swift CLI tools outside of Xcode, the same concerns as Linux apply.
If the binary dependencies are static only, most of these concerns go away, since the code would be linked into the binaries themselves, but since the proposal mentions dynamic as well, I think this needs to be further elaborated.
EDIT: Just found out about the bundle support thread which might alleviate some of these concerns.
To me this is more about how to bundle an application for installation because this is already an issue with dependencies and the @rpath(s) of your application.
I also like the idea of having a more top level control for allowing binaries from specific sources. That is like Daniel said something we haven't thought of and I think we should incorporate it into the proposal.
)