Since binary compatibility became possible in Xcode 11 and Swift 5.1, Alamofire has had the occasional issue reported where a user has enabled library evolution mode on the library and attempted to ship it as part of a dependency or use it in their app, only to encounter a runtime crash when attempting to use a newer binary. Alamofire doesn't support binary compatibility for a variety of reasons (which could be a separate discussion) but I'm wondering if there isn't something the language can do to make this scenario less painful, as it will keep happening.
The biggest pain point is that the addition of a parameter, even defaulted, is not a binary compatible change. So I wonder if it be possible to add an annotation, similar to @objc(), that would have the compiler emit the previous symbol for linking but which would call into the new symbol. I have no idea whether this is feasible at all, but it would work like this:
Say you had a function:
func async(completion: () -> Void) { }
Whoops, forgot to expose the completion queue, need to add it:
This is a source but not binary compatible change. You can either ship the incompatibility, or try to create an equivalent overload without the new parameter, which usually results in an ambiguity error. So perhaps an annotation could help:
Brainstorming broader solutions for the general problem...
What about a way to express it in the source? Would it help if the following were possible?
#if mode(libraryEvolution)
#warning("This library was not designed with ABI stability in mind; always build it from source together with any client code.")
#endif
I imagine it would become common enough to deserve a sugared short form if you don’t want to customize the message. Maybe something like one of these?
#noLibraryEvolution
#sourceStabilityOnly
#noABIStability
Or maybe the reverse would be better, so that the source declares that ABI has been considered, and library evolution mode would complain otherwise? That way developers who publish a package without ever even having heard of library evolution would get the warning for free. The onus would be on the ones who are thinking about ABI and are familiar with library evolution to enable it for their code.
As noted above, it's not source compatible either. Users can refer to functions by their fully-qualified name, like contains(where:). If you add a parameter to that function, defaulted-or-not, the function referred to by that fully-qualified name no longer exists, and the compile will break.
For exactly this reason, SwiftNIO forbids the addition of new parameters to methods, defaulted or not, in anything less than a semver major. We do manually what you're proposing to have the compiler do automatically here: define a new function that takes the extra parameter, then have the old one call into the new one. In some cases we also deprecate the old one (where that's appropriate).
The way I see it, you should still have two functions, but one could be annotated in a way the compiler will only resolve to it as a last resort. With this you can avoid ambiguities.
Most of those are due to default parameters. Is there any good reason why .init(_:) shouldn’t match init(_ decoder: Decoder, maximumBufferSize: Int? = nil) when .init(someDecoder) does?
At the very least it would require us to consider more generally the design we would want for partially applied method references (since that's what init(_:) would generate). If the more general feature is something we want to have a syntax for in the future, we would probably want to make sure that .init(_:) behaves exactly the same as (straw syntax) .init(_:maximumBufferSize: nil).
Even if we decided that we never wanted to support partial application more generally, there are still design questions (such as whether to evaluate default values at the time the partial reference is formed, or at the point where the reference is eventually applied) that would need to be considered and discussed.
We don't want deprecation to change the meaning of an existing program.
For a library shipped with the OS, deprecated APIs may be available on older OSs than their non-deprecated replacements. This doesn't apply to third-party libraries, but we don't really want to add more differences between library-evolution libraries and non-library-evolution libraries if possible.
On the other hand, the @lastResort idea has come up a number of times, and though it needs a lot of design (and may or may not be compatible with the current type checker implementation), I could see us doing it eventually. For now, though, if you want to be 100% source-compatible or binary-compatible at all, you have to leave a method's signature alone.
I feel like you’re, if not burying the lede, at least rushing by it on the way to the rest of the post. If Alamofire doesn’t support library evolution, your clients should not turn it on unless they’re willing to maintain an ABI-compatible fork. If they are turning it on and expecting everything to just work...well, I have questions.
Do they know that they’re turning it on, or is it happening accidentally?
Do they understand the consequences of turning it on, or are they (for instance) confused by the name of the build setting, or blindly following bad instructions from Stack Overflow?
Why are they turning it on? Is there some specific issue they’re turning it on in order to solve? Is there some better solution to this issue? Should there be?
Should it be more difficult for them to turn on? (@SDGGiesbrecht mentioned #error above; that’s the sort of thing I’m thinking about.)
The feature you’re proposing would reduce your ABI breaks, but if you’re not trying to be ABI-stable, presumably you would still break ABI sometimes. It might make more sense to focus on the cause, not the symptom.
By and large users are turning it on manually in an attempt to use binaries to optimize their builds or as part of a larger binary dependency.
Perhaps one of the biggest culprits are XCFrameworks. Everyone wants a way to distribute universal binaries within their projects, not understanding the implications of having to enable evolution mode to get them.
In general, expectations among the community do not match the reality of Swift’s binary support, both technically and as a matter of support among projects. I was wondering if there was a solution that could allow project to continue to evolve while maintaining some compatibility, but it doesn’t look like it.
Is Alamofire supposed to be a hidden implementation detail of these frameworks? If so, there’s no real reason these users need Alamofire to be ABI-stable—they just need to import it implementation-only, and they perhaps need to prefix its module name to keep it from colliding with other copies of Alamofire in the same process. @_implementationOnly import hasn’t been productized and I don’t think we have a good way to change a module’s name while building it, but these seem solvable.
Sometimes. Other times they’re just used as a way to optimize builds. I don’t think most users understand the subtleties of reexporting the symbols of a dependency or when they need ABI stability or not. They just want faster builds.
I can comment on why users do this. Sometimes an app is a thin layer on top of various frameworks that are the real app. These frameworks are developed and shipped together, often by the same people, are often imagined to be a single deliverable and so are barely versioned if at all. There are various reasons for this, most of them are either related to build speed or some kind of Conway's Law issue (each team has a framework).
Often, there is not a clear vision for which of these frameworks should do networking, and depending on the Conway's Law situation maybe they all need to. They were probably extracted from a single app target where import Alamofire worked in any file, so chances are the initial frameworkification will try to link it from most of its frameworks.
This mostly works fine as long as all these targets fit into an xcodeproj. Where you run into trouble is some team decides to circulate binaries instead. Maybe the org is compartmentalized and other developers aren't cleared to see some code, maybe one team needs a different build process than the others (once upon a time I worked around a 3rd-party bug with a weird build process).
IMO it would improve ergonomics if, when building for distribution, you were required to say @frozen import Foo for all Foo that were not built-for-distribution. That's much more clear that you are responsible for the fallout of some library being different at runtime, in the same way that you are responsible for @frozen struct etc.
As a second pass, it would be nice to get a warning/error (either at runtime or at ultimate-target-link-time) when one of these @frozen imports is a different image than it was when the immediate dependency was built. This is because the consequence of mismatch is hard to diagnose, it's UB basically. So some remark that identifies the problem (and perhaps goes on to UB) may prevent some of these bugs getting filed