Hi, everyone! It's about time we give an update on module stability. TLDR: it's looking good.
Aside: who's "we"? These days, it's mostly me, @harlanhaskins, and @Nathan_Hawes, but I also want to acknowledge contributions from @Graydon_Hoare before he left the project.
We've gotten our new stable module interface format up and working in the form of the ".swiftinterface" files you can see in a downloadable toolchain. The contents of these files is pretty much following the plan I laid out in "Plan for module stability", which is good! Though there are a few extra things in that file format that aren't found in normal source files (mostly around struct fields, as predicted way back in July).
By default, the interface files will be used whenever the corresponding "compiled module" (.swiftmodule file) is either absent or unusable. We have an environment variable that can change this preference for debugging purposes, but we don't expect anyone to need that in real life. Other than that, there's nothing special about using them; just have them show up in your search path wherever you'd currently use a .swiftmodule. (That includes both the "flat" layout "MyLibrary.swiftinterface" and the "target-specific" layout "MyLibrary.swiftmodule/x86_64-apple-macos.swiftinterface".)
There is still one big limitation: .swiftinterface files require -enable-library-evolution, the flag that Ben mentions in "Pitch: Library Evolution for Stable ABIs". Why? It's again because of those pesky stored properties: without -enable-library-evolution, the compiler has to be able to describe the layout of every type at compile time, but that might depend on private types that aren't present in the interface. We have ideas about describing type layout without actually mentioning concrete types, but it's tricky. (I'll write more about this in a later post; someone remind me to do that in a week or something.)
This limitation means that interface files can't yet be used for cross-module dependency analysis. But they're close! And that's a possibility even without implementing something that you can read back in; it just has to be something that changes whenever the layout of a public type changes. That's probably not something that we'll pick up in 5.1, though.
As mentioned in the original post, these interfaces aren't meant to support @testable import, nor can they take the place of the serialized ASTs in compiled modules for debugging purposes, because they don't contain information about non-public, non-@usableFromInline declarations. @Adrian_Prantl and others on Apple's LLDB team have been working on separate approaches to reduce the debugger's reliance on having the compiled modules in exactly the same form as they were when an app or library was built, but they're mostly separate from this.
As brought up in the original thread, we plan to limit this format to Swift 5 mode (i.e. no Swift 4 or Swift 4.2) just to cut down on the input space in the first release. I haven't actually put this restriction in yet, but I don't expect it to be burdensome since Swift 5 is much closer to 4 and 4.2 than, say, 4 was to 3. (We don't talk about 3 from 2.)
Oh, and we've been testing this in the main repository by building overlay interfaces to the old compiled module format. Within Apple, we've tried building various projects against those interfaces, as well as converting pieces of larger projects to generate interfaces as well.
I think that's about it! Any questions?
P.S. Final terminology: "module interface" (.swiftinterface file, the new thing) and "compiled module" (.swiftmodule file, the old thing, still useful, also our cached format as described in the original post)
Thank you Jordan! My only question is whether there is anything we will need to do in order to benefit from module stability in our own code, or if we just need to wait for your work to wrap up and ship in a release. The use case I have in mind is a framework that ships as a binary and is bundled with apps that use it.
As I understand it, the main benefit of .swiftinterface files is that they're a stable format that can be packaged with distributed frameworks/libraries. Then, when that module is imported, the compiler generates the binary .swiftmodule file and puts it in the module cache, just as Clang does with .pcm files.
That being the case, and ignoring all the limitations you mentioned above: for build systems like Bazel that build everything from source and ask the compiler to emit .swiftmodule files and propagate those up the dependency graph, is there any reason that we would want to switch to .swiftinterface files instead? Or should we continue to work directly with .swiftmodules because (1) presumably deserializing those is faster than re-parsing the text interface file and (2) those .swiftmodule files are only used during a single build and never distributed externally?
That sounds correct: if you're not going to mix compiler versions or send your module off somewhere else, the new interface files aren't providing you any real benefit today.
That's a good question. As noted, this is going to depend on the -enable-library-evolution mode for now, and that's probably what you want anyway for frameworks that ship as a binary. Even though you don't need the full ABI compatibility guarantee because of the bundling with the final app, someone else might make a binary framework of their own that depends on yours, and ideally their framework wouldn't be revlocked with yours. (Especially since there's no checking for that right now!)
Anyway, -enable-library-evolution does impose a few additional restrictions on code even if you don't use @_frozen, so you could try it out on your framework with a downloadable toolchain and see if you're going to hit any of those. Most libraries we've tried it with for testing purposes haven't run into those restrictions, though. (IIRC they're mostly around inlinable struct initializers.)
If I recall correctly, @harlanhaskins had suggested one more use case for .swiftinterface files, to increase build parallelism by separating interface generation and compilation into separate swiftc invocations. I was left with the impression .swiftinterfaces would be much faster to generate than .swiftmodules.
If so, then given module A, and module B which depends on A; module B could begin compilation much sooner than it would under the usual use case of waiting for A to be completely built before starting B.
The caveat here is that you're pushing the logic for typechecking, SILGenning, and pre-optimizing your modules to the clients, potentially slowing down each downstream client's build. This is a concern when a module has a lot of @inlinable code, such as the standard library.
In practice, few Swift modules are so full of inlinable code that this becomes an issue. Typechecking declarations listed in a .swiftinterface is always going to be faster than typechecking the function bodies that come from the source.
Yeah, I saw that it will depend on that flag for now. Thats why I mentioned waiting for your work to wrap up. I'm primarily curious about the answer in the case where we know the framework does not need resilience. I'm trying to understand the tradeoffs involved in the various levels of guarantee a framework might choose to provide. If it's too soon to answer the question about non-ABI stable binary frameworks that's ok, we can return to it in the future.
Thanks for the update!
Are there currently any thoughts on deployment target for module stable libs? For example would we be able to deploy to any iOS version, like iOS 9/10 etc or would we be limited to versions bundled with the 5.1 std libs and above (Say iOS 13 and beyond for arguments sake)?
Good question! Module stability is an entirely compile-time concept, so all that matters is that you use a 5.1 compiler (and new enough SDKs) rather than a 5.1 runtime. Library evolution support does have runtime dependencies, but only on things that made it into 5.0. So there should be no additional concerns around backwards deployment.
Now that -enable-library-evolution and -emit-module-interface-path have shipped with Swift 5.1 + Xcode 11, are there plans to start working on these limitations? If so is that being tracked somewhere I can follow? We're very excited about using these for cross-module dependency analysis.
if a library does not have binary-compatibility concerns (say, if it is distributed with the app that uses it), there is no benefit to handling structs and enums indirectly.
and
These considerations should not affect libraries shipped with their clients, including SwiftPM packages. These libraries should always have library evolution mode turned off
So if I ship a binary framework that clients bundle with their apps, should I enable -enable-library-evolution?
Also, I didn't see any mention of if there is an Xcode build setting to enable that compiler flag?
So if I ship a binary framework that clients bundle with their apps, should I enable -enable-library-evolution?
I think (not 100% sure) you need library evolution if you are distributing a binary framework to external clients. Otherwise, your clients will be forced to use the same compiler version as you're using; the swiftmodule format is not stable across compiler versions but the swiftinterface format is.
Also, I didn't see any mention of if there is an Xcode build setting to enable that compiler flag?
The new build setting is available in Xcode 11 -- it is under "Build Options -> Build Libraries for Distribution".
Forgot to mention I'm writing framework in Swift 5.1 and will require all clients to be developing their apps in Swift 5.1 or later. Not sure if that info affects your answer. Thanks for wwdc link.
For example, your framework might be compiled using the Swift 5.2 compiler, and some clients might still be using the Swift 5.1 compiler. Since that configuration needs to be supported, you need to provide a swiftinterface or you need to provide the source. Since you want to do binary distribution, you need to provide a swiftinterface and hence you need to enable library evolution.
I'm primarily curious about the answer in the case where we know the framework does not need resilience. I'm trying to understand the tradeoffs involved in the various levels of guarantee a framework might choose to provide. If it's too soon to answer the question about non-ABI stable binary frameworks that's ok, we can return to it in the future.