As you can probably tell, I've been saving a lot of discussions, waiting for Swift 4.2 to be in a pretty good place before we jump all in on Swift 5. This one's one of the main things I plan to be working on in the next year or so, so I wanted to make sure you all had the big picture. It is another massive Jordan Rose post, so if you want the TLDR, just read the bolded sentences in each section and then jump down to the plan at the end. Thanks!
ABI stability means that an executable compiled against Swift 5 will work with the Swift 6 libraries, and that an executable compiled against Swift 6 will work with the Swift 5 libraries. A related concept is module stability, which says that the interface for a Swift 5 library will work with the Swift 6 compiler. (The opposite direction is less interesting.) More generally, the interface for a library should be forward-compatible with future versions of the compiler. This is useful in a number of ways:
- Can test a new compiler without rebuilding all of an app's dependencies.
- May overlap with work to make the debugger work across Swift versions.
- May help reduce incremental build time by better tracking cross-target dependencies.
- Support for general non-resilient binary frameworks. (More on what this means below.)
C accomplishes module stability through source stability, by using manually-written header files to represent a library's interface. Swift can do something similar by printing a type-checked AST to a textual form and including any extra information needed to reproduce the original compilation environment (such as the deployment target). To avoid the cost of loading this textual form, the compiler will keep a cache of library ASTs it has seen, serialized in the current binary format.
This interface will only contain the
open parts of a library, plus any parts that are marked as inlinable or are made available as part of the library's ABI. (For libraries compiled from source, this will include the layout of structs and enums, for example.)
|C-based Languages||Swift 4||Swift + Module Stability|
|Source files||.c, .m, .cpp, ...||.swift||.swift|
|Interface files||.h||.swiftmodule||.swiftinterface (new)|
|Interface is||manually written||generated||generated|
|Interface contains||public API,
inlinable function bodies
inlinable function bodies
inlinable function bodies (see below)
|Binary format for faster import||.pcm (in module cache)||N/A (already binary)||.swiftmodule (in module cache)|
|Language version||Chosen by client||Stored in interface||Stored in interface|
|Platform / deployment target||Chosen by client||Stored in interface||Stored in interface|
|Respects -D flags||Yes||No||No|
|Affected by search paths||Yes||Yes||Yes|
Like C, the plan for inlinable functions is to copy their bodies verbatim into the interface file. This isn't a perfect answer, since it leaves the interface more vulnerable to perturbations in type checking, but it does rely on the same source compatibility mechanisms Swift is already using, rather than forcing us to commit to a stable version of SIL (the intermediate representation used for high-level optimizations that's stored in swiftmodule files). When the textual interface is loaded, these inlinable functions will be compiled to SIL and cached.
Configuration Conditions (
C headers handle platform and user conditions by guarding sections of source with preprocessor macros. Swift, however, has to generate its textual interface, and configuration conditions are resolved well before type-checking. This implies that the generated interface file will be platform-specific. Rather than attempt to merge several platform-specific interface variants back into a single file, it's probably simpler to distribute a folder containing one interface per architecture, or possibly per OS/architecture combination. The former is how binary swiftmodule files work today.
Note that this is an outstanding issue for the generated Objective-C header. The Swift compiler is invoked once for each architecture, but only one header gets copied into the build product, chosen arbitrarily by Xcode. This is one of the reasons why this feature is not supported by the package manager.
Configuration conditions that are not based on the target don't really fit into this model, including user-defined conditions (
-D) and Swift language version checks (
swift(>=5)). These conditions are not required to be the same across a library and its clients, and thus should continue to be excluded from the interface. A developer who wishes to create different versions of their library using
-D flags should give each version a different module name or ensure that they are never used in the same environment, as they must do today.
Library Evolution ("Resilience")
With ABI stability and module stability, most of the pieces will be in place to build distributable binary frameworks that aren't tied to a single compiler version. Compared to Objective-C frameworks, however, there's still one piece missing: support for library evolution, or resilience. This is the feature that allows you to change a framework in a backwards-compatible way without having to recompile a client application.
A good chunk of resilience has already been implemented in Swift 4.2, and is being tested in the standard library and SDK overlays on Apple platforms. Once the standard library is shipped with Apple OSs, Apple will need to be able to ship new versions of the stdlib without breaking existing apps. But it's not necessarily sufficient for libraries not shipped with an OS just yet:
- There's no tool that will tell a developer when an ABI-breaking change has been made.
- Features like
@_fixed_layoutthat the standard library is using haven't been formalized for general use.
- Clients may want to check the version of a library to see if a particular feature is present, which implies having some version of
@availablethat isn't tied to OS versions.
- The compiler, runtime, and debugger all still have known issues or unimplemented features when working with resilient libraries.
While the lack of full resilience support will not preclude making binary frameworks, developers who use them would need to recompile their apps when a new version of the framework comes out. There is also one tricky case: if binary framework ABCKit depends on binary framework XYZKit, and XYZKit changes, ABCKit will need to be recompiled as well. It would be great™ if there was a way to detect this mismatch when compiling or linking the downstream client. (I don't have any concrete ideas yet.)
Use a binary format
It would be possible to use the existing binary format as the stable interface for a module (or something based on the existing format) rather than use a source-based format. This could be a lot simpler, since it's how the existing module-import code works. However, it has a number of downsides:
- More difficult to inspect, compare, and test (requires a dump-to-text step).
- More difficult to debug when things go wrong (because an invalid binary archive won't dump properly)
- Encodes implementation details of the AST
- Requires establishing a stable subset of SIL or embedding the source of inlinable functions into the binary format
- Still needs a "check" phase on import to ensure that dependencies haven't changed in an incompatible way
In practice, it seems like a binary format would still require a fair amount of work while having unfortunate drawbacks, and it would be harder to maintain in the long-term.
Use a non-source format for function bodies
It's possible that some users would object to the source of their function bodies being displayed verbatim in the textual interface file—what about secrecy? However, this is equivalent to inline functions in header files in C: the function body needs to be serialized in some way in order for the client to inline it, and using the same format as regular source is what lets us lean on source compatibility for forward compatibility. (It's worth noting that only code the developer explicitly marks as "inlinable" or uses in a default argument will be included in the interface files.)
Eliminate swiftmodule files (except in the cache)
The existing binary swiftmodule format is still useful for a handful of reasons:
It's still used for debug info, which requires information about all types in the module (even private and local types).
The initial design for swiftinterface files only contains public APIs, so they're not sufficient for unit tests that use
@testable import. This wouldn't be too hard to add to the design, but it still means larger textual interfaces when building with
-enable-testing. It's therefore not an initial priority.
Textual interfaces provide little advantage for libraries built from source in the same development environment (the common case for the Swift Package Manager). An eventual cross-module optimization mode would likely benefit from being able to share arbitrary, compiler-specific information across module boundaries, something that textual interfaces probably shouldn't designed to do (at least at first).
- Hook up ASTPrinter to a new command-line option,
-print-interfaceor similar, to produce .swiftinterface files.
- Teach Swift to "compile" interface files into the existing binary format, so that they can be loaded the same way they are today.
- Turn the above into an on-demand "module cache" like Clang. (The "on-demand" part may not actually be a good idea, but it'll make it easier to test this without having to modify existing build systems too much.)
- Add support for ABI details that aren't in the normal interface (like private struct fields).
- Lots of testing against real projects.