It involves SUPPORTING old versions of the compiler just to link with previously build library binaries.
Because what is suggested here involves using Abstract Syntax Trees (intermediary compiler product) and source code of inline functions as PUBLIC BINARY INTERFACE of libraries.
Yes. Basically what is good about real-life DLLs (on Windows) is the persistence and stability of their APIs, because all of them use C linking/calling convention (or calling convention that is standard to the platform) and standard name mangling.
As the result of instability of C++ name mangling library authors wrap even C++ libraries into DLLs with C OOP-like API.
In turn it spawned very rich plugin and interoperability ecosystem: starting from OpenGL/Vulkan extensions (which end-user 3D games can load in runtime without any compiler) to Java JNI libraries that implement native functionality that work alongside JVM.
Something similar Apple have with Objective-C frameworks: because Obj-C dynamic dispatch is so old, so widespread on mac/ios and so stable, and because language itself uses compiler-independent dynamic dispatch, Obj-C frameworks became even better library standard in macOS than DLLs in Windows.
Not mentioning Java, where JVM bytecode backward compatibility is rock solid and ensures existance of binary package management ecosystem (Maven/Gradle) that iOS/macOS can't even dream of.
Your proposal dismisses all this progress and throws us into the C++ incompatiblity age and instead just promises better backward compatibility in the compiler.
What is needed instead is ABI standard independent of the compiler, similar to what Obj-C offers with its dynamic dispatch. Even if it will mean syntax restrictions on exported Swift "public" (exported from library) classes.
In fact, one of the good solutions for binary stability of Swift frameworks may be following: to require every public Swift class (and its methods) exported from library is to be visible to Obj-C (with dynamic dispatch enabled), so that Swift framework that wants ABI stability would externally look like Obj-C framework.
Swift frameworks that want all Swift features in their external API and that don't need ABI stability can use the same mechanism as today.
You already have very reliable ABI standard for frameworks — Obj-C dynamic dispatch — so it is wise to just use it (with possible minor upgrades, like eliminating Obj-C header files and replacing them with more formal class/method specs, without possibility of function inlining).
How would that work for every other platform that doesn't have an Objective-C runtime?
1 Like
Very simple: Apple have to implement Obj-C runtime (dynamic dispatch) for every platform officially supported by Swift. In practice it means Windows (which already have iTunes with Obj-C runtime implemented in it) and Linux (which have GNU half-implementation, and Apple macOS implementation is Unix-friendly anyway).
Swift have three official types of method dispatch in the language anyway, and Obj-C dynamic dispatch is one of them. So such approach to ABI will also make Swift better supported on other platforms.
I think you're confusing ABI with runtime usage of symbols provided in a loadable bundle/dylib. This proposal is independent of that. It's saying that we should standardize on a format so that swiftinterface files generated with Swift 5 compilers are accepted by Swift 6 compilers as inputs. But this only applies to the compiling stage, it has no effect on linking or runtime access to the symbols in different loadable binaries.
7 Likes
This is a reasonable approach, but I'll just note that it won't work for default arguments, which no longer have public entry points. If you can't parse the source for a default argument expression, you won't be able to call the function without specifying the argument at all.
What if Swift 6 adds a new attribute not recognized by Swift 5? Perhaps the new attribute just says the function is "pure" and that shouldn't make the function unavailable to Swift 5. Or perhaps the attribute indicates a new calling convention or other restrictions that would make the function incompatible with Swift 5.
If we stick closely to the source format, I have the feeling the Swift-6 generated interface files will not be parsable by Swift 5 if they use any new thing from Swift 6. If you want your API to be usable from Swift 5 you'll need to generate two sets of interface files. Is that the plan?
Exactly. That's why the whole paradigm discussed here is wrong.
You cannot build ABI without specifying compiler-independent exchange format for cross-library boundary.
They should not be accepted by Swift 6 compilers, because they were compiled by different compiler.
The original description talks about being forward compatible, meaning that it only guarantees that libraries compiled with Swift 5 will link in Swift 6. I don't think there should be an expectation that Swift 6 libraries are compatible with Swift 5 compilers. That would effectively block any kind of evolution of the language.
From the point of view of developers that want to release precompiled frameworks (along with the swiftinterface files), they should ensure that it they are compiled with the lowest version of Swift that they want to maintain and support, with the downside of not being able to use new language features available in later versions.
This is also pretty common in iOS for precompiled shared libraries.
1 Like
Taking Java as an example, since you mention it previously in the thread, are you saying that libraries compiled with Java 7 should not be linked against Java 8 binaries? Because they currently do. This proposal is analogous that approach.
No, this proposal is completely opposite to that approach, because Java have expandable forward-compatible JVM bytecode standard, so any compiled Java 7 library is also "Java 8 library" by definition.
This proposal is not only about not creating any bytecode standard (because Swift doesn't run on VM), but also not creating resilient BINARY BOUNDARIES between libraries, like currently DLLs and Obj-C frameworks do.
Instead, this proposal tells that to link and execute some Swift library in runtime, we will need full-fledged Swift compiler IN THE APP. And not only one Swift compiler, but ALL VERSIONS of previously released Swift compilers.
That's insane, and will increase runtime linking problems exponentially and will completely destroy any possibility of binary package ecosystem for Swift (like Maven for Java).
Let's say I make version 2 of an existing library and in the process of doing so I switch from Swift 5 to Swift 6. Also, I make sure the library is resilient so that users of version 1 don't have to recompile to use the new version.
Someone who build things with Swift 5 could gets the new version of the library, but won't be able to use it out of the box because the interface file is no longer compatible with Swift 5. There's two ways to use the new library: 1) switch to Swift 6, or 2) swap the interface file that comes with the new library with the one from the old library. The later option will work because the library is backward compatible.
It'd be nice if the backward compatible interface file could be auto-generated instead of having to be dug out from the old library. That's not going to be useful until Swift 6 though.
1 Like
I think this is the only thing where we agree. This proposal is not about creating a bytecode standard for Swift, and it's not about defining "binary boundaries" between libraries. It is about defining an approach for being able to compile (compile being an important keyword here) binaries with Swift N+M, against libraries compiled with Swift N (where M >= 0).
I'm still not entirely clear on what you want this proposal to be, but I'm certain it's not what you're talking about.
Where does it say that? It is not explicitly mentioned in the proposal (or any other reply), so maybe you're extrapolating consequences with wrong assumptions, which might be the core of your confusion.
It is indeed insane, (I'm actually impressed with your imagination) but again, not the purpose of or side-effect from this proposal. And IMO, this proposal is central to enabling a binary package ecosystem.
1 Like
Yes, I perfectly understand that. And such capability is only useful when your package/library ecosystem relies on availability of source code (like current github library ecosystem).
Imagine that you have Swift-based app. And now your app (or any Swift library in your app) wants to load IN RUNTIME plugin library that is written in Swift. Use cases for that are numerous: from graphics app plugin to game mods.
With approach from this proposal, your app (or any library that loads other libraries in runtime) needs to contain Swift compilers in itself.
No, it just strengthens current github javascript-like library ecosystem, where source code for any linked library is readily available.
With all associated downsides like long compile times, fragile linking, no runtime linking support, and impossibility of closed-source commercial libraries distributed through that ecosystem.
Contrary to Maven, where you download and link signed binary packages and can be 100% sure that linking will succeed as long these packages are compiled with current or previous version of compiler. And you don't need to have compiler in your app for that.
Imagine also that the version 2, in its implementation, not on the public interface, also uses a new stdlib data structure called NewStructure newly available in Swift 6. Because version 2 is compiled with Swift 6, there's no issue, it compiles fine, and you get a .a that your clients can link against. But NewStructure is not linked into version 2. It's part of the stdlib of Swift 6.
Now imagine that the swiftinterface files between version 1 (Swift 5) and version 2 (Swift 6) are exactly the same (i.e. Swift 6 didn't introduce incompatible changes in swiftinterface). Your clients will compile their code against version 2's swiftinterface, with Swift 5 and it will compile just fine.
But then the Swift 5 linker will see a reference to NewStructure from version 2. It doesn't know what it is, it's not in the stdlib, nor in any other library it's linking against. And thus will fail to link.
Because of these scenarios I would say that Swift 5 should explicitly fail to compile if any of the swiftinterface files were generated with a version greater than 5, and explicitly fail to link if any of the input libraries were linked with Swift version greater than 5. (@jrose don't know if the linking check would fall under this proposal or not, but I believe the swiftinterface check should be considered here).
1 Like
@virl It has been mentioned a few times that the .swiftinterface files are not for the runtime (just like C headers are not used by any runtime). Swift modules or frameworks link at runtime just fine - ABI stability takes care of that.
The .swiftinterface file serves a similar purpose to that of C headers to enable compilation of new code against a module.
So no Swift compiler is needed at runtime as I think you are suggesting.
2 Likes
Except, that if e.g. NewStructure (newly added in Swift 6) is used in the module's interface (i.e. as the parameter or result types of a new public function) then, by definition, either the Swift 5 compiler would be able to ignore the new function or the interface is incompatible, again by definition.
So, either way the Swift 5 linker will not be passed the new function.
Yeah, I'm all for making them incompatible and explicitly failing because of that. Not sure about the "ignoring" option. That sounds like a new vector for potential hard to debug issues.
About the linker, it will fail by virtue of NewStructure references being present in the library's .a. Ignoring symbols in linking also sounds infeasible.
Yeah, this is where you go off the rails. A runtime loader would not use the .swiftinterface file; it would use metadata embedded in the .dylib to do reflection. That metadata already exists, but the calls to introspect it from Swift’s userland don’t yet. They will eventually.
But when they do, they won’t contain certain pieces of essentially compile-time information, like doc comments or inlinable code.
3 Likes