Plan for module stability

Could this swiftinterface file be faster to generate than a swiftmodule file? The reason I ask is because with some build systems we can parallelize the compilation of Swift modules across multiple machines, but we currently don't gain that much as the the swiftmodule file is an artifact of a compilation step. And so dependent modules will need for the lower level dependencies to finish compilation before they can start being compiled, which increases the critical path when building.

With C based languages we get around this as we have all the interfaces available as header files that are available before compilation even starts. With Java it's possible to generate such an interface to increase parallelism.

If this effort to create a new swiftinterface file could be made in such a way that these are way faster to generate than compiling the sources, this could be a huge benefit for distributed build systems.

(Disclaimer: I know it's possible to decouple swiftmodule generation from compilation, and I haven't tested whether swiftmodule generation by itself could be faster, but according to Tony Allevato, it should not bring many benefits.)

Yes, but such optimisations are orthogonal to the output format, as we could have a mode that makes swiftmodules (just as) fast to generate too, by skipping type checking etc. for the bodies of non-inlineable functions. This removes all interaction with the constraint system and expression type checker for such functions, and those can use a significant amount of CPU time, depending on the project. In either case, this would have to be opt-in, because it means errors in that code wouldn't be caught.

It is moderately faster, because it skips interactions with LLVM and everything after that (e.g. no low-level optimisations, no code generation, and no linking). The standard library's build system currently uses this scheme to provide more parallelism, by compiling the swiftmodule and the object files/dylib separately.

1 Like

The swiftinterface file will have essentially the same information as a swiftmodule file, just in a stable, textual format. Since the swiftmodule format is less constrained than the swiftinterface format, it should always be at least as efficient to use: if it's ever more efficient to use the swiftinterface format than the binary format of a swiftmodule, we'll just abandon the binary format.

As Huon says, we can definitely find ways to speed up the generate of a module description, but that's independent of the format of that description.

2 Likes

To elaborate on this, my concern for Bazel's Swift support has been whether splitting module and object generation into separate actions—which would each need to parse and typecheck N sources (and where typechecking can sometimes have its own performance issues)—would end up being an improvement over what we're doing now: using a JSON output map to produce all the artifacts we want in a single action. But the latter, as @sergiocampama mentions, means that downstream targets need to wait for full codegen when they otherwise might have been able to start building as soon as their .swiftmodules were ready (stopping after typechecking and some SILGen).

@huon (and others in this thread), would you say that the overhead of optimization and codegen is high enough that we'd see benefits of increased parallelism by separating out the .swiftmodule-generating actions from the ones generating the objects? This could help us in distributed builds where the actions are executed on completely different machines, but I'm less sure whether it would help or hurt in the more core-constrained local development case.

If it is the case that parallelizing them yields improvements, then I imagine .swiftinterface files would add a slight boost on top of that because it would remove the SILGen step from those actions as well.

1 Like

swiftinterface doesn't actually remove the SILGen step, because we don't want to include code that has bugs in it (like failing to initialize a variable). But both serialization formats would allow you to only SILGen and check inlinable code, rather than all of it, which is an optimization we don't do today.

1 Like

+1 for this.

I started wondering about this a while ago when we were discussing removing the Playground Quicklook APIs from the standard library (SE-0198 — Playground QuickLook API Revamp).

I gather that with this solution, we should be able to generate a swiftInterface containing the Playground Quicklook types/protocols, which would be available at compile-time in all contexts (regardless of whether your environment actually has an implementation of the framework available to load at runtime).

@jrose So, instead of implementing rock-solid and stable cross-library barrier/mechanism (like C DLLs, Objective-C Frameworks and Java Libraries have) you decided to just dump library compilation settings into text file?

In other words, instead of implementing expandable ABI standard you chose as paradigm constant creation of undefined number of binary compatibility layers.

In my opinion, it is the worst decision possible for binary frameworks. Even implementing constrained sub-syntax for framework's public API is better, because it would not break silently and due to unobvious reasons.

Inclusion of sources of inline functions into framework public API just shows that you chose completely wrong and insane paradigm.
Because the whole point of binary frameworks (and stable binary ABI) is to be INDEPENDENT of compiler, not require one to be able to be runtime-linked with the app.

1 Like

I don't think there's a reason for that tone, but I'll address the concern anyway: modules are only used at compile time. As shown in the table above, both the binary swiftmodule files we have today and the proposed textual formats are analogous to C/Objective-C headers (which do contain inlinable code as source), not to the DLLs / dylibs. The effort for compatibility across compiled code versions is ABI stability, and that's well underway.

7 Likes

Yes, sorry for the tone.

I'm just disappointed to realise the upcoming impossibility of binary libraries ecosystem (like Maven) for Swift due to discussed solution.

The whole problem with it is that it solves "ABI stability" by clashing together undefined number of old compilers just to make the linked libraries work.

It's to proper ABI standard is what Apple's Bitcode to JVM bytecode.

C/Objective-C headers containing inlined functions are just artefact of C era. True library interoperability / stability in industry relies on dynamic linking — be it linking of platform-specific binaries (Windows DLLs, for example) or binaries for virtual machines (Java, C#).

Basically the result of your proposal will make Swift binary library ecosystem extremely fragile outside of apple platform and tools. Detecting ABI-breaking source code changes via linter/tooling is inline with that.

1 Like

I'm not sure how this solution is different from DLLs. Can you elaborate?

I think you’ve completely misunderstood what’s being suggested here if you think it involves distributing old versions of the compiler and giving up on binary interoperation.

2 Likes

It involves SUPPORTING old versions of the compiler just to link with previously build library binaries.
Because what is suggested here involves using Abstract Syntax Trees (intermediary compiler product) and source code of inline functions as PUBLIC BINARY INTERFACE of libraries.

Yes. Basically what is good about real-life DLLs (on Windows) is the persistence and stability of their APIs, because all of them use C linking/calling convention (or calling convention that is standard to the platform) and standard name mangling.

As the result of instability of C++ name mangling library authors wrap even C++ libraries into DLLs with C OOP-like API.

In turn it spawned very rich plugin and interoperability ecosystem: starting from OpenGL/Vulkan extensions (which end-user 3D games can load in runtime without any compiler) to Java JNI libraries that implement native functionality that work alongside JVM.

Something similar Apple have with Objective-C frameworks: because Obj-C dynamic dispatch is so old, so widespread on mac/ios and so stable, and because language itself uses compiler-independent dynamic dispatch, Obj-C frameworks became even better library standard in macOS than DLLs in Windows.

Not mentioning Java, where JVM bytecode backward compatibility is rock solid and ensures existance of binary package management ecosystem (Maven/Gradle) that iOS/macOS can't even dream of.

Your proposal dismisses all this progress and throws us into the C++ incompatiblity age and instead just promises better backward compatibility in the compiler.

What is needed instead is ABI standard independent of the compiler, similar to what Obj-C offers with its dynamic dispatch. Even if it will mean syntax restrictions on exported Swift "public" (exported from library) classes.

In fact, one of the good solutions for binary stability of Swift frameworks may be following: to require every public Swift class (and its methods) exported from library is to be visible to Obj-C (with dynamic dispatch enabled), so that Swift framework that wants ABI stability would externally look like Obj-C framework.

Swift frameworks that want all Swift features in their external API and that don't need ABI stability can use the same mechanism as today.

You already have very reliable ABI standard for frameworks — Obj-C dynamic dispatch — so it is wise to just use it (with possible minor upgrades, like eliminating Obj-C header files and replacing them with more formal class/method specs, without possibility of function inlining).

How would that work for every other platform that doesn't have an Objective-C runtime?

1 Like

Very simple: Apple have to implement Obj-C runtime (dynamic dispatch) for every platform officially supported by Swift. In practice it means Windows (which already have iTunes with Obj-C runtime implemented in it) and Linux (which have GNU half-implementation, and Apple macOS implementation is Unix-friendly anyway).

Swift have three official types of method dispatch in the language anyway, and Obj-C dynamic dispatch is one of them. So such approach to ABI will also make Swift better supported on other platforms.

I think you're confusing ABI with runtime usage of symbols provided in a loadable bundle/dylib. This proposal is independent of that. It's saying that we should standardize on a format so that swiftinterface files generated with Swift 5 compilers are accepted by Swift 6 compilers as inputs. But this only applies to the compiling stage, it has no effect on linking or runtime access to the symbols in different loadable binaries.

7 Likes

This is a reasonable approach, but I'll just note that it won't work for default arguments, which no longer have public entry points. If you can't parse the source for a default argument expression, you won't be able to call the function without specifying the argument at all.

What if Swift 6 adds a new attribute not recognized by Swift 5? Perhaps the new attribute just says the function is "pure" and that shouldn't make the function unavailable to Swift 5. Or perhaps the attribute indicates a new calling convention or other restrictions that would make the function incompatible with Swift 5.

If we stick closely to the source format, I have the feeling the Swift-6 generated interface files will not be parsable by Swift 5 if they use any new thing from Swift 6. If you want your API to be usable from Swift 5 you'll need to generate two sets of interface files. Is that the plan?

Exactly. That's why the whole paradigm discussed here is wrong.

You cannot build ABI without specifying compiler-independent exchange format for cross-library boundary.

They should not be accepted by Swift 6 compilers, because they were compiled by different compiler.