The Swift compiler is difficult to efficiently integrate into build systems such as Bazel and Buck as it is a scheduling multithreaded compiler. This proposal suggests a minimal change to make it possible to efficiently integrate into build systems.
Currently the Swift compiler has two components, the driver and the frontend. The driver is responsible for creating and running separate frontend tasks to get the various tasks done. This is covered in detail in the docs, with recommendations for build system integration discussed here. There are a couple of problems with integrating as suggested:
- For whole module optimisation mode, it is not possible to schedule concurrent module builds without over or under subscription of threads. This is because the swift frontend itself will parallelise for part of the build if requested to create the separate object files in a single process. If multiple compiles are run concurrently and they generate object files concurrently this can result in much slower builds as the processes fight with each other for resources. If single threaded mode is used then the build system threads are not always effectively utilised, resulting in bottlenecks waiting for dependencies to build.
- One optimisation to reduce build graph bottlenecks is to generate a module’s swiftmodule in a separate pass. It is then possible to start building dependent modules without having to wait for object file generation to complete. Unfortunately this involves a lot of wasted work as there is no way to share the work done when generating the AST for the module during the swiftmodule generation pass, it will have to be repeated for the object file generation pass.
Both of these problems can be resolved with the same solution: produce an intermediate compiler output of a serialized AST that can be reused in subsequent compiler invocations for object file generation. This allows a build system to change the build graph as shown in the diagram from single threaded WMO builds to single threaded AST generation and multiple object file output processes:
This solves the wasted work problem as the compiler only has to generate an AST once, subsequent calls will focus on output generation and can be parallelised. This allows for a large number of modules to be efficiently compiled concurrently.
There already exists an
-emit-sib option in the frontend, which seems to be ideal for this purpose:
A representation of captured compiler state after semantic analysis and SIL generation, but before LLVM IR generation ("SIB", for "Swift Intermediate Binary").
Using this as input currently seems to fail at the IR verification stage, and disabling verification results in object files missing many symbols. Extending this intermediate binary format to work successfully as input to compile swiftmodules and object files would be the optimal solution for this issue.