Modules are the core unit of code compilation and distribution in Swift. The task of taking a collection of Swift source files and orchestrating construction of a module is the prerogative of swift-driver. This post summarizes our recent work in Swift’s compilation model around modules and discusses some planned changes to enhance this process to achieve better efficiency and transparency.
Swift Driver as a Library
Within the last year, Swift transitioned to the new swift-driver as the default compiler driver, with the C++-based legacy driver now being phased out wherever possible. (If you find yourself still using the -disallow-use-new-driver option, please consider removing it. And if you cannot, please let us know; the option to use the legacy driver will soon be deprecated).
A major design goal of the new driver, since day 1, has been a Library-based architecture that allows better integration with build tools. Deploying a compiler driver, itself written in Swift, with this architecture, has made it possible to integrate the driver library directly into SwiftPM and Xcode - giving the higher-level build system broader visibility into compilation sub-tasks required to build a given module. Being able to centrally schedule all such subtasks individually, across all compiled targets, yielded significant improvements to build performance. (More details in this session from WWDC22: Demystify parallelization in Xcode builds)
Module Dependency Resolution and Explicit Builds
As outlined by @Douglas_Gregor a while back, building Swift code with explicitly-built modules is a goal to transition the compiler away from implicit dependency discovery and compilation. To re-state the problem: today, with implicit module loading, encountering an import statement in program code causes the Swift compiler to perform a filesystem search for the imported module. It may mean either:
- Discovering and loading an existing binary
.swiftmodule. - Discovering a
.swiftinterfacefor the required module and launching a compiler sub-invocation in a separate thread to build the textual interface into a binary module. - Querying Swift’s built-in Clang instance to perform module lookup and find a pre-built binary Clang module (
.pcm). - Similarly to (2), have Swift’s built-in Clang instance launch a module build for a discovered
.modulemapinto a.pcmthat can be loaded by the Swift compiler.
Notably, when loading/building dependencies, this process nests and repeats itself for all transitive module dependencies as well.
This approach results in several classes of problems that we are now in a position to tackle:
- Brittle build artifact sharing and work duplication: With many Swift compilations simultaneously in-flight, they are likely to have common dependencies, each attempting to both locate and build the same dependency modules. To deal with this, both Swift and Clang utilize filesystem synchronization mechanisms to attempt to avoid races with multiple compilers writing the same or similar modules into the module cache simlutaneously. Filesystem locking mechanisms can be brittle and are not sufficient to ensure maximal re-use of common shared artifacts. They also introduce a performance penalty of having compilers waiting on an arbitrary number of file-locks placed by other in-flight compilers. Moreover, all such compilations with shared module dependencies are each re-doing the work searching the filesystem for module dependencies.
- Difficult to debug failure modes: The resulting module cache state is prone to cache invalidation issues and races in the file-system, which are failure modes that are exceedingly difficult to reason about. Moreover, other dependency build failures may occur in deeply-nested compiler sub-invocations: understanding a dependency compilation error can mean having to reason about state of many levels of compiler sub-invocations from the original compiler frontend invocation.
- Opaque to build systems/clients: Dependency discovery and build operations are invisible to compilation clients (either the Swift Driver itself, or any other higher-level Build System). Meaning they do not participate in global task scheduling and failures during dependency build actions manifest from arbitrary parent compilation tasks that encounter them first.
Dependency Scanning and Explicit Module Builds
Explicit module loading moves compilation of textual module dependencies (both Swift and Clang) into separate compiler invocations. This is achieved by having a separate new stage of compilation: Dependency Scanning, which is invoked early during build planning by the driver. The dependency scanner is a component of the Swift compiler (invoked either via swift-frontend -scan-dependencies... or via API calls to libSwiftScan) which resolves imports to their underlying modules discovered on the file-system, which may fall into one of the four kinds of dependencies outlined above. The output of the scan is a build graph that contains the complete set of dependencies, direct and transitive, required to build the current module, as well as recipes on how to build them, for textual modules. Swift Driver takes this graph and produces a set of compiler invocations required to build all these dependencies into modules (either swift-frontend -compile-module-from-interface or swift-frontend -emit-pcm), which are then used as explicit inputs to compilation tasks. This results in the following improvements which directly address the shortcomings of implicit module loading outlined above:
- None of the compilation tasks of a given target search the filesystem for module dependencies.
- This is now done only once in the dependency scanner, ahead of time, saving redundant work from having to be repeated by each compiler invocation.
- None of the compilation tasks of a given target spawn compilation sub-invocations in a separate thread to build a textual module dependency (either Swift or Clang) into a binary module dependency.
- Defining away the potential errors caused by filesystem-locking and module cache consistency and inheritance of compilation state from parent invocations.
- Dependency build actions are explicit compilation tasks, with a standalone compilation command and input/output dependencies to all other compilation tasks. Failures during dependency build actions are unambiguously-attributable and reproducible.
- Dependency build actions are exposed to the build system and can be scheduled in-concert with all other compilation tasks.
Moreover, further removing the Swift compiler frontend from the filesystem state makes it more amenable to potential fine-grained build-caching technologies.
Explicit Modules Today
As per the previous update, SwiftDriver contains an initial implementation of Explicit Module Builds:
-
An individual target can be built with Explicit Modules using:
swiftc -explicit-module-build... -
SwiftPM can use it for explicit package builds with
swift build --``use-integrated-swift-driver --experimental-explicit-module-build... -
The Driver’s dependency scanner integration is being used in Swift Package Manager to gain build-time insight into direct target dependencies in order to diagnose usage of targets not explicitly-specified in the package manifest.
Ongoing Work
We are continuing our efforts to bring Explicit Modules up-to-par with today’s status quo in terms of both functionality and build performance. Our goal is to eventually fully replace/remove Swift compiler’s module-searching and ad-hoc dependency-building functionality. In the meantime, areas of focus span
- Working on ensuring correctness and maximizing re-use in the dependency scanner cache across different scanning invocations with the same scanner instance.
- Working towards eliminating the use of auto-linking directives when building with Explicit Modules: instead of inserting an auto-link directive, the driver can utilize its complete knowledge of the build graph to generate a complete linker invocation.
In the end state, the Swift compiler frontend will have only two module loaders remaining: explicitly-provided binary Swift module loader and explicitly-provided binary Clang module loader. With the rest subsumed by the dependency scanner and SwiftDriver’s dependency build planning.
Module Loading Future is Swift
The Swift compiler has been gaining in components written in Swift: the new Driver, libSwift as a means to write SIL Optimizations in Swift, and now the new Swift Parser. When explicit module loading becomes the default, the dependency scanner will become an integral early step in the compilation pipeline, invoked by the driver. Today, this involves either launching a swift-frontend task or querying libSwiftScan requiring the following functionality and logic that live in the compiler today:
- Knowledge of all (user-specified and default) search paths and how to traverse them on the filesystem looking for modules
- Ability to deserialize a binary
.swiftmodulein order to get at its list of module dependencies. - Ability to parse a
.swiftinterfacein order to extract its compilation flags and collect itsimportstatements. - Direct API access to the Clang dependency scanner to lookup Clang modules.
Because parsing Swift source-code will soon be done purely in Swift, the dependency scanner becomes the next natural candidate for itself being re-imagined in Swift. Today, the logic of implicit module loaders and dependency scanning is greatly intertwined. Without the need to do implicit module loading, the dependency scanner stands to benefit from being greatly simplified, affording an opportunity to do so in Swift along the way. Aside from this and other obvious benefits that using a language like Swift entails, this will, crucially, allow Swift’s Parser to reason about #if canImport() statements without involving the compiler and allow Swift Driver to perform build planning without involving compiler logic, further simplifying the compilation flow.
The new dependency scanner will then be required to address the above-outlined functionality:
- The Swift Driver has knowledge of user-provide search paths which it would convey to the scanner. The scanner will then need to abstract the file-system using something akin to what today is provided by STSC, or what is being built up in swift-system.
- An implementation of LLVM Bitcode deserialization already exists in Swift in STSC, and is already used in Swift Driver to read out incremental build information. This same implementation can be extended to read out a list of a binary module’s dependencies.
- The new Swift Parser makes it simple to be able to parse a
.swiftinterfacein order to extract interface-specific information (command-line flags, version, etc.) and gather allimportstatements. - The Clang dependency scanner vends a stable C API that Swift clients can use directly.