Explicit Module Builds, the new Swift Driver, and SwiftPM

Development Compiler
#5

This is a great explanation, thanks a lot!

+1 to what Max said, I'd love to see a single document that tries to summarize everything that is going on and how the community can help by testing or contributing to the new tools.

2 Likes
#6

Just to echo this, I think the best way to get people using these experimental features would be to enable their usage through Xcode, as it's still by far the most popular way to build Swift code. This would (hopefully) include being able to build apps as well as packages using the new settings. Otherwise I doubt we'll see much usage of these features until they ship.

4 Likes
#7

Agreed. This is definitely a known pain point currently, that it isn't as easy as we would like to have new builds of SwiftPM fully used in Xcode.

#8

Yes, I agree it's a pain point. swift-driver has some instructions for dropping the new Swift driver into Xcode builds, but they're a little hard to follow. Further integration of swift-driver into the toolchain will make it easier.

Doug

2 Likes
#9

Hello. Just want to know if there is any incentive to make swift support build-time metaprogramming, given the presence of libraries such as swiftsyntax?

#10

This is an awesome post. We need more of these. @Douglas_Gregor Could u point to a place where I can read more to understand how swift compiler makes this decision to use the cache or not?

#11

The module interface loader in the Swift frontend is responsible for deciding when to use a cached version of a module vs. rebuilding it.

Doug

3 Likes
#12

I'm assuming you mean something like a macro system? It's been mentioned in the past, but I don't know of anyone actively working on it.

Doug

#13

Would this allow to create several modules within one project? Something similar to namespaces without defining them explicitly as separate units of compilation?

Swift behavior of -gmodules and dSYMs
#14

No, this does not change the programming model at all. It’s a change to the implementation that should be invisible to users except for any benefits in build performance and robustness that it’s intended to bring (well, and bugs it might have).

2 Likes
#15

This might be not the perfect place to ask, but still. Is there something planned to do with underlying module importing? As for now, I see it like feature implemented with some hacks here and there, that will get in a way when implementing explicit modules (or I'm wrong?). May we see some different approach to it in the future?

#16

There are no plans to change anything about underlying module imports. The dependency scanner models Clang and Swift modules as separate entities in the module graph it emits, and the command-line flags are very different for building them, so there shouldn't be any extra issues here that we haven't accounted for.

Doug

#17

Hi everyone,

A few months have passed since this announcement was made and I wanted to provide a quick update on the progress that @Xi_Ge, @Douglas_Gregor and I have been making on this project, spanning the Swift compiler, new Swift driver, and Swift Package Manager.

Status

SwiftPM can now self-host using Explicit Module Builds. The package manager itself is a reasonably complex Swift package, and building it exercises most of the new machinery across the involved components. This was an important milestone for getting the basics of the new compilation flow functional.

Highlights

  • The new Swift driver's Swift library architecture made it simple to integrate the basics of explicit modules into SwiftPM by providing APIs for the new build planning flow.
  • Relying exclusively on explicit module dependencies has been a powerful learning tool for understanding the fine details of how today's Implicit Module Builds work as we try to achieve functional parity between the two flows. Interesting example:
    • Dependencies on Clang modules in a build graph mean that each Clang module must potentially be built and scanned multiple times (once for each depending Swift module with a distinct target).
  • Explicit Module Builds are allowing us to diagnose a class of problems at build plan time that would otherwise be caught much later in the compilation process or even cause compiler hangs and crashes.

Call to action

While the new build flow is still very much experimental, there are several ways to get your hands on it to start contributing bug reports and patches, please give it a try!

  • When using the new Swift driver as a drop-in replacement for today's driver [how-to] on the command line, Explicit Module Build of the target module is enabled with the -experimental-explicit-module-build flag.
  • SwiftPM packages can be built using Explicit Modules with the --use-integrated-swift-driver --experimental-explicit-module-build flag combination.

In expectation, the above flags should not affect the result of compilation; however, explicit module build jobs are not yet capable of interacting with the module cache, which can result in more computation and slower build times than their implicit counterparts.

15 Likes
The current state of swift-driver and...roadmap?
Swift-Driver Platform Support
#18

Is there a way to tell the driver to use an explicit Clang module? That is, an option equivalent to Clang's -fmodule-file=[<name>=]<file>?

#19

You should be able to pass in this very same Clang flag to the Swift Driver with something like:

-Xcc -Xclang -Xcc -fmodule-file=[<name>=]<file>

Then, when Swift is getting Clang to load this module, it should prefer the PCM you specify here over building one from-scratch.

For a very simple example, the above seems to work; but, it is very easy to run into various compatibility issues if your pre-built Clang PCM was built with a different set of command-line flags than those that the current compiler invocation will pass to Clang.

#20

Thanks a bunch! Works perfectly. :slight_smile:

#21

Hi. I'm new to compiler stack. I'm interested in new swift-driver because I want to use this to scan files that been effectected by the modification of single source file. Which may boost the build time for compiler-related tools like Static Analysis, IR generation, and so on like even Bazel build system, without having to fully compile all the files in module.

After reading current swift-driver and swift/libSwiftScan, I found for ".swiftinterface" deps, we use the Parser::getTopLevelDecls() to analysis the import module syntax, but we still parse the other uninterested decls like typealias

For clang module we use clang-scan-deps, which use the custom tokenizer to token only "include/import #if #else #end" decls.

Is there any plan to perform faster analysis speed using the similar tech ?

1 Like
#22

There are not specific plans, as far as I know; but, that sounds like it would be a great improvement.

#23

Thanks for the great write up! I wanted to ask about the limitation of .swiftmodule files being tied to a specific compiler version (and to some degree .swiftinterface files, since they are only forwards compatible).

I believe this is the issue of "module stability" which Swift does not yet support, is that correct? It's been quite a pain point for my team as we distribute a compiled .xcframework containing Swift code to our customers and there have been a lot of issues caused by folks being on different versions of Xcode.

For example, when we started compiling the framework with Xcode 13, the framework stopped working with anyone using an older version of Xcode because our .swiftinterface file now contains import Concurrency which is unknown to those older versions, even though we're not using any language-level concurrency features. I could be wrong on the exact details but I think that's the gist of it.

Is there any way to work around this other than compiling multiple versions of the framework with different Xcode versions? Do these changes you've described get us any closer to module stability?

Any info or advice you can provide is much appreciated!

#24

Swift has module stability, and has since Swift 5. For precompiled binaries you need to make sure to build for distribution, which ensures library evolution mode is on, along with other settings to maximize compatibility. Swift.org - Library Evolution in Swift With Xcode 13.2, there's also a back compatibility library for the Swift concurrency module which deploys back to the 2019 OS versions. It is buggy so back deployment may need to wait for Xcode 13.3, which fixes many of those issues. But if you aren't deploying any concurrency APIs building for distribution should be all you need. If it isn't you should report bugs.

2 Likes