Swift incremental build external dependencies

I was reading through this document https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.md and the following seemed surprising to me.

External dependencies, including imported Swift module files and Clang headers, are tracked using a special depends-external set. These dependencies refer to files that are external to the module. The Swift driver interprets this set specially and decides whether or not the cross-module dependencies have changed.

Because every Swift file in the module at least has its imports resolved, currently every file in the module has the same (complete) list of external dependencies. This means if an external dependency changes, everything in the module is rebuilt.

Is there any technical reason that all files in a module need to depend on all the external dependencies? It seems like each file could have a list of external dependencies based on its imports and the the references symbols.

Is this just work that hasn’t been done? Am I missing something? Or best case, has this or similar optimizations happened already and the docs are out of date?

2 Likes

@Douglas_Gregor and I were actually discussing this today. There are two things here.

First, the list of dependencies coming from the module's bridging header, if there is one, are duplicated between each source file, so writing them out in each file's swiftdeps (and reading it all back in!) is a huge waste of time. We should centralize this to be per-module and not per-file. In a large mixed-source project you might have hundreds of header files imported from your bridging header, multiplied by hundreds of source files. It adds up.

Making the external dependencies corresponding to imported Swift modules more fine-grained is something we could do at some point as well. It's not quite as easy as tracking used declarations, though. Consider this example:

  • My module imports modules A and B
  • I call a top-level function A.foo() with a value of some type
  • Module B introduces its own overloads of foo()
  • I have to recompile my module, even if A didn't change, because any calls of foo() might have to be recompiled, since the preferred overload might change

Today within a module we handle this by tracking names. The dependency information written out for each source file records a list of names the source file defines, and a list of names the source file uses. If any other file that defines a name that I use has changed, we have to recompile the other file, too.

Something similar could be done for imported modules.

Thanks for the detailed answer. I didn’t consider different overloads. Glad that people are thinking about ways to improve this. I normally work in a highly modularized project, so this jumped out to me because it could potentially have a large effect on reducing incremental builds.

I was wondering if there was any optimization made when the public api (or everything in swiftinterface) doesn’t change that would allow you to not have to rebuild modules that depend on that one. But I guess that would happen outside of the driver and be in the build system.

Terms of Service

Privacy Policy

Cookie Policy