I was reading through this document https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.md and the following seemed surprising to me.
External dependencies, including imported Swift module files and Clang headers, are tracked using a special
depends-external set. These dependencies refer to files that are external to the module. The Swift driver interprets this set specially and decides whether or not the cross-module dependencies have changed.
Because every Swift file in the module at least has its imports resolved, currently every file in the module has the same (complete) list of external dependencies. This means if an external dependency changes, everything in the module is rebuilt.
Is there any technical reason that all files in a module need to depend on all the external dependencies? It seems like each file could have a list of external dependencies based on its imports and the the references symbols.
Is this just work that hasn’t been done? Am I missing something? Or best case, has this or similar optimizations happened already and the docs are out of date?
@Douglas_Gregor and I were actually discussing this today. There are two things here.
First, the list of dependencies coming from the module's bridging header, if there is one, are duplicated between each source file, so writing them out in each file's swiftdeps (and reading it all back in!) is a huge waste of time. We should centralize this to be per-module and not per-file. In a large mixed-source project you might have hundreds of header files imported from your bridging header, multiplied by hundreds of source files. It adds up.
Making the external dependencies corresponding to imported Swift modules more fine-grained is something we could do at some point as well. It's not quite as easy as tracking used declarations, though. Consider this example:
- My module imports modules A and B
- I call a top-level function
A.foo() with a value of some type
B introduces its own overloads of
- I have to recompile my module, even if A didn't change, because any calls of
foo() might have to be recompiled, since the preferred overload might change
Today within a module we handle this by tracking names. The dependency information written out for each source file records a list of names the source file defines, and a list of names the source file uses. If any other file that defines a name that I use has changed, we have to recompile the other file, too.
Something similar could be done for imported modules.
Thanks for the detailed answer. I didn’t consider different overloads. Glad that people are thinking about ways to improve this. I normally work in a highly modularized project, so this jumped out to me because it could potentially have a large effect on reducing incremental builds.
I was wondering if there was any optimization made when the public api (or everything in swiftinterface) doesn’t change that would allow you to not have to rebuild modules that depend on that one. But I guess that would happen outside of the driver and be in the build system.
What's the latest on this? I see that there's been some work under way to enable cross-module incremental builds but if this works in the latest toolchain, I can't figure out the right incantation to enable it. Even a change to the private API in one module seems to trigger recompilation of dependent modules.
I ran into performance issues on big Objc/Swift project when a lot of Objc modules are imported into a large swift module. It causes all the dependsExternal to appear in each swiftdeps file. I described the issue here. Maybe we could emit dependsExternal in a separate swiftdeps file to remove duplications?