Include-What-You-Use and reducing time to build the Swift toolchain from source

Development Compiler
1

Background

include-what-you-use (IWYU) is a Clang-based tool that analyzes #includes in a file and makes suggestions to add or remove #includes based on usage in the code. This has two key benefits:

  • Removing unused #include statements reduces work for the compiler.
  • Adding #include statements for usage avoids a refactoring in a header file from breaking downstream implementation files due to accidental transitive usage.

Last week, I submitted a cleanup PR to remove a bunch of #includes that were not used, based on a hacky bash script. @compnerd suggested that I could try to get IWYU working on the Swift project codebase, as that would help attack the problem better.

Status

The cleanup PR I mentioned earlier was not intended to improve build times. Turns out, it did not make any dent in build times at all. :neutral_face:

/usr/bin/time -p utils/build-script --skip-watchos --skip-tvos --skip-build-ios-simulator --skip-build-benchmark --release --swift-darwin-supported-archs 'x86_64'
* before:
  real 1627.33
  user 25485.47
  sys 1116.88
* after:
  real 1619.04
  user 25488.07
  sys 1108.77

cd ../build/Ninja-ReleaseAssert/swift-macosx-x86_64
find lib name '*.o' -type f -delete
/usr/bin/time -p ninja
* before:
  real 399.75
  user 5532.17
  sys 242.22
* after:
  real 399.67
  user 5530.14
  sys 241.34

cd ../build/Ninja-ReleaseAssert/swift-macosx-x86_64
find lib name '*.o' -type f -delete
/usr/bin/time -p ninja swift
* before:
  real 287.96
  user 5034.64
  sys 208.26
* after:
  real 287.54
  user 5027.38
  sys 207.93

After that mild disappointment, I spent some time to get IWYU working. There were some challenges with ordering of #include paths and CMake being CMake but thankfully no fragile physical objects were damaged in the process. There are two artifacts of this work:

Notably missing are (a) a PR incorporating the IWYU suggestions and (b) an incredible build time speedup that will make you cry tears of joy. Sorry.

Questions and Next Steps

Why have I not submitted a PR incorporating IWYU's suggestions? For one thing, I don't feel qualified enough to judge the quality of IWYU's suggestions. Let's consider a couple of examples to make things concrete:

  1. IWYU suggests removing #include <string> in favor of adding #include "iosfwd". Is that a good suggestion? I don't know. If look at iosfwd's cppreference page, that doesn't mention that std::string is forward-declared by iosfwd. So is IWYU accidentally relying on the implementation details of the C++ standard library I'm using? :thinking:
  2. IWYU suggests adding headers with a bunch of __s like #include "__functional_base". That doesn't seem quite right... I'm somewhat confident that the __-prefixed headers are for the implementation of the standard library. :man_shrugging:

I also have an (unsubstantiated, quite possibly wrong) hunch that taking the philosophy of IWYU too literally by directly including each and every thing that is used (as opposed to relying on some intermediate headers and getting some things transitively) will lead to worse build times on the whole.

For these reasons, and a bunch of other practical reasons (time available to spare, risk of merge conflicts etc.), I have not taken further steps in trying to apply (a subset of) IWYU's suggestions. If someone wants to take that on and diligently measure impact on build times along the way, I'm happy to chat if you need help or just generally have questions.

N.B. If you want to start a broader discussion about improving build times and possible ways on doing that (not necessarily involving IWYU), it might be better to create a different thread and link this one there, so that this thread can be used primarily for discussing IWYU-related stuff.

5 Likes