Looking for tools or techniques to reduce swift-toolchain build size

Hello! I'm looking to start a discussion on ways to reduce the space required to build the swift-toolchain (GitHub - apple/swift: The Swift Programming Language).

For context, I am trying to set up the swift toolchain + xcode + dependencies on my laptop (2017 macbook air, 128gb) to solve some type constraint bugs reported on GitHub. However, the build artifacts take 50-70gb, and xcode takes another 12gb. I run out of room before the build completes.

Would it be possible to replace parts of the toolchain with pre-compiled versions to reduce size? For example, I noticed the build scripts pulls in a llvm-project dependency (4gb).

I'm sure the memory requirements have been a headache for many before, so happy to hear thoughts.

3 Likes

Have you tried release builds by passing --release or --release-debuginfo to build-script? That could make debugging harder, but maybe it will reduce the amount of produced artifacts enough for the build to complete in the first place?

I find you can download one of the Swift.org - Download Swift toolchains then perform the much smaller utils/build-script ... --xcode build and then copy the swift-frontend executable from the DerivedData Xcode build products into the downloaded toolchain to test. You have to make sure to line up the clone date of your Xcode build with that of the downloaded toolchain but even a symbolic link from the toolchain to the Xcode built binary seems to work.

While I'm online, as a general announcement to compiler developers, I've created a small project which can speed up code iterations by allowing to you avoid having to build the swift-frontend executable with every edit GitHub - johnno1962/Compilertron: InjectionIII for the Swift compiler. It uses "interposing" (something a lot like a "swizzle") to patch in new implementations of any member function. You can have this work for a C++ program relatively easily by using the -interposable linker option.

The Compilertron app runs in the background watching for edits to C++ sources and prepares a patch dynamic library which is loaded at the last minute by a small patch you compile into the compiler itself. In this way, your iteration time is reduced to the time taken to compile the file. More details in the repo.

Thanks for the responses! I've been chugging away on my end to get things working.

As an experiment, I grabbed a USB stick (128gb) and compiled the source files on there. It was slow (~8 hours to compile from scratch) but worked! We will see how stable it will be. Of course, that doesn't alleviate the problem of swift-toolchain requiring a lot of storage to begin with.

@Max_Desiatov I've passed --release-debuginfo to build-script per the documentation here and see ~55gb build artifacts. It's quite possible there are more flags to reduce the artifact size, so I'll keep investigating! Definitely the lowest-cost path forward.

I found this thread which links to a YouTube video walking through the build system at a high level. Good viewing for anyone interested.

@johnno1962 Good to know! I'll have to dig deeper into this suggestion. Always love to hear interesting new dev tooling.

Will update with my findings when I find them.