Why does it take so long to build and test SwiftPM on the CI?

Development Package Manager
,
1

This has been bothering me lately, as it now takes the same amount of time to build, test, and install SwiftPM as to build, test, and install the Swift compiler itself, just over an hour on linux. Here are the latest numbers from Fedora 41 x86_64 building the Swift 6.3 toolchain:

[2026-05-23T05:42:08.762Z] Total Duration: 13859.84 seconds (3h 50m 59s)
[2026-05-23T05:42:09.019Z] --- Build Script Analyzer ---
[2026-05-23T05:42:09.019Z] Build Script Log: /home/build-user/build/.build_script_log
[2026-05-23T05:42:09.019Z] Build Percentage 	 Build Duration (sec) 	 Build Phase
[2026-05-23T05:42:09.019Z] ================ 	 ==================== 	 ===========
[2026-05-23T05:42:09.019Z] 16.1%             	 2226.39               	 Running tests for swiftpm
[2026-05-23T05:42:09.019Z] 9.2%              	 1268.92               	 linux-x86_64-swift-build
[2026-05-23T05:42:09.019Z] 9.0%              	 1240.51               	 linux-x86_64-swift-test
[2026-05-23T05:42:09.019Z] 8.5%              	 1173.86               	 Building llvm
[2026-05-23T05:42:09.019Z] 6.0%              	 833.61                	 Building swiftpm
[2026-05-23T05:42:09.019Z] 4.7%              	 649.72                	 Building wasmswiftsdk
[2026-05-23T05:42:09.019Z] 4.6%              	 638.81                	 Installing swiftpm
[2026-05-23T05:42:09.019Z] 4.0%              	 550.0                 	 linux-x86_64-foundation-build
[2026-05-23T05:42:09.019Z] 3.5%              	 490.55                	 Building swiftformat
[2026-05-23T05:42:09.019Z] 3.0%              	 422.25                	 Building sourcekitlsp
[2026-05-23T05:42:09.019Z] 2.6%              	 367.07                	 Running tests for swiftdriver
[2026-05-23T05:42:09.019Z] 2.6%              	 360.29                	 Building wasmstdlib
[2026-05-23T05:42:09.019Z] 2.6%              	 357.47                	 Building wasmthreadsstdlib
[2026-05-23T05:42:09.019Z] 2.5%              	 341.55                	 Running tests for wasmstdlib
[2026-05-23T05:42:09.019Z] 2.1%              	 285.74                	 Installing swiftdocc
[2026-05-23T05:42:09.019Z] 2.0%              	 278.38                	 Running tests for foundationtests
[2026-05-23T05:42:09.019Z] 1.9%              	 257.76                	 linux-x86_64-lldb-build
[2026-05-23T05:42:09.019Z] 1.6%              	 227.6                 	 Running tests for sourcekitlsp
[2026-05-23T05:42:09.019Z] 1.6%              	 226.04                	 linux-x86_64-package
[2026-05-23T05:42:09.019Z] 1.3%              	 181.15                	 Running tests for swiftfoundationtests
[2026-05-23T05:42:09.019Z] 1.3%              	 173.7                 	 linux-x86_64-lldb-test
[2026-05-23T05:42:09.019Z] 1.2%              	 163.74                	 Building swiftdriver
[2026-05-23T05:42:09.019Z] 1.0%              	 138.23                	 Installing swiftdriver
[2026-05-23T05:42:09.019Z] 0.9%              	 122.69                	 Building earlyswiftdriver
[2026-05-23T05:42:09.019Z] 0.8%              	 110.99                	 Building swiftdocc
[2026-05-23T05:42:09.019Z] 0.8%              	 106.49                	 Building benchmarks
[2026-05-23T05:42:09.019Z] 0.6%              	 85.76                 	 Installing sourcekitlsp

Building and testing the 6.3 compiler takes 4X the time as for SwiftPM on macOS x86_64, but that SwiftPM time could probably be shaved down also. Note that these are timings for Swift 6.3, ie before swift-build was made the default build system in SwiftPM, but the trunk compiler:SwiftPM CI time ratios on linux and mac are pretty much the same, ie after switching to swift-build.

There are two main issues here:

  1. The Swift compiler is still mostly written in C++, whereas SwiftPM is mostly in Swift. Not a good sign that our build times are worse than notoriously slow C++, but build time is a constant complaint from Swift devs.
  2. The Swift CI invokes SwiftPM three separate times to build, test, and install with slightly different flags, which kicks off rebuilds of the same SwiftPM source. We could do a better job of build caching or making sure the flags don't do that. For example, the 10+ minutes to install SwiftPM, most of which is wasted on an unnecessary rebuild of the same source, needs to be cut way down.

Seems like improving the SwiftPM build would be a good way to dogfood improving Swift build times for everyone, either by applying existing techniques it isn't using yet or coming up with new ones.

2

Last I looked, the toolchain builds were Ninja and / or Cmake, so I don't think Swift's build system is an issue on CI, just the compiler itself for the Swift bits. While the performance there could be undoubtedly improved, it's not responsible for 13 hour build times.

Looking at that SwiftPM build log, the first issue is that it waited almost two hours just to get picked up by runner. These runners seem to be Amazon EC2 hosted macOS instances, which are absurdly overpriced, largely due to Apple's own licensing shenanigans (you can only "loan" instances for 24 hours at a time, so EC2 has to be vastly over provisioned and only offer whole machines). Solution here is just more runners, so either more money, or a host that won't gouge you. (Apple would really benefit from dog fooding macOS like this, macOS build nodes are actually a huge pain.)

Second issue, there seems to be no setup caching, so even after a build is picked up by a runner it takes 15 minutes just to start the actual build, just due to git operations.

Third, like you said, are the multiple unnecessary rebuilds, and a lack of integration in the pipeline. You could likely parallelize various parts of the pipeline, but it runs all on a single machine. There's no visualization of this stuff, so it's hard to see how much this might get you, but it's probably a lot. (I also see the version of sccache is 0.10.0, which is over a year old. 0.15.0 added tiered caching so you can actually pass whole caches around. Something to investigate.)

But by far the biggest issue here is the absolutely terrible performance of the build hosts. Looking at that SwiftPM build, I think it's running on an Amazon hosted EC2 x86 macOS runner, which, in reality, is a 2018 Intel Mac mini with a 6 (12 hyper) core 3.2GHz i7, with 32GB of RAM. Needless to say, this is a terrible Swift builder in 2026. This machine and its tools are so old it has to build LLVM/clang before it can even start the real build (seems to use Apple clang 16 / Xcode 16) to build the OSS clang 21, which is then used in the build. (Why it does that rather than just installing the precompiled clang 21, I don't know.) Even if Apple doesn't want more runners, just upgrading their existing runners from EC2 to Apple Silicon machines would be a huge boost in performance, and could even come out cheaper given EC2 time-based billing.

If you have build you want me to run locally for comparison, feel free to post it, and we can compare it to an M3 Ultra.

3 Likes
3

For everything before the Swift package manager is built, you are right, but not when building SwiftPM itself and after, where SwiftPM is mostly used.

In particular, SwiftPM has a bootstrap process, where it quickly builds a cut-down debug version of SwiftPM called swift-bootstrap using CMake/ninja, then uses that to build the full SwiftPM, after which that full SwiftPM is favored for all later components in the toolchain build, such as swift-driver and Docc.

I am intimately familiar with how it works because I'm one of the few people who cross-compile SwiftPM itself, so it runs on Android.

The issue here isn't the long build times for the full toolchain on the macOS CI, but improving the SwiftPM portion in particular, which in all the CI logs I linked above is one of the slowest toolchain components to build.

Whether this greatly reduces total CI time is not the concern, the focus is on improving SwiftPM build time because it should be generalizable to all Swift package builds, as SwiftPM is primarily built and tested using SwiftPM, even if it's just propagating existing Swift build performance techniques more.

The rest of your post is about lessening total CI time with other techniques, that have little to do with improving the SwiftPM build, so I've put my responses below the fold.

Jibber jabber about slow Swift CI generally

Ironic, as the Swift CI is Apple paying money that ultimately goes back to... Apple. :wink:

I'm sure they are, but maybe there's a disconnect.

All these points probably apply to the swiftlang pull request CI, which are each run dozens of times on weekdays, but all the CI I linked here are the full toolchain CI, which generate the full snapshot toolchains and are only run once or twice daily, often integrating dozens of pulls from several repos on each CI run. These are tougher to cache between separate toolchain CI runs because of all the changing source, whereas I was talking about caching the multiple SwiftPM builds in a single toolchain CI run above.

A native macOS arm64 toolchain CI is being spun up now, currently broken.

All builds of the Swift compiler build the corresponding Swift-forked LLVM and clang from source first, to ensure they work well together and because the Swift-forked LLVM is continuously worked on and updated.

No need, I regularly build the Swift toolchain on an M4 Mini since late last year, including in a linux container sometimes, so I'm well aware of how much faster that is. :smiley:

This thread is mostly about improving the Swift compiler and package manager, the CI is just an easy environment to look for such gains.