For everything before the Swift package manager is built, you are right, but not when building SwiftPM itself and after, where SwiftPM is mostly used.
In particular, SwiftPM has a bootstrap process, where it quickly builds a cut-down debug version of SwiftPM called swift-bootstrap using CMake/ninja, then uses that to build the full SwiftPM, after which that full SwiftPM is favored for all later components in the toolchain build, such as swift-driver and Docc.
I am intimately familiar with how it works because I'm one of the few people who cross-compile SwiftPM itself, so it runs on Android.
The issue here isn't the long build times for the full toolchain on the macOS CI, but improving the SwiftPM portion in particular, which in all the CI logs I linked above is one of the slowest toolchain components to build.
Whether this greatly reduces total CI time is not the concern, the focus is on improving SwiftPM build time because it should be generalizable to all Swift package builds, as SwiftPM is primarily built and tested using SwiftPM, even if it's just propagating existing Swift build performance techniques more.
The rest of your post is about lessening total CI time with other techniques, that have little to do with improving the SwiftPM build, so I've put my responses below the fold.
Jibber jabber about slow Swift CI generally
Ironic, as the Swift CI is Apple paying money that ultimately goes back to... Apple. 
I'm sure they are, but maybe there's a disconnect.
All these points probably apply to the swiftlang pull request CI, which are each run dozens of times on weekdays, but all the CI I linked here are the full toolchain CI, which generate the full snapshot toolchains and are only run once or twice daily, often integrating dozens of pulls from several repos on each CI run. These are tougher to cache between separate toolchain CI runs because of all the changing source, whereas I was talking about caching the multiple SwiftPM builds in a single toolchain CI run above.
A native macOS arm64 toolchain CI is being spun up now, currently broken.
All builds of the Swift compiler build the corresponding Swift-forked LLVM and clang from source first, to ensure they work well together and because the Swift-forked LLVM is continuously worked on and updated.
No need, I regularly build the Swift toolchain on an M4 Mini since late last year, including in a linux container sometimes, so I'm well aware of how much faster that is. 
This thread is mostly about improving the Swift compiler and package manager, the CI is just an easy environment to look for such gains.