Yes, it built the compiler multiple times, first without the portions of the Swift frontend written in Swift, then after building the Swift stdlib using that first stage, compiling the portions of the compiler written in Swift and linking those in too. It's relatively fast because it can keep reusing the built C++ components, just building and using more Swift components at each bootstrap stage. @etcwilde's alternative rebuilds everything, including LLVM, three times, which will take forever.
The problem is that the built-in bootstrapping of the compiler fell into disrepair once the CI all switched to using a prebuilt compiler, ie --bootstrapping=hosttools, which you do not even need to specify as it is the default on linux.
I strongly suggest you simply use that hosttools mode with the prebuilt compiler for Fedora as the hosttools. You have nothing to gain from a multi-stage build.
Yes, I think that's why, was just too lazy to confirm. 