Swift compiler hangs and I'm looking for a way to find the cause

After massive changes to a codebase of ~10k LOC of Swift, the Swift 5.1 compiler of Xcode 11.3.1 unfortunately hangs (Swift process goes to 100% CPU and never finishes) when trying to compile everything for release. A debug build still works. The "-O" is the problem.

I'm looking for some ideas to narrow down the problem so we can try to find a workaround. It's kind of urgent so we cannot wait for Apple to release a new version of Xcode. I tried to test a Swift 5.2 snapshot but it seems there's no Combine library included and therefore I built with that version at all.

Is there some way to switch on debug output so that the one call to swiftc which is run with all source files passed as command line argument can tell us a bit more where to look?

Stefan

Unfortunately I don't have a solution to your issue, but I do want to chime in with the 5.2 snapshot issue; I've also been trying to test the 5.2 toolchain and hit issues due to combine not being included. I filed feedback about it FB7555976. I don't know if the Swift team can highlight why this might be the case?

Bare in mind if you do manage to get it to compile with the 5.2 snapshot you won't be able to submit it to the store.

@sma Have you tried looking to see what file it hangs on and then removing those files from the target to narrow things down?

I have never compiled Swift using Xcode, but I have recently noticed some build flags that are passed by the Swift package manager on linux and Android that may affect you. When working on this SPM pull, I noticed that SPM will spin up a swiftc process per target, which will then spin off several swift frontend processes in Debug mode but keep them all in the one swiftc process in Release mode, by passing -num-threads instead.

I don't know if you're using SPM or something else to drive your builds, but it may help you isolate or even fix your problem if you check that that -num-threads flag is being passed in and remove it if so. At the very least, that should tell you which input file is causing the hang because you can see which few Swift files that swift frontend process was invoked on.

This is why I asked for some kind of debug output. Because everything gets compiled by just one call to swiftc I have no idea what source file might cause the endless loop.

I noticed that there's a -num-threads 8 argument and that only one swift process eats up 100% CPU so I guess that I don't reach the parallel compilation step but the compiler gets stuck in some kind of dependency tracing or something else that happens before the actual compilation.

I managed to extract my code to a package manager based project and therefore can compile without Xcode now – but it still hangs if I use -c release to build my framework for release.

I will try to incrementally remove code until I will hopefully work again…

This is why I asked for some kind of debug output.

I don't think the Swift compiler has anything other than the -v flag for some verbose output, and a couple other hidden ones like -dump-clang-diagnostics.

I noticed that there's a -num-threads 8 argument and that only one swift process eats up 100% CPU so I guess that I don't reach the parallel compilation step

No, if you pass both -j8 and -num-threads 8 to swiftc, the latter seems to override the former. I'd try to only pass the former flag, if only because you can then see each individual process and its flags using ps.

I managed to extract my code to a package manager based project

If you're invoking swift build from the command line, try adding the -v flag to see what swiftc commands it invokes, then re-run the hanging swiftc command with -j8 -v and the -num-threads 8 removed. That might help narrow it down.

Combine is an Apple-private framework. The swift toolchain is only the open-source components. People seem to forget that SwiftUI and Combine are still Apple-only, and are only released during an Xcode/MacOS upgrade.

If you're running on macOS or another BSD derivative, pressing ^T while the compiler is running will dump its current state to the terminal. The spindump utility on macOS can also capture stackshots of hung processes. Otherwise, you may want to run the offending compiler job in lldb or gdb and try to get a backtrace at the point it's hung.

1 Like

While that's true I'm not asking for Combine to be on any non Apple platform here. What I would expect is that a new Swift snapshot should be able to interface with the already existing Combine library available on the system. It managed to interface with UIKit, SwiftUI etc just fine. I don't understand why Combine is the exception.

1 Like

I think, you can import Combine but you don't have Apple's combine-specific extensions to other Foundation and UI classes like for example dataTaskPublisher. By adding this stub

extension URLSession {
    @available(swift 5.2)
    public func dataTaskPublisher(for url: URL) -> AnyPublisher<(data: Data, response: URLResponse), URLError> {
        fatalError()
    }
}

plus removing some code that depended on the fact, that DispatchQueue conforms to Scheduler in Apple's library (because I was too lazy to fake that conformance), helped me to make my code eventually compile with 5.2.

The compiler crashes with Abort trap 6 instead of hanging… yeah, progress… sort of.

Just to give you a quick update and to thank you for your replies – I found a workaround.

It seems, the compiler got confused by polymorphic makeBody methods returning some View – I changed them to return AnyView and Swift 5.2 no longer crashes while compiling the code and my release build with Swift 5.1 compiles without the compiler getting stuck in some C++ function called

swift::GenericSignatureBuilder::EquivalenceClass::getTypeInContext(swift::GenericSignatureBuilder&, swift::GenericEnvironment*) (in swift) + 268 [0x10a466f3c]
2 Likes

I took some time to reproduce the failure that you're seeing. There's an impedance mismatch between the Foundation in the public SDKs and the Foundation that gets shipped out in the swiftmodules and overlays built in the nightly toolchain. When you ran your code with Swift 5.1.3, you were most likely in an IDE environment and were using the appropriate OS toolchain's SDKs. When you use the nightly toolchains, we build your code against the Foundation overlay that lives in the public Swift project. That does not currently include the extension of URLSessionWebSocketTask that has the APIs you're looking for. It looks like those APIs were added fairly recently, so there's probably just a little bit of a waiting period here where Foundation needs to get the overlays resync'd.

Update: I have filed rdar://59238698 to track Foundation resync'ing these overlays. Your original feedback report will most likely be closed because of that. Cheers.

1 Like

@codafi thanks very much for having a look at this issue for me. I don't have the issue with the Xcode 11.4 beta so I've been able to progress with experimenting with Swift 5.2. Its good to know that this issue shouldn't exist in future with the vendored toolchains too. :+1:t2:

Check out my reply here. It will very likely help you.