Illegal instruction: 4 when trying to compile project

We are attempting to compile our project with Xcode 10 GM. We have been seeing Illegal instruction: 4 since the first beta.
We have reported this issue, but we cannot submit our code and have not been able to come up with a sample project to demonstrate the error.
The illegal instruction shows up on roughly 52 of our about 1000 swift files.

We have noticed that enabling whole module optimization for our project makes the issue disappear, but we would prefer to not always have that on and to take advantage of the incremental builds. Example:
SWIFT_COMPILATION_MODE[sdk=iphonesimulator12.0] = wholemodule

What are the next steps we should take to debugging the issue?

Swift Crash:

0  swift                    0x00000001061cd64a PrintStackTraceSignalHandler(void*) + 42
1  swift                    0x00000001061ccdfe SignalHandler(int) + 302
2  libsystem_platform.dylib 0x00007fff609b3f5a _sigtramp + 26
3  libsystem_platform.dylib 0x0000000100fbe80a _sigtramp + 2690689226
4  swift                    0x00000001024fc41a swift::irgen::ClangTypeConverter::convert(swift::irgen::IRGenModule&, swift::CanType) + 394
5  swift                    0x00000001024fe9ea (anonymous namespace)::GenClangType::visitBoundGenericType(swift::CanTypeWrapper<swift::BoundGenericType>) + 650
6  swift                    0x00000001024fc41a swift::irgen::ClangTypeConverter::convert(swift::irgen::IRGenModule&, swift::CanType) + 394
7  swift                    0x00000001025c9348 getObjCEncodingForTypes(swift::irgen::IRGenModule&, swift::SILType, llvm::ArrayRef<swift::SILParameterInfo>, llvm::StringRef, unsigned long long, bool) + 88
8  swift                    0x00000001025c908e getObjCEncodingForMethodType(swift::irgen::IRGenModule&, swift::CanTypeWrapper<swift::SILFunctionType>, bool) + 414
9  swift                    0x00000001025c8d4e swift::irgen::emitObjCMethodDescriptorParts(swift::irgen::IRGenModule&, swift::AbstractFunctionDecl*, bool, bool, llvm::Constant*&, llvm::Constant*&, llvm::Constant*&) + 286
10 swift                    0x0000000102507275 (anonymous namespace)::ClassDataBuilder::buildMethodList(llvm::ArrayRef<(anonymous namespace)::ClassDataBuilder::MethodDescriptor>, llvm::StringRef) + 949
11 swift                    0x000000010250c536 swift::irgen::emitObjCProtocolData(swift::irgen::IRGenModule&, swift::ProtocolDecl*) + 1766
12 swift                    0x00000001025c7b2f swift::irgen::IRGenModule::emitLazyObjCProtocolDefinitions() + 63
13 swift                    0x000000010263be45 swift::irgen::IRGenModule::finalize() + 325
14 swift                    0x0000000102620658 performIRGeneration(swift::IRGenOptions&, swift::ModuleDecl*, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >, llvm::StringRef, swift::PrimarySpecificPaths const&, llvm::LLVMContext&, swift::SourceFile*, llvm::GlobalVariable**, unsigned int) + 1960
15 swift                    0x0000000102460a10 performCompile(swift::CompilerInstance&, swift::CompilerInvocation&, llvm::ArrayRef<char const*>, int&, swift::FrontendObserver*, swift::UnifiedStatsReporter*) + 46928
16 swift                    0x0000000102451dc5 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 7717
17 swift                    0x00000001023f7a35 main + 1349
18 libdyld.dylib            0x00007fff606a5015 start + 1

Generally when I ran into these cryptic compiler crashes in early Swift, it sometimes resorted to looking at how the complier crashed (i.e. the stack trace), and if that gave any helpful indication about what might be choking the compiler. Judging from the stack trace, are you using any @objc stuff?

Is this your first time testing the beta with your code?

Another way to try and narrow down what lines are causing problems is start commenting out blocks of code until the crashes start to go away.

We have been building with the Xcode betas since they have come out. We were able to get it compiling (with WMO on) and all our tests to run. But one of the last hurdles is getting it to work with incremental builds.

Yes, we do have plenty of @objc and @objcMembers. Our code base is roughly 65% Swift, so we still need to make some things available to objective c.

We do have our own framework that builds ok but the main app that imports the framework is having the issues.

Have been unable to determine how the files that are marked with the illegal instruction are related or if they are even the files that are producing the error.

I can try commenting out certain pieces of code to see if it reveals new info.

Ah, I missed that in your OP. Hm, if it's always been present that means it'll be a bit harder to narrow down what exactly broke things.

If the stack trace is anything to believe (and I'm far from an IRGen expert) it looks like something to do with generating the objective-c definition of a @objc protocol.

Also I've moved this to the complier topic so that the compiler gurus can maybe lend a hand.

We’ve had the same issue, but not for all targets in our project. If we find any further information on this I’ll post it here.

No differences in target settings have stood out to cause this.

EDIT: Actually all targets now fail in the GM seed.

If any of you can file a bug, either or bugs.swift.org or bugreport.apple.com, with your project or a subset of the project that reproduces the crash, we'll look into it as soon as we can.

Unfortunately i'm in a similar situation where I can't send the project. I'll make another attempt at replicating with a test project.

Thanks! Some other general things you might try if you haven't yet are changing the optimization level, which can sometimes affect crashes, enabling or disabling the new batch mode for incremental builds, or enabling or disabling whole module optimization.

@jrose notes that we have some known bugs with incremental batch mode that can manifest as a crash in ClangTypeConverter. Disabling batch mode may avoid the problem.

Optimization levels don't affect it.
Turning on whole module optimization does allow everything to compile.

This may be a dumb question, but where do i turn on/off incremental batch mode? I'm not seeing that in the project file.

I don't actually think we have any evidence that batch mode vs. Xcode-9.3-style incremental compilation makes a difference. The only consistent thing we've seen is incremental vs. whole-module, and even that doesn't make sense to me [yet].

I created [SR-8738] Project doesn't compile with Incremental compilation, something about ObjC protocols · Issue #51247 · apple/swift · GitHub that i'll add notes to as i test things and can provide more context that may be useful.

I also wasn't sure how to turn off batch mode in the Xcode GUI.
I copied the swiftc command Xcode produced to compile and removed the -enable-batch-mode flag.
After compiling on the command line, it still threw illegal instruction 4, but only seemed to appear on 2 of the files rather than around 50ish.
So turning off batch mode doesnt seem to fix it, only turning on whole-module.

If you look at the bug link, @jrose commented that batch mode can result in an error being shown for all files in the batch. Is there anything about those 2 files that stands out? (I assume disabling batch mode will result in single file compilation)

I stumbled upon the same error, and finally found out that it was caused by having this syntax:

@interface BaseModel<__covariant ModelType: BaseModel *> : NSObject

in Objective-C, and trying to use it in Swift.

More details here: [SR-11683] SourceKit crashes when trying to view Swift generated interface of an Objective-C header file · Issue #54092 · apple/swift · GitHub.

I was getting Illegal instruction: 4.

I narrowed in on and detailed the culprit here:
https://bugs.swift.org/browse/SR-12441