Swift-frontend crash when building 5.8.1 on CentOS-Stream 10 under aarch64

Hi all-

I'm trying to build Swift 5.8.1 for CentOS-Stream 10 as the base version that will allow me to eventually get to 6.x. Because all Fedora-related software (Fedora being upstream of CentOS-Stream) requires the software to be built from source on Fedora/Red Hat servers, I can't just use an existing RPM file and use that; it must be buildable from source.

Okay, that said, I am able to build 5.8.1 on CentOS-Stream 10 on x86-64 platforms. For ARM/aarch64, however, the swift-frontend binary crashes when invoked, even without any arguments (e.g. /root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend).
The stack trace I get is:

swift-frontend: /root/rpmbuild/BUILD/swift-source/llvm-project/llvm/include/llvm/ADT/DenseMap.h:410: void llvm::DenseMapBase<llvm::DenseMap<swift::irgen::LinkEntity, llvm::Function *>, swift::irgen::LinkEntity, llvm::Function *, llvm::DenseMapInfo<swift::irgen::LinkEntity>, llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function *>>::moveFromOldBuckets(BucketT *, BucketT *) [DerivedT = llvm::DenseMap<swift::irgen::LinkEntity, llvm::Function *>, KeyT = swift::irgen::LinkEntity, ValueT = llvm::Function *, KeyInfoT = llvm::DenseMapInfo<swift::irgen::LinkEntity>, BucketT = llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function *>]: Assertion `!FoundVal && "Key already in new map?"' failed.
Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the crash backtrace.
Stack dump:
0.	Program arguments: /root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend -frontend -c /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/StdlibUnittest.swift /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/CheckStrideable.swift /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/InspectValue.swift /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/LifetimeTracked.swift /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/MinimalTypes.swift /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/OpaqueIdentityFunctions.swift /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/RaceTest.swift /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/Statistics.swift /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/StdlibCoreExtras.swift /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/StringConvertible.swift /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/StringTestHelpers.swift /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/TestHelpers.swift /root/rpmbuild/BUILD/swift-source/swift/stdlib/private/StdlibUnittest/TypeIndexed.swift -supplementary-output-file-map /tmp/supplementaryOutputs-c83b60 -target aarch64-unknown-linux-gnu -Xllvm -aarch64-use-tbi -disable-objc-interop -sdk / -I /root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/./lib/swift/linux -warn-implicit-overrides -enable-library-evolution -module-cache-path /root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/./module-cache -module-link-name swiftStdlibUnittest -resource-dir /root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/./lib/swift -swift-version 5 -tools-directory /root/rpmbuild/BUILD/swift-source/build/buildbot_linux/llvm-linux-aarch64/./bin -O -library-level api -D SWIFT_ENABLE_EXPERIMENTAL_CONCURRENCY -D SWIFT_ENABLE_EXPERIMENTAL_DISTRIBUTED -D SWIFT_ENABLE_EXPERIMENTAL_DIFFERENTIABLE_PROGRAMMING -D SWIFT_ENABLE_EXPERIMENTAL_STRING_PROCESSING -D SWIFT_RUNTIME_OS_VERSIONING -D SWIFT_STDLIB_ENABLE_UNICODE_DATA -D SWIFT_STDLIB_ENABLE_VECTOR_TYPES -D SWIFT_STDLIB_HAS_COMMANDLINE -D SWIFT_STDLIB_HAS_STDIN -D SWIFT_STDLIB_HAS_ENVIRON -D SWIFT_THREADING_LINUX -D SWIFT_ENABLE_REFLECTION -D SWIFT_ENABLE_REFLECTION -require-explicit-availability=ignore -enforce-exclusivity=unchecked -disable-autolinking-runtime-compatibility-concurrency -disable-objc-interop -disable-objc-attr-requires-foundation-module -enable-ossa-modules -enable-lexical-lifetimes=false -disable-implicit-concurrency-module-import -disable-implicit-string-processing-module-import -disable-implicit-string-processing-module-import -prespecialize-generic-metadata -define-availability "SwiftStdlib 9999:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999" -define-availability "SwiftStdlib 5.0:macOS 10.14.4, iOS 12.2, watchOS 5.2, tvOS 12.2" -define-availability "SwiftStdlib 5.1:macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0" -define-availability "SwiftStdlib 5.2:macOS 10.15.4, iOS 13.4, watchOS 6.2, tvOS 13.4" -define-availability "SwiftStdlib 5.3:macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0" -define-availability "SwiftStdlib 5.4:macOS 11.3, iOS 14.5, watchOS 7.4, tvOS 14.5" -define-availability "SwiftStdlib 5.5:macOS 12.0, iOS 15.0, watchOS 8.0, tvOS 15.0" -define-availability "SwiftStdlib 5.6:macOS 12.3, iOS 15.4, watchOS 8.5, tvOS 15.4" -define-availability "SwiftStdlib 5.7:macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0" -define-availability "SwiftStdlib 5.8:macOS 13.3, iOS 16.4, watchOS 9.4, tvOS 16.4" -define-availability "SwiftStdlib 5.9:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999" -target-min-inlining-version min -Xcc -DSWIFT_STDLIB_HAS_ENVIRON -parse-as-library -module-name StdlibUnittest -o /root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/stdlib/private/StdlibUnittest//LINUX/aarch64/StdlibUnittest.o -runtime-compatibility-version none -disable-autolinking-runtime-compatibility-dynamic-replacements
1.	Swift version 5.8.1 (swift-5.8.1-RELEASE)
2.	Compiling with the current language version
3.	While evaluating request IRGenRequest(IR Generation for module StdlibUnittest)
4.	While emitting prespecialized metadata for type 'Optional<Int>'
 #0 0x0000000005d3aeb0 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend+0x5d3aeb0)
 #1 0x0000000005d38ef4 llvm::sys::RunSignalHandlers() (/root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend+0x5d38ef4)
 #2 0x0000000005d3b2d8 SignalHandler(int) Signals.cpp:0:0
 #3 0x0000ffff873087a0 (linux-vdso.so.1+0x7a0)
 #4 0x0000ffff866cbe40 __pthread_kill_implementation (/lib64/libc.so.6+0x8be40)
 #5 0x0000ffff8667a900 gsignal (/lib64/libc.so.6+0x3a900)
 #6 0x0000ffff86665988 abort (/lib64/libc.so.6+0x25988)
 #7 0x0000ffff866738a0 __assert_fail_base (/lib64/libc.so.6+0x338a0)
 #8 0x0000ffff86673914 (/lib64/libc.so.6+0x33914)
 #9 0x0000000000a00c50 llvm::DenseMapBase<llvm::DenseMap<swift::irgen::LinkEntity, llvm::Function*, llvm::DenseMapInfo<swift::irgen::LinkEntity, void>, llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function*>>, swift::irgen::LinkEntity, llvm::Function*, llvm::DenseMapInfo<swift::irgen::LinkEntity, void>, llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function*>>::moveFromOldBuckets(llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function*>*, llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function*>*) crtstuff.c:0:0
#10 0x0000000000a00ae0 llvm::DenseMap<swift::irgen::LinkEntity, llvm::Function*, llvm::DenseMapInfo<swift::irgen::LinkEntity, void>, llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function*>>::grow(unsigned int) crtstuff.c:0:0
#11 0x0000000000a00a2c llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function*>* llvm::DenseMapBase<llvm::DenseMap<swift::irgen::LinkEntity, llvm::Function*, llvm::DenseMapInfo<swift::irgen::LinkEntity, void>, llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function*>>, swift::irgen::LinkEntity, llvm::Function*, llvm::DenseMapInfo<swift::irgen::LinkEntity, void>, llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function*>>::InsertIntoBucketImpl<swift::irgen::LinkEntity>(swift::irgen::LinkEntity const&, swift::irgen::LinkEntity const&, llvm::detail::DenseMapPair<swift::irgen::LinkEntity, llvm::Function*>*) crtstuff.c:0:0
#12 0x00000000009f85f4 swift::irgen::IRGenModule::getAddrOfValueWitness(swift::CanType, swift::irgen::ValueWitness, swift::ForDefinition_t) (/root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend+0x9f85f4)
#13 0x0000000000ada2a8 addValueWitness(swift::irgen::IRGenModule&, swift::irgen::ConstantStructBuilder&, swift::irgen::ValueWitness, swift::irgen::FixedPacking, swift::CanType, swift::SILType, swift::irgen::TypeInfo const&, llvm::Optional<(anonymous namespace)::BoundGenericTypeCharacteristics>) GenValueWitness.cpp:0:0
#14 0x0000000000ad9a80 addValueWitnessesForAbstractType(swift::irgen::IRGenModule&, swift::irgen::ConstantStructBuilder&, swift::CanType, bool&) GenValueWitness.cpp:0:0
#15 0x0000000000ad96c4 swift::irgen::emitValueWitnessTable(swift::irgen::IRGenModule&, swift::CanType, bool, bool) (/root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend+0xad96c4)
#16 0x0000000000a69340 swift::irgen::emitSpecializedGenericEnumMetadata(swift::irgen::IRGenModule&, swift::CanType, swift::EnumDecl&) (/root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend+0xa69340)
#17 0x00000000009e8c7c swift::irgen::IRGenerator::emitLazyDefinitions() (/root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend+0x9e8c7c)
#18 0x00000000008d3450 swift::IRGenRequest::evaluate(swift::Evaluator&, swift::IRGenDescriptor) const (/root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend+0x8d3450)
#19 0x000000000091f09c swift::GeneratedModule swift::SimpleRequest<swift::IRGenRequest, swift::GeneratedModule (swift::IRGenDescriptor), (swift::RequestFlags)9>::callDerived<0ul>(swift::Evaluator&, std::integer_sequence<unsigned long, 0ul>) const crtstuff.c:0:0
#20 0x00000000008e2db4 llvm::Expected<swift::IRGenRequest::OutputType> swift::Evaluator::getResultUncached<swift::IRGenRequest>(swift::IRGenRequest const&) crtstuff.c:0:0
#21 0x00000000008d4354 swift::performIRGeneration(swift::ModuleDecl*, swift::IRGenOptions const&, swift::TBDGenOptions const&, std::unique_ptr<swift::SILModule, std::default_delete<swift::SILModule>>, llvm::StringRef, swift::PrimarySpecificPaths const&, llvm::ArrayRef<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, llvm::GlobalVariable**) (/root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend+0x8d4354)
#22 0x00000000005da96c generateIR(swift::IRGenOptions const&, swift::TBDGenOptions const&, std::unique_ptr<swift::SILModule, std::default_delete<swift::SILModule>>, swift::PrimarySpecificPaths const&, llvm::StringRef, llvm::PointerUnion<swift::ModuleDecl*, swift::SourceFile*>, llvm::GlobalVariable*&, llvm::ArrayRef<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>) FrontendTool.cpp:0:0
#23 0x00000000005d6d5c performCompileStepsPostSILGen(swift::CompilerInstance&, std::unique_ptr<swift::SILModule, std::default_delete<swift::SILModule>>, llvm::PointerUnion<swift::ModuleDecl*, swift::SourceFile*>, swift::PrimarySpecificPaths const&, int&, swift::FrontendObserver*) FrontendTool.cpp:0:0
#24 0x00000000005d5ec4 swift::performCompileStepsPostSema(swift::CompilerInstance&, int&, swift::FrontendObserver*) (/root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend+0x5d5ec4)
#25 0x00000000005e6c88 withSemanticAnalysis(swift::CompilerInstance&, swift::FrontendObserver*, llvm::function_ref<bool (swift::CompilerInstance&)>, bool) FrontendTool.cpp:0:0
#26 0x00000000005d84f8 performCompile(swift::CompilerInstance&, int&, swift::FrontendObserver*) FrontendTool.cpp:0:0
#27 0x00000000005d7708 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) (/root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend+0x5d7708)
#28 0x000000000044ab08 swift::mainEntry(int, char const**) (/root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend+0x44ab08)
#29 0x0000ffff866660dc __libc_start_call_main (/lib64/libc.so.6+0x260dc)
#30 0x0000ffff866661bc __libc_start_main@GLIBC_2.17 (/lib64/libc.so.6+0x261bc)
#31 0x0000000000449bb0 _start (/root/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-aarch64/bin/swift-frontend+0x449bb0)
<unknown>:0: error: unable to execute command: Aborted
<unknown>:0: error: compile command failed due to signal 6 (use -v to see invocation)

I got around this problem before when building on aarch64 for Fedora by explicitly using Clang 15, which did not cause the problem. That version of Clang is unavailable in CentOS-Stream 10 as it's basically a brand-new version of the distro without any of the legacy versions to fall back on; Clang 19 is the version I have to work with and thus I guess I really have to figure out the issue.

I'm hoping someone may have an idea what to check for insofar as this is very architecture-specific; the x86-64 problem has never exhibited this behavior and the resulting RPM file works well.

Any help would be appreciated!

Ron

I don't know about that AArch64 crash, but can you try building 5.9.2 from source instead? The CMake config was switched to default to a prebuilt "hosttools" compiler right before that release, so if you delete those 5 lines, it should default to bootstrapping and still work.

Hi, @tachoknight. Good to see you again! :) I haven't tried building Swift from source recently, but I did last year. My patch in #70345 (or copr-dist-git) fixed the crash issues on Fedora with Aarch64. Also, @melizasw mentioned that this patch also works on Debian. I believe it will fix the issue on CentOS Stream 10 as well. I hope this helps. Thanks!

Hi @bclee ! Yes it was your patch I remember fixing the issue. The problem was that I didn't really want to spend a lot of time as 5.8.1 was the last version that didn't need an existing Swift install (until @Finagolfin mentioned a workaround that I didn't know about).
While looking for your patch/info I added a tiny patch to set the build to debug, and not release, and that actually made it build under aarch64; my goal is to immediately follow that up with going straight to 6.0.3-RELEASE so theoretically the changes of someone running an aarch64 version of CentOS Stream and installing Swift are likely pretty small.
It will be great to have EPEL 10 finally part of the release chain when deploying a new version.