It looks to me like the debugging symbols you can download for macOS from Swift.org - Download Swift are built for x86_64 only:
/Library/Developer/Toolchains/swift-5.7-RELEASE.xctoolchain/usr/lib/libLTO.dylib.dSYM/Contents/Resources/DWARF/libLTO.dylib: Mach-O 64-bit dSYM companion file x86_64
Is this expected? Would it be possible for the Apple folks to include the arm64 debug symbols as well?