Async Await crash on iOS14 with Xcode 13.2.1

Thank you both for testing it!

2 Likes

So what happens next?

Hopefully now it's very easy to fix. Fingers crossed we will have the fix in 13.2.x soon. :crossed_fingers:

1 Like

What a great Kon & Bal's puzzle page we've got here :nerd_face:

SCORING
1-10: There is nothing impossible to they who will try.
10-20: Believe you can and you’re halfway there.
20-30: I swear I am up to no good.
30-40: No, it’s not a Suzuki fans forum!

PS. it's not off-topic, you are just too young.

Also checked with lipo -remove arm64e and it helps! Maybe we can try to make some build phase script which removes arm64e on build as tmp solution.

Added such script into build scheme settings as post action script (need to select your main target in 'Provide build settings from') and now it always work without any additional magic:

FRAMEWORK_EXECUTABLE_PATH="${TARGET_BUILD_DIR}/${WRAPPER_NAME}/Frameworks/libswift_Concurrency.dylib"
if test -f "$FRAMEWORK_EXECUTABLE_PATH"; then
  EXTRACTED_ARCHS=()

  for ARCH in $ARCHS
  do
    echo "Extracting $ARCH from $FRAMEWORK_EXECUTABLE_PATH"
    lipo -extract "$ARCH" "$FRAMEWORK_EXECUTABLE_PATH" -o "$FRAMEWORK_EXECUTABLE_PATH-$ARCH"
    EXTRACTED_ARCHS+=("$FRAMEWORK_EXECUTABLE_PATH-$ARCH")
  done

  lipo -o "$FRAMEWORK_EXECUTABLE_PATH-merged" -create "${EXTRACTED_ARCHS[@]}"
  rm "${EXTRACTED_ARCHS[@]}"

  echo "Replacing original executable with thinned version"
  rm "$FRAMEWORK_EXECUTABLE_PATH"
  mv "$FRAMEWORK_EXECUTABLE_PATH-merged" "$FRAMEWORK_EXECUTABLE_PATH"
fi

Note: need to run this script only if build locally via xcode. I've just also added check if this is debug configuration (as we mostly run debug config via xcode):

if [ ${CONFIGURATION} = "Debug" ]; then 
...
fi
2 Likes

Fantastic detective work, Keith!

The fact that DYLD_* settings on launch affect the result explains why I wasn't able to reproduce; I had something similar set in my OS environment. I'm able to reproduce this on iOS 14 now, and can confirm that removing the arm64e slice addresses the issue.

Doug

4 Likes

It turns out dyld_info also surfaces this error in its output, although interestingly it still exits with a 0:

% dyld_info DerivedData/CrashTest2/Build/Products/Debug-iphoneos/CrashTest2.app/Frameworks/libswift_Concurrency.dylib
DerivedData/CrashTest2/Build/Products/Debug-iphoneos/CrashTest2.app/Frameworks/libswift_Concurrency.dylib [arm64]:
    -platform:
        platform     minOS      sdk
             iOS     13.0      15.2
    -segments:
        load-offset   segment section        sect-size  seg-size perm
        0x00000000    __TEXT                               256KB r.x
        0x000076F4             __text           187908
        0x000354F8             __stubs            2028
        0x00035CE4             __stub_helper      2028
        0x000364D0             __const           10931
        0x00038F90             __cstring          4918
        0x0003A2C6             __swift5_typeref   1677
        0x0003A954             __swift5_capture    388
        0x0003AAE0             __swift5_reflstr    467
        0x0003ACB4             __swift5_assocty   1056
        0x0003B0D4             __swift5_fieldmd   2764
        0x0003BBA0             __swift5_builtin     80
        0x0003BBF0             __swift5_protos      24
        0x0003BC08             __swift5_proto      212
        0x0003BCDC             __swift5_types      280
        0x0003BDF4             __objc_methname     241
        0x0003BEE5             __objc_classname     20
        0x0003BEF9             __objc_methtype     412
        0x0003C095             __info_plist        591
        0x0003C2E4             __unwind_info      5640
        0x0003D8F0             __eh_frame        10000
        0x00040000    __DATA_CONST                          16KB rw.
        0x00040000             __got               248
        0x000400F8             __mod_init_func       8
        0x00040100             __const            7080
        0x00041CA8             __objc_classlist     40
        0x00041CD0             __objc_imageinfo      8
        0x00044000    __DATA                                32KB rw.
        0x00044000             __la_symbol_ptr    1352
        0x00044548             __objc_const       1312
        0x00044A68             __objc_selrefs        8
        0x00044A70             __objc_superrefs      8
        0x00044A78             __objc_ivar           4
        0x00044A80             __objc_data          80
        0x00044AD0             __data             2152
        0x00045340             __bss             15280
        0x00048EF0             __common             56
    -dependents:
        attributes     load path
                       /usr/lib/libobjc.A.dylib
                       /usr/lib/libc++.1.dylib
                       /usr/lib/libSystem.B.dylib
                       /usr/lib/swift/libswiftCore.dylib
dyld_info: 'DerivedData/CrashTest2/Build/Products/Debug-iphoneos/CrashTest2.app/Frameworks/libswift_Concurrency.dylib' chained fixups, seg_count exceeds number of segments

It looks like part of the problem is that the chained fixups seg_count is 5, which corresponds to:

chained starts in image
  seg_count = 5
    seg_offset[0] = 0 (__TEXT)
    seg_offset[1] = 24 (__DATA_CONST)
    seg_offset[2] = 48 (__DATA)
    seg_offset[3] = 0 (__LLVM)
    seg_offset[4] = 0 (__LINKEDIT)

But then when you build your app, Xcode appears to strip bitcode from the dylib (since it isn't used by default in debug mode), but the fixups seg_count remains 5, so the output from otool actually appears to overflow and show something that's no a segment in its previous place:

chained starts in image
  seg_count = 5
    seg_offset[0] = 0 (__TEXT)
    seg_offset[1] = 24 (__DATA_CONST)
    seg_offset[2] = 48 (__DATA)
    seg_offset[3] = 0 (__LINKEDIT)
    seg_offset[4] = 0 (libobjc)

Interestingly if I set the undocumented BITCODE_GENERATION_MODE=bitcode build setting to force bitcode even in debug mode, the dylib still contains bitcode so it gets past the original issue, but I get a different crash still in dyld.

1 Like

Amazing work @Keith and thanks for finding the issue and providing more information.

@Douglas_Gregor / @zavsby / @Keith - I am wondering how does these findings gets into Xcode as a stable resolution, that we can use in production? I am aware of the excellent build phase script Sergey has put together but we would prefer to avoid workarounds. Any thoughts?

Ah, interesting. And running lipo -remove arm64e corrects the problem. This may be the reason why iOS is refusing to load libswift_Concurrency.dylib.

I'm curious about this different crash you're seeing. For me, setting BITCODE_GENERATION_MODE=bitcode in the build settings worked around the issue.

This cannot be addressed without a new Xcode release, and we (== those of us that work at Apple, like myself) cannot tell you when that will be. The issue is tracked by FB9780976, and we're looking into fixing it.

The build phase script is likely the safest workaround for now.

Doug

Assuming that you're targeting iOS 13+ shouldn't the binary link libswift_Concurrency not weakly? I noticed some of the other swift libraries are also weak like this that I assume could also not be since they are also required?

Interesting, I had assumed it would as well, but with just that change (like this Comparing main...bitcode-mode · keith/backdeploy-concurrency-crash · GitHub) I see this crash on my iOS 13 device:

* thread #1, stop reason = EXC_BAD_ACCESS (code=1, address=0x1831de43c)
  * frame #0: 0x000000010235dd78 dyld`dyld3::MachOAnalyzer::forEachObjCMethod(unsigned long long, bool, void (unsigned long long, dyld3::MachOAnalyzer::ObjCMethod const&) block_pointer) const + 224
    frame #1: 0x000000010235d128 dyld`invocation function for block in dyld3::MachOAnalyzer::forEachObjCClass(Diagnostics&, bool, void (Diagnostics&, unsigned long long, unsigned long long, unsigned long long, dyld3::MachOAnalyzer::ObjCClassInfo const&, bool) block_pointer) const + 504
    frame #2: 0x0000000102353a2c dyld`invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 544
    frame #3: 0x0000000102352c4c dyld`dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 168
    frame #4: 0x00000001023537c4 dyld`dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 180
    frame #5: 0x000000010235cf14 dyld`dyld3::MachOAnalyzer::forEachObjCClass(Diagnostics&, bool, void (Diagnostics&, unsigned long long, unsigned long long, unsigned long long, dyld3::MachOAnalyzer::ObjCClassInfo const&, bool) block_pointer) const + 204
    frame #6: 0x000000010236a5a8 dyld`dyld3::closure::ClosureBuilder::optimizeObjCSelectors(objc_opt::objc_selopt_t const*, dyld3::Map<char const*, dyld3::closure::Image::ObjCImageOffset, dyld3::closure::ClosureBuilder::HashCString, dyld3::closure::ClosureBuilder::EqualCString> const&, dyld3::closure::ClosureBuilder::ObjCOptimizerImage&) + 588
    frame #7: 0x00000001023691e0 dyld`dyld3::closure::ClosureBuilder::optimizeObjC(dyld3::Array<dyld3::closure::ImageWriter>&) + 876
    frame #8: 0x000000010236d364 dyld`dyld3::closure::ClosureBuilder::makeLaunchClosure(dyld3::closure::LoadedFileInfo const&, bool) + 1176
    frame #9: 0x0000000102333f1c dyld`dyld::buildLaunchClosure(unsigned char const*, dyld3::closure::LoadedFileInfo const&, char const**) + 368
    frame #10: 0x0000000102332b0c dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 2844
    frame #11: 0x000000010232d22c dyld`dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) + 432
    frame #12: 0x000000010232d038 dyld`_dyld_start + 56

The details from the device are similarly slim:

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x000000018391243c
Termination Signal: Segmentation fault: 11
Termination Reason: Namespace SIGNAL, Code 0xb
Terminating Process: exc handler [650]
Highlighted by Thread:  0

Backtrace not available

Unknown thread crashed with ARM Thread State (64-bit):
    x0: 0x000000016d609d50   x1: 0x0000000000000008   x2: 0x0000000000000000   x3: 0x0000000000000000
    x4: 0x000000016d609d20   x5: 0x0000000000000000   x6: 0x7366657272657075   x7: 0x0000000000000d60
    x8: 0x000000008000000c   x9: 0x000000016d609e93  x10: 0x0000000000046430  x11: 0x0000000000000004
   x12: 0x0000000000ff0006  x13: 0x0000000000048000  x14: 0x0000000000000000  x15: 0x0000000102b8c018
   x16: 0xfffffffffffffff6  x17: 0x0000000102a858d4  x18: 0x0000000000000000  x19: 0x000000016d609e3b
   x20: 0x0000000000000000  x21: 0x00000001038cc000  x22: 0x0000000000000000  x23: 0x0000000000000001
   x24: 0x000000008004643c  x25: 0x0000000000000001  x26: 0x000000016d609e93  x27: 0x0000000103912428
   x28: 0x000000018391243c   fp: 0x000000016d609ef0   lr: 0x0605290102a91e24
    sp: 0x000000016d609db0   pc: 0x0000000102a91d78 cpsr: 0x80000000
   esr: 0x00000000  Address size fault

Binary images description not available

Error Formulating Crash Report:
Failed to create CSSymbolicatorRef - corpse still valid ¯\_(ツ)_/¯

EOF

@Douglas_Gregor Makes sense! We will try the scripts in the meantime and will report back if we find any issues. Thanks!

It was intentional. There are is an unfortunate configuration where folks build binaries that use concurrency on iOS 15/macOS 12+ but deploy earlier and aren't app bundles (e.g., a command-line utility that's not within an app bundle). Non-weak linkage breaks these programs on launch; weak linkage allows them to continue to work so long as they are careful not to actually use concurrency on older OS versions. Aside from a small launch-time cost on older OS versions, using weak linking here doesn't hurt.

I'm not seeing this crash on iOS 14, perhaps because iOS 13 isn't expecting to see bitcode at all. It implies that BITCODE_GENERATION_MODE=bitcode isn't a viable workaround for everyone. The script that lipo's out the arm64e slice seems to be the best workaround for now.

Doug

2 Likes

I wonder for this case if it would also be better to strongly link the framework, given that folks who don't understand the inner workings of copying dylibs might not know about this, and hope to target older OS versions and have this "just work", IMO it might actually be nice to be explicit about having them weak link the target, but I also understand that less configuration can be nicer in some cases.

Either way my only thought is maybe this issue would have been caught sooner if it wasn't weakly linked, but I'm glad we found a resolution, thanks everyone!

This crash seems to occur only with debug build runs from Home Screen. To test this theory, I tested the code without build script provided by Sergey, in Adhoc and Test Flight build and it did not crash.

Could you guys explain why it is crashing only for debug build run from Home Screen? What is the difference between debug build from Xcode and run from Home Screen? Thanks!

This appears to be the same issue I ran into last month here — Using actor in iOS 14 crashes on initialization using Xcode 13.2

I can confirm that my app works when run initially from Xcode, but terminating the app and relaunching from the Home Screen crashes with the same 0x0000000000000000 EXC_BAD_ACCESS error.

It seems like the issue is reproducible / others are experiencing the problem now but I'm happy to help with any debugging as well.

When running from Xcode, some DYLD environment variables are automatically set, which force the process to load the arm64 slice rather than failing to load the shared library. As Keith showed, unchecking all of the options in the scheme that map to DYLD environment variables (eg, Main Thread Checker) will result in the program crashing on launch from Xcode as well.

Doug

Makes sense and thank you very much @Douglas_Gregor . I know you already answered this question but going to ask again :) Any thoughts when a new Xcode version resolving this issue would be released? Just a ball park, One week, one month?

Jay Muthialu

(post deleted by author)