1

There is a recurring bug on 32bit Android (armv7a) whereby s-c-f builds crash at runtime with a bus / alignment error:

libc    (11789): Fatal signal 7 (SIGBUS), code 1, fault addr 0xcc7a8e74 in tid 11789 (flowkey.testapp), pid 11789 (flowkey.testapp)
DEBUG   (11929): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
DEBUG   (11929): Build fingerprint: 'samsung/gtaxlwifixx/gtaxlwifi:8.1.0/M1AJQ/T580XXU4CRK5:user/release-keys'
DEBUG   (11929): Revision: '6'
DEBUG   (11929): ABI: 'arm'
DEBUG   (11929): pid: 11789, tid: 11789, name: flowkey.testapp  >>> com.flowkey.testapp <<<
DEBUG   (11929): signal 7 (SIGBUS), code 1 (BUS_ADRALN), fault addr 0xcc7a8e74
DEBUG   (11929):     r0 00000000  r1 cc7a8e74  r2 cc77ec08  r3 00000019
DEBUG   (11929):     r4 cc7a8e6c  r5 e2069860  r6 cc788448  r7 ff816540
DEBUG   (11929):     r8 00000019  r9 00000000  sl 00000000  fp cc7a8e6c
DEBUG   (11929):     ip cc7abbac  sp ff816138  lr cc1e05af  pc cc1e05c6  cpsr 600d0030
DEBUG   (11929): 
DEBUG   (11929): backtrace:
DEBUG   (11929):     #00 pc 005895c6  /data/app/com.flowkey.testapp/lib/arm/libFoundation.so (CFStringFindWithOptionsAndLocale+61)
DEBUG   (11929):     #01 pc 0058ebff  /data/app/com.flowkey.testapp/lib/arm/libFoundation.so (CFStringHasPrefix+142)
DEBUG   (11929):     #02 pc 005a8777  /data/app/com.flowkey.testapp/lib/arm/libFoundation.so
DEBUG   (11929):     #03 pc 005a870d  /data/app/com.flowkey.testapp/lib/arm/libFoundation.so (CFURLCreateWithFileSystemPath+14)
DEBUG   (11929):     #04 pc 0055b50f  /data/app/com.flowkey.testapp/lib/arm/libFoundation.so (CFBundleGetMainBundle+70)
DEBUG   (11929):     #05 pc 001e63e4  /data/app/com.flowkey.testapp/lib/arm/libFoundation.so
DEBUG   (11929):     #06 pc 000621dd  /data/app/com.flowkey.testapp/lib/arm/libc++_shared.so (std::__ndk1::__call_once(unsigned long volatile&, void*, void (*)(void*))+84)
DEBUG   (11929):     #07 pc 00410b15  /data/app/com.flowkey.testapp/lib/arm/libswiftCore.so (swift_once+44)
DEBUG   (11929):     #08 pc 001e64e8  /data/app/com.flowkey.testapp/lib/arm/libFoundation.so ($s10Foundation6BundleC4mainACvgZ+24)

I tracked this down to an access of the builtin FILE_ID_PREAMBLE macro in _pathHasFileIDPrefix, which may or may not be implemented as a static const (my C skills are intermediate at best so bear with me):

static Boolean _pathHasFileIDPrefix(CFStringRef path)
{
    // path is not NULL, path has prefix "/.file/id=".
#if defined(__CONSTANT_CFSTRINGS__) && !DEPLOYMENT_RUNTIME_SWIFT // see rdar://44356272 — the implementation of CFSTR() in Swift is not constexpr, but the compiler will miscompile it and allow it to be assigned to static variables anyway.
    static const 
#endif
    CFStringRef fileIDPreamble = CFSTR(FILE_ID_PREAMBLE);
    return ( path && CFStringHasPrefix(path, fileIDPreamble) );
}

For some reason this access ends up being misaligned in CFStringHasPrefix leading to the above crash. Looking at the libFoundation.so binary, I can see the entire CFString section has an alignment of 4 (2**2) and, in cases where it crashes, the DATA section appears to be misaligned:

33 cfstring      00006018  00b4c46c  00b4c46c  00b4b46c  2**2
                  CONTENTS, ALLOC, LOAD, DATA

The data section from which the constant string is being read (source verified via cat /proc/<pid>/maps) needs to have an alignment of 8 to work on Android, but currently – seemingly randomly – is moving between aligned and not aligned depending on the build (sometimes the 4-alignment of the data section happens to be divisible by 8, sometimes – as again in this build – it is not).

In collaboration with @compnerd on this issue, he created CoreFoundation: enforce 8-byte alignment on CFRuntimeBase by compnerd · Pull Request #2427 · apple/swift-corelibs-foundation · GitHub in the hope that it'd fix this issue. Unfortunately, the few builds in the meantime that worked seem to have been a coincidence.

Does anyone have an idea of how to fix this issue "for good"? I'm about the equivalent of the Mariana's Trench out of my depth here and need some more help from LLVM / linker gurus (beyond @compnerd, who has enough on his plate) to get this on the road again.

FWIW this is being linked with gold from NDK 16b. Also, 64bit builds do not have an issue.

2

Is CFString being laid correctly by the compiler in this configuration? This is generally the first constant string an app uses, which can be a canary crash for issues with constant CFStrings not being correctly set up by the compiler.

We switched to letting clang layout constant strings in 4.2 (accounting for s-c-f's layout differences via -fcf-runtime-abi=…).

3

(At the very least, the compiler may need to be patched to conform to the new alignment setup. @compnerd)

4

Thanks @millenomi for your replies here. FWIW @compnerd made CodeGen: ensure 8-byte aligned String Swift CF ABI by compnerd · Pull Request #348 · apple/swift-clang · GitHub with this issue in mind

5

Yeah, sorry for not circling back — this thread spurred the patch, and now things should be working again. Can you check?

6

I can confirm more definitively this week but early tests seem to show it’s doing what it should!

1 Like
7

Thanks for pinging us about it!

8

I still haven’t checked the binaries but we haven’t seen this crash come up again with various builds so I assume it’s fixed! Thanks to you both!

9

I'm running into this exact issue on some armv7a Android devices. I verified that I have the changes from PRs 2427 and 348.

I'm building using a fork of Vlad Gorlov's toolchain, which adds the clang alignment fix along with few other fixes.

Any ideas on what I could try?

10

Here are the results from the Google Play Console pre-launch report. All are armeabi-v7a but only some of them crash.

Motorola Moto G4 Play    6.0    Qualcomm MSM8916    SIGBUS/BUS_ADRALN
Nokia 1                  8.1GE  Mediatek MT6737M    SIGBUS/BUS_ADRALN
Google Pixel 2           8.1    Qualcomm MSM8998
Google Pixel 3           9.0    Qualcomm SDM845
LGE K3 2017              6.0    Qualcomm MSM8909    SIGBUS/BUS_ADRALN
Huawei Mate 9            7.0    HiSilicon KIRIN960
Motorola Moto Z          7.0    Qualcomm MSM8996
Samsung Galaxy S9        8.0    Qualcomm SDM845
Sony Xperia XZ1 Compact  8.0    Qualcomm MSM8998
Google Pixel             7.1    Qualcomm MSM8996
Google Pixel 2           9.0    Qualcomm MSM8998
11

Well this is interesting. I added some logging to CFBundleGetMainBundle to test out variations of string creation, rebuilt Foundation and my app, and then... no crash. Uploaded to Google Play and all devices ran without an issue.

Either there is magic in my logging or I just needed to build "one more time" to get everything into the right alignment.

12

Depending on how you’re building, some caches at some point along the line may have been kept of the old broken binary. Sounds like you’ve got it working though!

13

Vlad has some nice scripts for managing the Android build.

I reverted the debugging and the alignment crash returned. I added back the debugging and the app works again. So my debugging code is (at least indirectly) fixing the problem.

Unfortunately, PRs 2427 and 348 did not resolve the issue, and I'm guessing that depending on which commit one picks, the problem will come and go. Either that or something else is needed, such as an additional argument in the build somewhere.

Here is the debug code, which is added to CFBundleGetMainBundle. Perhaps it is padding the data enough so it is aligned?

#if TARGET_OS_ANDROID
#include <android/log.h>

struct TestConstStr {
    struct {
        uintptr_t _cfisa;
        uintptr_t _swift_rc;
        uint64_t _cfinfoa;
    } _base;
    uint8_t *_ptr;
#if TARGET_RT_64_BIT && defined(__BIG_ENDIAN__)
    uint64_t _length;
#else // 32-bit:
    uint32_t _length;
#endif // TARGET_RT_64_BIT && defined(__BIG_ENDIAN__)
};
#endif

CF_EXPORT CFBundleRef CFBundleGetMainBundle(void) {
#if TARGET_OS_ANDROID
    __android_log_print(ANDROID_LOG_ERROR, "Swift", "Test 1");

    CFLog1(kCFLogLevelError, CFStringCreateWithCString(kCFAllocatorSystemDefault, "Test 2", kCFStringEncodingUTF8));

#if __BIG_ENDIAN__
    static struct TestConstStr testStr = {{(uintptr_t)&_CF_CONSTANT_STRING_SWIFT_CLASS, _CF_CONSTANT_OBJECT_STRONG_RC, 0x00000000C8070000}, (uint8_t *)("Test 3BE"), 8};
#else
    static struct TestConstStr testStr = {{(uintptr_t)&_CF_CONSTANT_STRING_SWIFT_CLASS, _CF_CONSTANT_OBJECT_STRONG_RC, 0x07C8}, (uint8_t *)("Test 3LE"), 8};
#endif
    CFLog1(kCFLogLevelError, (CFStringRef)&testStr);

    __android_log_print(ANDROID_LOG_ERROR, "Swift", "Test 4 %llx", (unsigned long long)__builtin___CFStringMakeConstantString("Test 4"));

    CFLog1(kCFLogLevelError, (CFStringRef)__builtin___CFStringMakeConstantString("Test 5"));
#endif
14

I don’t know what to say other than, we’re not using that toolchain - just a standard build - and it works fine.

15

Are you building the toolchain on MacOS or Linux?

The official docs seem a bit outdated. But is that what you are using to build?

The docs claim Linux is required, but with Vlad's script I'm able to build on MacOS.

16

Id probably be willing to update the documentation. I've been building android from Windows, Linux, and I've also done macOS in the past.

17

That would be very helpful. Vlad's scripts are very helpful for automating the process, but it would be great to compare with the manual process to see if there are any differences. Thank you.