On the topic of Swift runtimes, C++ instability, and mismatched NDK versions

I'm writing up notes on the challenges of C++ on Android and how it affects
distributing Swift runtime libraries and Swift binaries on Android.
This came up during the Android workgroup meeting and I was asked to document it
further for discussion.

As of Android 18, the Android distribution no longer includes a full C++ runtime
as part of the OS. Android applications that use C++ are expected to bundle
the C++ runtime as part of the application, either as a shared library
(libc++.so) bundled in the application, or statically linked into the
application binary (libc++.a). For a standalone application, this model is fine.
Everything is coming from a single version of the NDK, which is used to build
the application and any supporting dependencies. We'll call this the
"Application NDK". Different versions of the NDK may ship different versions of
the C++ runtimes, which can break ABI between them, but because the entire
application is only built with a single version of the NDK, it doesn't matter.

Introducing Swift complicates things. The Swift runtime depends on the C++
runtime, and the distributor of the Swift runtime only has control over the
runtime used to compile the Swift runtime. We'll call this the "Swift Runtime
NDK".

If both the "Swift Runtime NDK" and the "Application NDK" are the same version,
then the layouts of structures are the same and nothing will break. Otherwise,
the layouts of structures can change between versions of the runtime.

What is the essence of the ABI break? The C/C++ compiler use headers to describe
the layout of structures, and then generate address-offset loads into the
structure to access the members. To keep things simplified, lets say we're
writing an application that operates on 2D points and we have another library
consuming it, that our application also uses.

In an earlier version of the library, we declare a Point2D with the x first,
and the y second.

struct Point2D {
    int x;
    int y;
};

Then in a later version, we decided to flip them:

struct Point2D {
    int y;
    int x;
};

Then in a helper library, we create a function to extract the x value.
The helper implementation access the x directly and the header only exposes
the getX function.

int getX(struct Point2D pt) { return pt.x; }

If we build the helper library against the earlier version, the pt.x will
point at the first member of the Point2D structure. If we compile the
application against the later version of the Point2D library, the x will
actually be in the second position. If we construct

#include "helper.h"
#include "point2d.h"

struct Point2D pt = { 1, 2 };

printf("%d, %d\n", pt.x, getX(pt));

This will print '2, 1'.

Both versions will compile and link. The types are named the same way with the
same members, and the symbol names don't change. The x changes offsets though,
which results in the wrong member being accessed in the code compiled against
the old version of the library.

In this example, the Point2D library is analogous to the C++ runtime, the
library exposing getX(struct Point2D) is the Swift runtime built against an
older version of the C++ runtime, and the application is the application, built
against the headers for the newer version of the C++ runtime library.

In what is one of the more egregious versions of this, changes to the C++
standard resulted in an ABI break in std::string between C++03 and C++11.
Android does not make guarantees about the stability of the C++ runtime, and can
make this kind of change. More often than not though, most changes will likely
result in changes to symbol names which will result in a linker error instead
of quietly misbehaving at runtime.

With the exception of the C++ interop library, the Swift runtimes do not expose
C++ symbols. Dynamic libraries (shared objects) have facilities for hiding
private symbols. We can statically link the C++ runtime into the
libswiftCore.so, effectively hiding the dependency on the C++ runtime and we
can actually build the standard library against any version of C++ that we
choose.

Unlike dynamic libraries, static archives do not have facilities to hide symbols
like this. We can't link and resolve the C++ uses in the libswiftCore.a
archive. While one can technically use the link-flag ordering on POSIX linkers
to influence which C++ runtime symbols from each library are resolved against,
doing so is fraught with peril. Furthermore, lld is not a POSIX linker, so we
cannot use this trick here. I don't know that we can safely offer the Swift
runtime as a static archive that works with mismatched NDK versions.

The libCxxStdlib.a static archive that is part of the C++ interoperability
story does expose C++ symbols from the library. Shipping this library as a
dynamic library would not help us, as memory objects are passing the
library-boundary and can be accessed and manipulated by external C++ code.

This outlines the issues with the mismatched C++ runtimes coming from different
versions of the Android NDK. I've thrown out half-a-solution of only shipping
dynamic version of the Swift standard library, which would then allow us to hide
the C++ runtime dependency. This doesn't help us with the C++ interoperability
libraries though, which expose C++ symbols for consumption. Those libraries need
to build against a matching set of definitions as what is consuming it.

5 Likes

In this layout, isn't swiftCore the only library with a direct dependency on the C++ runtime? In that case, only swiftCore needs to be dynamic, doesn't it?

And swift_Concurrency at the very least, and a few others of the standard libraries.

If libc++ exposes an API to retrieve its version (or the NDK version it comes from) at runtime, each library relying on libc++ can check whether that version is acceptable and throw a runtime error if not. This works regardless of whether it is linked statically or dynamically.

If no such API exists, there may be ways to fingerprint the libc++ and deduce the version.

1 Like

Even better, could SwiftPM detect this at build time? And then fail — or issue a stern warning — like:

Error: The Swift Android SDK was built with Android NDK r27. Your application is utilizing NDK r29. You must use the matching NDK version when building projects that include C++.

2 Likes

Thanks for the detailed writeup. An additional advantage of starting to "statically link the C++ runtime into thelibswiftCore.so" is that we may be able to slim the dynamic Swift runtime libraries down further through Dead-Code Elimination (DCE). I don't think limiting our SDK bundle to "only shipping [the] dynamic version of the Swift standard library" is a workable solution though, as there will undoubtedly be those who want to statically link the Swift runtime libraries, particularly for further DCE that way.

The best way to deal with all this is to document it for the user and let them decide, particularly since we currently provide no easy way to bundle all this together into an Android apk anyway.

This is not some upcoming change: it has always been the case that native code in Android apps has to include its own C++ runtime with each app, since the system doesn't provide one, unlike with Bionic libc.

Yeah, I was wondering about that. For test cases in CI, we've always had to copy over the NDK's libc++_shared.so, which also needs to be bundled alongside the Swift JNI libraries within an .apk.

We've done a fair amount of mix-and-match testing with Android SDKs that were built against NDK 27 or 28, but whose runtime libc++_shared.so is from a different NDK (28, 29, or 30-beta1), and I have never encountered any issues that I suspected were related to a C++ mismatch. For example, the swift-android-native CI runs against both NDK r27d and r30-beta1.

I don't doubt that there is potential peril here, I've just never encountered an issue from a runtime mismatch.

1 Like

Actually, if the Swift target is distributed as a dynamic library, both the Swift runtime and libc++ can be linked to statically. The libc++_shared.so doesn't even need to be bundled in that case (i.e. it must not), avoiding the ABI mismatch in the first place.

Granted, unless: https://developer.android.com/ndk/guides/cpp-support#shared_runtimes

If your application includes multiple shared libraries, you should use libc++_shared.so.

Read "multiple shared libraries that depend on C++".

In that case, they would absolutely need to ensure that they are using the same NDK that we used to compile the runtimes with and we're still in the awkward position of having to serve multiple SDKs per Swift version for each NDK version. If we migrate the non-Windows Swift for Android SDKs to the new build system, we could consider making a single dynamic SDK available for all NDK versions, and then have a handful of static SDKs for each NDK that we want to support and then folks download the one that they want.

Right, I'm not saying NDK r18 is new. I just took what the Google NDK documentation said about C++ runtime. If they're wrong about how the NDKs work, I don't think there is much hope for anyone else. I should have said "NDK r18" instead of "Android 18" though.

This case still has the issue. We're pre-building the Swift runtime and providing that in the Swift SDK for Android. If we build the Swift runtime against a different NDK than what the developer is building the Swift target against, the references to C++ types in the Swift runtime may not match how they are implemented in the static archive that you're linking everything against while linking the dynamic library (or executable). If the developer's Swift library is a static archive, then there are just three places where bad things can happen.

Another thought. ELF and the Android loader do support symbol versioning. I don't have an NDK on me at the moment, but if someone wanted to check if the C++ runtime has versioned symbols, we could potentially use the dynamic library (symver is only handled by the dynamic loader, not part of static linking, so still doesn't help the static case). If symbol versioning is available, we could use an older NDK to compile the Swift runtime, and then the dynamic libraries in the newer NDKs would have both the symbols with any new ABI layouts in addition to the versioned symbols for the old version.

Technically, sure. The point I'd like to make is that the more we hide from the user, the better. I wouldn't go down the rabbit hole of versioned symbols to ensure that the ABI is compatible because it still involves a degree of reliance on the NDK.

I see how the Swift Android SDK, instead, could write down to a dumb metadata file the NDK version it was built with. The build system (SwiftPM or CMake) can then compare that with the NDK installation that the user is building against.

At that point, the policy is fully in control. You can make it a hard version equality requirement (like @marcprux mentioned earlier), or tolerate more relaxed "greater than" rules.

Okay, on a machine with an NDK. As suspected, it doesn't look like they are symbol-versioning it.

That's what symbol versioning helps with. We build against one NDK, the application builds against another, and as long as the other is at least as new as the one we used, things will link and the loader will extract the right ones at the right time. If the NDK is older, the versions of the symbols won't exist and it will fail to link. But that doesn't matter because they aren't versioning their symbols. I don't think we can get away from relying on the NDK, it's the source of truth on how to interoperate with the system.

In terms of emitting errors, if we want to guarantee safety checking, I think we would need to move it to the compiler/driver. There are other build systems out there with Swift support, like Gradle. Though I think that would be contingent on extracting the NDK version, which I'm not seeing in the metadata. I don't see anything in the NDK that we can key off of directly to get the NDK version.

This has come up a few times, i.e. platform version constraints for Android like we have for Apple platforms. There are similar needs for all our platforms. We definitely need to do something there.

You have everything about the NDK version in the root source.properties.

2 Likes

This is almost an argument for building Swift SDKs from source when building the apps that consume them to allow them to configure what platform parameters they need. You could always provide prebuilts for popular combinations so not everyone has to endure those build times, but it would be hard to satisfy everyone or keep up with NDK releases which have their own schedules.

I have big dreams for solutions like that, i.e. Swift SDKs defined in packages uploaded to the registry and built with SwiftPM. But it's still early days and we need to work through how feasible that would all be.

1 Like

No, we don't have to provide an Android SDK for each NDK: I think we should only provide one for the LTS NDK at that time, which will likely be LTS NDK 30 soon. For those who want to go to the trouble of statically linking using libraries from other NDKs, I don't think it's too much to direct them to build the SDK themselves, using the handy build scripts we provide that are run on our Android CI to produce the official SDK.

You misread what they wrote, "LLVM's libc++ is the C++ standard library that has been used by the Android OS since Lollipop, and as of NDK r18 is the only STL available in the NDK." They never provided a full system C++ stdlib to user apps in Android, but simply changed which C++ stdlib was available to apps to bundle starting with NDK 18. If you see the next section on the "system" C++ stdlib option from that NDK doc link, it never really provided much.

All this is to say that nothing has changed in this regard: Android app devs employing C++-derived native code have always had to bundle a C++ stdlib along with their app.

They started versioning their system Bionic libc symbols at some point, but that doesn't make sense for libc++ since it is supposed to be bundled with the dev's Android app anyway:

> ../ndk-27d/toolchains/llvm/prebuilt/darwin-x86_64/bin/llvm-readelf -sW ~/swift/ndk-27d/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/24/libc.so | grep LIBC_
    22: 00000000000172a8     8 FUNC    GLOBAL DEFAULT    10 clock_adjtime@@LIBC_N
    32: 0000000000017390     8 FUNC    GLOBAL DEFAULT    10 pthread_spin_trylock@@LIBC_N
    58: 0000000000017330     8 FUNC    GLOBAL DEFAULT    10 preadv@@LIBC_N
    86: 0000000000017280     8 FUNC    GLOBAL DEFAULT    10 __getcwd_chk@@LIBC_N
    90: 0000000000017290     8 FUNC    GLOBAL DEFAULT    10 __pwrite64_chk@@LIBC_N
   100: 0000000000017308     8 FUNC    GLOBAL DEFAULT    10 getifaddrs@@LIBC_N
   141: 00000000000172a0     8 FUNC    GLOBAL DEFAULT    10 adjtimex@@LIBC_N
   204: 00000000000172b8     8 FUNC    GLOBAL DEFAULT    10 fileno_unlocked@@LIBC_N
   205: 00000000000173b8     8 FUNC    GLOBAL DEFAULT    10 scandirat64@@LIBC_N
   216: 0000000000017340     8 FUNC    GLOBAL DEFAULT    10 pthread_barrierattr_destroy@@LIBC_N
   246: 0000000000017270     8 FUNC    GLOBAL DEFAULT    10 __fread_chk@@LIBC_N
   251: 00000000000172d8     8 FUNC    GLOBAL DEFAULT    10 fseeko64@@LIBC_N
   252: 00000000000173b0     8 FUNC    GLOBAL DEFAULT    10 scandirat@@LIBC_N
   253: 000000000001905c     4 OBJECT  GLOBAL DEFAULT    12 in6addr_any@@LIBC_N
   267: 0000000000017378     8 FUNC    GLOBAL DEFAULT    10 pthread_spin_destroy@@LIBC_N
   300: 0000000000017310     8 FUNC    GLOBAL DEFAULT    10 if_freenameindex@@LIBC_N
   307: 00000000000172e0     8 FUNC    GLOBAL DEFAULT    10 fsetpos64@@LIBC_N
   310: 0000000000017300     8 FUNC    GLOBAL DEFAULT    10 getgrnam_r@@LIBC_N
   355: 0000000000017328     8 FUNC    GLOBAL DEFAULT    10 lockf64@@LIBC_N
   374: 0000000000017348     8 FUNC    GLOBAL DEFAULT    10 pthread_barrierattr_getpshared@@LIBC_N
   391: 00000000000172c8     8 FUNC    GLOBAL DEFAULT    10 freeifaddrs@@LIBC_N
   425: 0000000000017278     8 FUNC    GLOBAL DEFAULT    10 __fwrite_chk@@LIBC_N
   462: 00000000000172f0     8 FUNC    GLOBAL DEFAULT    10 funopen64@@LIBC_N
   469: 0000000000017388     8 FUNC    GLOBAL DEFAULT    10 pthread_spin_lock@@LIBC_N
   513: 0000000000017380     8 FUNC    GLOBAL DEFAULT    10 pthread_spin_init@@LIBC_N
   517: 0000000000017320     8 FUNC    GLOBAL DEFAULT    10 lockf@@LIBC_N
   525: 00000000000172d0     8 FUNC    GLOBAL DEFAULT    10 freopen64@@LIBC_N
   526: 00000000000172f8     8 FUNC    GLOBAL DEFAULT    10 getgrgid_r@@LIBC_N
   589: 0000000000017358     8 FUNC    GLOBAL DEFAULT    10 pthread_barrierattr_setpshared@@LIBC_N
   598: 0000000000017288     8 FUNC    GLOBAL DEFAULT    10 __pwrite_chk@@LIBC_N
   639: 0000000000017318     8 FUNC    GLOBAL DEFAULT    10 if_nameindex@@LIBC_N
   640: 0000000000017370     8 FUNC    GLOBAL DEFAULT    10 pthread_barrier_wait@@LIBC_N
   673: 0000000000017398     8 FUNC    GLOBAL DEFAULT    10 pthread_spin_unlock@@LIBC_N
   684: 0000000000019060     4 OBJECT  GLOBAL DEFAULT    12 in6addr_loopback@@LIBC_N
   689: 00000000000173c0     8 FUNC    GLOBAL DEFAULT    10 strchrnul@@LIBC_N
   753: 00000000000172c0     8 FUNC    GLOBAL DEFAULT    10 fopen64@@LIBC_N
   754: 00000000000173a8     8 FUNC    GLOBAL DEFAULT    10 pwritev64@@LIBC_N
   821: 00000000000173d0     8 FUNC    GLOBAL DEFAULT    10 __system_property_wait_any@@LIBC_DEPRECATED
   910: 0000000000017298     8 FUNC    GLOBAL DEFAULT    10 __write_chk@@LIBC_N
   927: 0000000000017368     8 FUNC    GLOBAL DEFAULT    10 pthread_barrier_init@@LIBC_N
   953: 00000000000173c8     8 FUNC    GLOBAL DEFAULT    10 tmpfile64@@LIBC_N
  1019: 0000000000017350     8 FUNC    GLOBAL DEFAULT    10 pthread_barrierattr_init@@LIBC_N
  1077: 00000000000172b0     8 FUNC    GLOBAL DEFAULT    10 fgetpos64@@LIBC_N
  1120: 0000000000017360     8 FUNC    GLOBAL DEFAULT    10 pthread_barrier_destroy@@LIBC_N
  1129: 00000000000172e8     8 FUNC    GLOBAL DEFAULT    10 ftello64@@LIBC_N
  1135: 00000000000173a0     8 FUNC    GLOBAL DEFAULT    10 pwritev@@LIBC_N
  1164: 0000000000017338     8 FUNC    GLOBAL DEFAULT    10 preadv64@@LIBC_N

I think we are getting far ahead of ourselves here, as nobody has reported an issue with using conflicting libc++ versions and Marc, who vends the SDK to the most devs in his own Skip toolchain, says he's never seen the problem. Of course, it's always possible people aren't using C++ interop much yet on Android and the Swift stdlib calls some subset of C++ stdlib methods that haven't changed, so this may crop up later if we're not careful.

Let's test this C++ compatibility more and keep an eye on it, but I don't think it is worth delving too much into yet.

It feels like we have two different problems here:

  • The core and concurrency runtime libraries, which internally use the C++ stdlib in a fully encapsulated way.
  • The C++ interoperability library, which depends on matching the C++ stdlib that the user code is using.

The first problem feels like something we can fix by just building Swift differently, maybe statically linking the stdlib into the Swift dynamic libraries. The second feels like it demands an NDK-specific build of the interop library.