Let's add #if available(Android ?, *)

[Joe_Groff]

@Geordie_J

If it's possible to "weakly link" and make a shim that only works in case the symbol exists, that'd be my preferred outcome as well of course. From what I understand thought that's only possible by using dlopen at runtime as you suggest, which might well be impossible on Android for this use-case:

1. dlopen ing system libraries (anything from /system/lib(64) ) is a fatal exception on Android 7.0+
2. I actually have no idea which library we'd be dlopen ing here anyway (is getifaddrs provided by the C stdlib / runtime?). On Android > 5.0 Foundation "just works" without linking anything else at runtime.

You should be able to use dlsym(RTLD_DEFAULT, "symbolName") to search the entire image without dlopen-ing any specific image, since ELF has a flat symbol namespace (unless Android's linker also artificially hides symbols in system dynamic libraries from dlsym).

[jrose (Jordan Rose)]

@compnerd

@jrose, I'm sorry, I don't follow the model you are describing (that is, what are the values in play and how does that work together). The model being suggested here is that you have a version V , which is the minimum version that you can deploy to.

No, no. The point of a static check is that you're asking about the maximum level of API you can use in the program. On iOS, that's "literally anything"; on Android-with-Java, that's presumably "your target SDK" (not your SDK); on Android-with-NDK, that's "your min SDK" only because we don't have weak linking to pick something newer.

I think it helps to think about the iOS use case for the static check: I want a codebase that compiles with either Xcode 10.2 or Xcode 11. If I'm compiling with Xcode 11 (and therefore using the iOS 13 SDK), I can reference APIs introduced in iOS 13, even if I'm deploying back to iOS 12, as long as they're guarded by appropriate dynamic checks. If I compile with Xcode 10.2, however, I can't even reference those APIs—the program won't compile.

The question of whether we can invent weak linking for Android is an interesting one, but we'd have to know what to weak-link, since I assume the NDK headers don't list out when decls were introduced. They're either available or they're not. So I don't think that's the thing to pursue right now.

[John_McCall]

Here's how I think about this:

• Testing for the dynamic availability of a newer API is a common problem that certainly affects Android programmers, and IIUC there's no design intent in the Android deployment model why the existing availability features wouldn't be appropriate to extend to Android, so extending those features seems like a reasonable goal.

• For this to be useful, we need to have accurate information at build time about different versions of the SDK: we need to know when specific APIs became available, and we need to know how to link against those APIs in the oldest targeted SDK version that provides them. If this information isn't available in the headers then we'll have to layer it on, but we already have infrastructure for doing that kind of retroactive annotation via API notes, and (worst case) API notes can be generated by comparing different versions of the SDK.

• To implement the dynamic semantics, we need to be able to check the SDK available on the current system, and we need to be able to conditionally link in symbols. Both of these should be feasible:

  • The former is presumably just an OS version check, which I assume there's a standard and approved way to do; this should be easy.

  • The latter is easiest if the linker and loader support weak linking. However, that's by no means actually necessary, and we can definitely just emit global constructors that initialize our own weak-linking tables. A flat namespace isn't required for this, either, as long as we have some way of recovering what library a symbol is supposed to come from statically, which — worst case — can presumably be recovered by linking a test program.

Many of these implementation solutions would require a significant amount of new work, but that doesn't mean this is the wrong technical direction. Like all projects, this just needs to be approached with a realistic understanding of the total effort required to complete it.

[compnerd (Saleem Abdulrasool)]

@jrose - I think that is the disconnect: for other platforms, the level is the minimum level that you can target with your SDK/NDK. The desire is to have a static check to eliminate the reference at compile time as shims are not available for system functions (e.g. for functions from libc).

@John_McCall:

@John_McCall

Testing for the dynamic availability of a newer API is a common problem that certainly affects Android programmers, and IIUC there's no design intent in the Android deployment model why the existing availability features wouldn't be appropriate to extend to Android, so extending those features seems like a reasonable goal.

I don't think that the argument is that this doesn't fit into Android, its more a pragmatic issue. Apple is in a unique position of having vertical control - fragmentation in android is real, and makes it difficult to ensure that the system provides this. If it were possible to have that, I agree that would be the most desirable outcome.

@John_McCall

For this to be useful, we need to have accurate information at build time about different versions of the SDK: we need to know when specific APIs became available, and we need to know how to link against those APIs in the oldest targeted SDK version that provides them. If this information isn't available in the headers then we'll have to layer it on, but we already have infrastructure for doing that kind of retroactive annotation via API notes, and (worst case) API notes can be generated by comparing different versions of the SDK.

Unfortunately, most of the other targets do not have the ability to tightly control the system headers as Apple does. This means that we may have to support this on the Swift side. I think that the APINotes approach is absolutely viable - I'd like to get @jrose's take on that, since I may be under the incorrect assumption that he was interested in reducing the reliance on APINotes.

@John_McCall

• The former is presumably just an OS version check, which I assume there's a standard and approved way to do; this should be easy.

The dynamic checks are not a concern - there is a way to query the OS version and that can be done relatively easily. I'm happy to draft that up, its just feature parity work against the Darwin platforms.

@John_McCall

The latter is easiest if the linker and loader support weak linking. However, that's by no means actually necessary, and we can definitely just emit global constructors that initialize our own weak-linking tables. A flat namespace isn't required for this, either, as long as we have some way of recovering what library a symbol is supposed to come from statically, which — worst case — can presumably be recovered by linking a test program.

The dynamic linking is the problem - weak linking is not supported everywhere (is absolutely not available on PE/COFF). We could certainly implement this through the compiler, but I think that this will get interesting as we will need provide data for the source of the modules (this is a hard requirement since PE/COFF is strictly a 2-level namespace). I think that we could provide this information through APINotes. However, I'm not certain I agree that we could do this using the test program - cross-compilation breaks that model - you may be building on a foreign OS, architecture, or different deployment version.

Personally, I am more interested in determining the correct technical direction. I feel that if we have the correct technical direction set out, even if that means additional work, we can look to possibly finding some ways to reduce the up-front cost as long as we remain on the right technical path.

[John_McCall]

@compnerd

Apple is in a unique position of having vertical control - fragmentation in android is real, and makes it difficult to ensure that the system provides this. If it were possible to have that, I agree that would be the most desirable outcome.

If you're saying that there's substantial "lateral" fragmentation where API availability is frequently more complex than just a single linear progression of SDK versions, then I agree that we'll need a more complex language solution than what's already provided by availability. But as Jordan says, that's a problem that's interesting for Darwin as well. And I suspect that even with all of Android's fragmentation there's still a substantial core that can be reasonably formalized as a linear progression of SDKs.

@compnerd

This means that we may have to support this on the Swift side.

I don't know what I wrote that suggested that this support would not be in some way "on the Swift side".

I think something like APINotes is an inevitability for adopting third-party libraries. Maybe what we have already isn't the best solution technically, though, I won't claim to speak to that.

@compnerd

However, I'm not certain I agree that we could do this using the test program - cross-compilation breaks that model - you may be building on a foreign OS, architecture, or different deployment version.

We don't need to execute the test program; we need to inspect the namespacing decision the linker made. Of course if we can duplicate that decision ourselves, all the better.

[compnerd (Saleem Abdulrasool)]

@John_McCall

If you're saying that there's substantial "lateral" fragmentation where API availability is frequently more complex than just a single linear progression of SDK versions, then I agree that we'll need a more complex language solution than what's already provided by availability. But as Jordan says, that's a problem that's interesting for Darwin as well. And I suspect that even with all of Android's fragmentation there's still a substantial core that can be reasonably formalized as a linear progression of SDKs.

Absolutely, I think that for the vast majority of the APIs, we can distill it down to a single shared set which can provide a linear progression.

@John_McCall

I don't know what I wrote that suggested that this support would not be in some way "on the Swift side".

I think something like APINotes is an inevitability for adopting third-party libraries. Maybe what we have already isn't the best solution technically, though, I won't claim to speak to that.

Oh, I must've misunderstood something. It just seemed to me that the first paragraph was more of a modify the NDK. Sorry if thats not what you meant. Sounds like we're in violent agreement here.

@John_McCall

We don't need to execute the test program; we need to inspect the namespacing decision the linker made. Of course if we can duplicate that decision ourselves, all the better.

Clearly a lack of thought on my part. Of course we don't need execution in this case. I'm still slightly biased towards the metadata approach, but, yes, both solutions should work here.

I think that I would prefer not having to emulate the weak symbols (code size, complexity, etc). If we can come up with a way to describe the levels of the platform, it would be the ideal solution in my mind. Barring that, I agree that there are other technical approaches which are viable.

[John_McCall]

Code size shouldn’t be a problem; we could put a standard weak-linking function in the runtime that interprets some metadata, and that metadata will probably be smaller than ELF’s normal linking tables because unlike ELF we do actually care about dynamic linking.

[drodriguez (Daniel Rodríguez Troitiño)]

@jrose

since I assume the NDK headers don't list out when decls were introduced

They do have an __INTRODUCED_IN(x) set of macros which mark the symbols as available or not. My understanding is that it interacts with the passed __ANDROID_API__ and warns about symbols that might not be available.

See platform/sysroot/usr/include/android/versioning.h - platform/prebuilts/ndk - Git at Google

[Geordie_J (ephemer)]

Sorry for my lack of activity here. Some of this discussion is over my head, but I have found it interesting.

If we can implement a general solution of implementing weak linking for symbols that may or may not be available via dlsym, that would be fantastic. In the meantime, that's the solution I started working on for the issue I linked in the OP (i.e. using a – non-generic – runtime check).

For the issue at hand, that'd be preferable to any compile-time checks that may get implemented, because we don't want to have multiple builds / distributions of libFoundation.so depending on which minimum Android target devs plan to support. The tradeoff of having slightly bloated code-size there is very much worth it IMHO.

I can't comment about many of the other things that have been brought up here, other than that it'd be great to have a more generic runtime mechanism for using symbols that may or may not exist (either being given a fallback implementation or defaulting to either a runtime crash or a no-op, depending on the use-case).