Dladdr() and the clang importer

IanPartridge · June 27, 2019, 3:56pm

I'm trying to call dladdr() from Swift on Linux.

Related functions from dlfcn.h are available in the Glibc module, such as dlopen() and dlsym(), however dladdr() is not.

The reason for the omission is because dladdr() requires the _GNU_SOURCE macro to be defined, and the Clang importer does not define it on Linux.

_GNU_SOURCE is defined on Fuschia and Android though, and there is a comment in the importer suggesting why it's not enabled on Linux in general:

// Ideally we should turn this on for all Glibc targets that are actually
// using Glibc or a libc that respects that flag. This will cause some
// source breakage however (specifically with strerror_r()) on Linux
// without a workaround.

My question is how we could turn on _GNU_SOURCE on Linux, so I can call dladdr().

The source breakage concern around strerror_r() is real, for example this Swift package would break if Glibc vended a version of strerror_r() with a different signature. There are other examples to be found on GitHub.

The comment mentions a "workaround" but I can't envisage what that could be. Does anyone have any ideas?

Also, is strerror_r() the only function with source compatibility issues around _GNU_SOURCE or are there others to consider too?

Thanks.

IanPartridge · June 27, 2019, 4:17pm

Aside: I do have the workaround of building with swift build -Xcc -D_GNU_SOURCE=1 but it is not reasonable to ask package users to do that.

jrose · June 27, 2019, 4:51pm

One workaround would be to find all functions that behave differently under _GNU_SOURCE and overload or shadow them in the Platform overlay, so that we can flip the switch without breaking source compatibility.

We could also just say swiftc should take the hit and define _GNU_SOURCE all the time on Linux, source compat notwithstanding. My concern with that is that _GNU_SOURCE isn't the only such interesting macro. (For instance, LLVM headers used to need __STDC_LIMIT_MACROS and __STDC_CONSTANT_MACROS to be defined.) What's the policy on what's on by default? What happens if two different packages disagree on what they need?

compnerd · June 27, 2019, 6:57pm

__STDC_LIMIT_MACROS and __STDC_CONSTANT_MACROS are more interesting because they are standards defined. C99 states that certain macros should only be available if they are defined. Fun thing is that C++11 decided that the behaviour was not desirable and explicitly overruled the behaviour of C99. Subsequently, C11 deprecates these macros. So, now you have this fun interaction of:

C++03/C99 -> define __STDC_LIMIT_MACROS, __STDC_CONSTANT_MACROS
C++11/C11 -> don't define __STDC_LIMIT_MACROS, __STDC_CONSTANT_MACROS

This is in fact part of the problem that has caused a serious problem for C++ usage: C++11 requires wrappers for stdint.h to define these macros around the inclusion of the cstdint. Given that the clang importer uses -std=gnu11 (where as I would prefer -std=c11, but I digress), I don't think that we need to worry about the STDC macros.

_GNU_SOURCE is actually far more tricky as it permits non-portable, behaviour altering paths. It changes the behaviour of POSIX defined defined functions within libc even! I think that the better thing to do is create a baseline for POSIX conformance. If POSIX 2017 (i.e. IEEE 1003.1-2017 - released July 22nd, 2018 is too new, we could say that the baseline is 2008. This can be specified by _POSIX_C_SOURCE=200809l. We could also enable conforming extensions via _XOPEN_SOURCE=700 which enables the X/OPEN 7 extensions from POSIX.1-2008.

Also note that with glibc, _GNU_SOURCE is particularly bad because it changes behaviour according to the version of the library. Furthermore, the version of the library is relevant as the behaviour of _GNU_SOURCE changed across versions. 2.19+ has it imply _DEFAULT_SOURCE, _BSD_SOURCE and _SVID_SOURCE. The current behaviour in musl is different and I believe for the most part it treats _BSD_SOURCE and _GNU_SOURCE synonymously where as glibc does not always do that.

Shadowing the declarations in the platform overlay seems much more reasonable. This is part of the motivation that I have to rename Glibc to glibc and introduce bionic and musl so that we can actually properly handle the behaviour of the library. (That is also ignoring the details of uclibc, newlib, and dietlibc).

IanPartridge · June 28, 2019, 9:33am

Thanks for your replies.

My goal is to call dladdr() from Swift on Linux, and as far as I know, only _GNU_SOURCE allows this. No incantation of _POSIX_C_SOURCE or _XOPEN_SOURCE will work.

I'm interested in pursuing this. Could we just accept some breakage of strerror_r() on Linux?

Here is the list of .swift files on GitHub which use strerror_r() - there are 20: https://github.com/search?utf8=✓&q=strerror_r+extension%3Aswift&type=Code&ref=advsearch&l=&l=

Most of these are Darwin only so would not be affected by any change.

The only ones I can see that would be affected are:

github.com

mxcl/swift-sh/blob/a0fb8cf83e46ce7c33eb149a0c8cbfb2092e8410/Sources/Utility/etc.swift#L26


      
          #else
          import func Darwin.strerror_r
          import var Darwin.EINVAL
          import var Darwin.ERANGE
          #endif
          
          public func strerror(_ code: Int32) -> String {
              var cap = 64
              while cap <= 16 * 1024 {
                  var buf = [Int8](repeating: 0, count: cap)
                  let err = strerror_r(code, &buf, buf.count)
                  if err == EINVAL {
                      return "unknown error \(code)"
                  }
                  if err == ERANGE {
                      cap *= 2
                      continue
                  }
                  if err != 0 {
                      return "fatal: strerror_r: \(err)"
                  }

github.com

michael-yuji/CKit/blob/ada1ff7dc0062c3b57f8e035ad0a39c8da043b9d/Sources/Error.swift#L9


      
          
          public struct SystemError: Error, CustomStringConvertible {
              public var errno: Int32
              public var umsg: String?
          
              public var description: String
              {
                  var buf = [CChar](repeating: 0, count: 128)
                  _ = xlibc.strerror_r(errno, &buf, 128)
                  return "\(umsg ?? ""): " + String(cString: buf)
              }
              
              public static func last(_ umsg: String?) -> SystemError
              {
                  return SystemError(errno: xlibc.errno, umsg: umsg)
              }
              
              public static func lastErrorString() -> String
              {

https://github.com/apple/swift-package-manager/blob/master/Sources/Basic/misc.swift#L197

The last one is the package manager which we control (and already has a different path here for Windows) so is not relevant. That leaves 2 packages, one of which does not build on Swift 5 anyway (it has a Swift 3 package manifest)... not exactly a lot.

norio_nomura · June 28, 2019, 10:48am

I avoided the issue by writing a function using dladdr in the Clang target and using it from Swift.

github.com

norio-nomura/SwiftBacktrace/blob/1b4bbe87c19d5e2e04786d8140ab36fec4b7881b/Sources/CSwiftBacktrace/CSwiftBacktrace.cpp#L13-L16


      
          const char* _Nullable dli_fname(const void* _Null_unspecified addr) {
              Dl_info info;
              return dladdr(addr, &info) ? info.dli_fname : NULL;
          }

This issue have been filed at: [SR-3250] Glibc does not have `dladdr()`. · Issue #45838 · apple/swift · GitHub

lukasa · June 28, 2019, 12:33pm

In SwiftNIO we work around this by restricting our use of _GNU_SOURCE to C-level shim files that re-expose those functions to our Swift code: swift-nio/shim.c at main · apple/swift-nio · GitHub

jrose · June 28, 2019, 3:22pm

Hm, the fact that Darwin has the POSIX strerror_r makes me less inclined to support the GNU one by default. Even if it's better.

IanPartridge · June 28, 2019, 3:24pm

Thanks @norio_nomura and @lukasa for the advice about just doing C/C++ instead and calling dladdr() from there. I got this working, so my immediate problem is solved

norio_nomura · June 29, 2019, 12:21am

As far as I have tried, using .cpp instead of .c as an extension eliminates the need to define _GNU_SOURCE by ourselves.

nerdsupremacist · March 26, 2020, 11:07am

this worked for me too

github.com

nerdsupremacist/Runtime/blob/master/Sources/CSymbols/symbols.c


#ifdef __linux__
#define _GNU_SOURCE
#endif

#include <CSymbols.h>
#include <stddef.h>
#include <dlfcn.h>
#include <stdint.h>

void loadSymbol(const void *address, symbolInfo *symbol) {
    Dl_info info;
    dladdr(address, &info);
    uintptr_t pointerValue = (uintptr_t)info.dli_sname;
    if (info.dli_sname && pointerValue > 0xff) {
        symbol->name = info.dli_sname;
    }
    if (info.dli_saddr) {
        symbol->address = info.dli_saddr;
    }

This file has been truncated. show original

However if you're using dladdr from linux and want to get any results you have to make sure the symbols end up in the dynamic symbols table. You can force it to export all symbols with the -E Linker flag...

So run:
swift build -Xlinker -E -c debug