Crash backtraces

Server Pitches
19

Hi all, here's an update on my progress with backtraces.

I started off by looking at the various options we have:

  1. backtrace() for unwinding and backtrace_symbols() for symbolication. This is the approach currently used in GitHub - swift-server/swift-backtrace: 💥 Backtraces for Swift on Linux and Windows and while it does successfully unwind it does not fully symbolicate, because backtrace_symbols() does not read DWARF debug info. addr2line can be used post-mortem to symbolicate.

  2. Then I thought "Well if addr2line is symbolicating, can't we just do whatever it is doing internally?" Yes we can, and https://oroboro.com/printing-stack-traces-file-line/ explains nicely how to do it. Basically you can use libbfd which is part of GNU binutils. The problem with this is that libbfd is GPL licensed, which makes it unusable for SSWG purposes.

  3. Next I looked at libunwind. This is liberally licensed and can symbolicate from DWARF info so could be a good option. GitHub - norio-nomura/SwiftBacktrace: Stack traces for Swift on Mac and Linux using `libunwind`. has proven that it can work. The only downside is that it requires installing the libunwind8 package. It would be nice if we had a solution that worked out-of-the-box.

  4. Next I looked at libdw. This is part of elfutils and again looks like it ticks the boxes. It is used successfully by the Haskell runtime for stacktraces and symbolication, and is also an unwind/symbolicate backend for Linux perf. It doesn't look easy to work with though from a quick look at the API, and the licensing is LGPLv3+ which may be problematic. It may well be worth investing more time looking at it though.

  5. Next I looked at eu-stack. This is part of elfutils again and is a command-line tool that can print stacktraces of any process. It uses libdw internally. We could run it by catching SIGILL, calling fork() then execve()ing eu-stack and capturing the output. I'm not really a fan of these out of process options though, and it would require users to install the elfutils package.

  6. libcwd - seems aimed at C++ and I couldn't spot easy to use API. Didn't spend much time looking at this one.

  7. Lastly I looked at libbacktrace. It's used as GCC's unwinder and is designed to run in and out of process. It's liberally licensed. It has a very simple (read: idiot-proof) API and is not much code in and of itself.

I decided to try and get libbacktrace working. I vendored it into an SPM target (many hacks were done, as a PoC) and used its backtrace_print() API. Here's my first results:

This Swift program (main.swift):

import libbacktrace

@inline(never)
func f1() {
    f2()
}

@inline(never)
func f2() {
    f3()
}

@inline(never)
func f3() {
    bt(CommandLine.arguments[0]) // call shim into vendored libbacktrace
}

f1()

outputs:

0x55b62337ca7a bt
	/root/src/libbacktrace/Sources/libbacktrace/include/backtrace.h:189
0x55b62337c936 $s4test2f3yyF
	/root/src/libbacktrace/Sources/test/main.swift:17
0x55b62337c8b8 $s4test2f2yyF
	/root/src/libbacktrace/Sources/test/main.swift:12
0x55b62337c8a8 $s4test2f1yyF
	/root/src/libbacktrace/Sources/test/main.swift:7
0x55b62337c7ba main
	/root/src/libbacktrace/Sources/test/main.swift:20
0x7f2fe6278b96 ???
	???:0
0x55b62336e629 ???
	???:0
0xffffffffffffffff ???
	???:0

So it works!

I think this could be a viable backend for our backtrace library. It requires no additional system packages to be installed, it is quick to build (just 13 C files), it can be vendored inside an SPM package, and it seems to give good stacktraces including function names, source files and line numbers.

I was using its simplest backtrace_print() API - there are others which give more control and would enable us to feed the symbolicated function names to the Swift demangler. Together it could be quite nice.

What do people think?

11 Likes