Debugging crashes in deployed swift apps

Server Development
2

I'm glad you ask this here! I have absolutely not much time right now, so I'll give you a braindump of what should work.

So first of all, Swift applications should run in release mode (as you do) because otherwise they're super slow. And btw, in debug mode you also don't always get stack traces, for example on overflow.

Ok, how do we get stacktraces then?

Step 1: Debug Info

We need debug information in the binary, to get that we should compile with

swift build -c release -Xswiftc -g

done.

Step 2: Let's print stack traces

To do that, we need to do something dodgy, we'll install a signal handler that will print us the backtrace :slight_smile:. How do we do that? Well, first, we'll need access to the backtrace and backtrace_symbols functions from glibc.

Step 2.1 Getting access to backtrace and backtrace_symbols

The easiest is to create a new SwiftPM target with the following source layout:

Sources/CBacktrace
Sources/CBacktrace/c_backtrace.c  # this file can be empty 
Sources/CBacktrace/include/CBacktrace.h

So the .c file can be empty, and CBacktrace.h just needs to contain:

#include <execinfo.h>

Step 2.2

In Package.swift, just create the target and depend on it as usually. In your main.swift do import CBacktrace

Step 3 Let's make the stack traces printed on SIGILL

Okay, this is murky, but there you go. If you put the following code in your main.swift you should be getting stack traces in release mode.

import Glibc
import CBacktrace

private func setupHandler(signal: Int32, handler: @escaping @convention(c) (CInt) -> Void) {
    typealias sigaction_t = sigaction
    let sa_flags = CInt(SA_NODEFER) | CInt(bitPattern: CUnsignedInt(SA_RESETHAND))
    var sa = sigaction_t(__sigaction_handler: unsafeBitCast(handler, to: sigaction.__Unnamed_union___sigaction_handler.self),
						 sa_mask: sigset_t(),
						 sa_flags: sa_flags,
						 sa_restorer: nil)
    withUnsafePointer(to: &sa) { ptr -> Void in
        sigaction(signal, ptr, nil)
    }
}

setupHandler(signal: SIGILL) { _ in
	// this is all undefined behaviour, not allowed to malloc or call backtrace here...

	let maxFrames = 50
	let stackSymbols: UnsafeMutableBufferPointer<UnsafeMutableRawPointer?> = .allocate(capacity: maxFrames)
	stackSymbols.initialize(repeating: nil)
	let howMany = backtrace(stackSymbols.baseAddress!, CInt(maxFrames))
	let ptr = backtrace_symbols(stackSymbols.baseAddress!, howMany)
	let realAddresses = Array(UnsafeBufferPointer(start: ptr, count: Int(howMany))).compactMap { $0 }
	realAddresses.forEach {
		print(String(cString: $0))
	}
}

So as I said, this is very undefined behaviour but we don't really mind as we're crashing anyway. What we're doing here is installing a one-off signal handler for SIGILL, that's the signal that will be triggered when Swift crashes. And in that signal handler, we call backtrace and backtrace_symbols and print the result.

Now, when our binary crashes, we should see something like:

root@f56a02308177:/tmp/crashtest# swift run -c release -Xswiftc -g
Fatal error: file /tmp/crashtest/Sources/crashtest/main.swift, line 33
.build/x86_64-unknown-linux/release/crashtest(+0x1a71) [0x55dac5f2ea71]
.build/x86_64-unknown-linux/release/crashtest(+0x1429) [0x55dac5f2e429]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7ff45b033890]
/usr/lib/swift/linux/libswiftCore.so(+0x321aba) [0x7ff45aa1faba]
/usr/lib/swift/linux/libswiftCore.so(+0x1485e9) [0x7ff45a8465e9]
.build/x86_64-unknown-linux/release/crashtest(+0x14b8) [0x55dac5f2e4b8]
.build/x86_64-unknown-linux/release/crashtest(+0x1445) [0x55dac5f2e445]
.build/x86_64-unknown-linux/release/crashtest(+0x1480) [0x55dac5f2e480]
.build/x86_64-unknown-linux/release/crashtest(+0x1465) [0x55dac5f2e465]
.build/x86_64-unknown-linux/release/crashtest(+0x140b) [0x55dac5f2e40b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7ff4599efb97]
.build/x86_64-unknown-linux/release/crashtest(+0x115a) [0x55dac5f2e15a]
Illegal instruction

Already much better, looks like a stacktrace!

Step 4

If we copy the stack trace output, so basically this part

.build/x86_64-unknown-linux/release/crashtest(+0x1a71) [0x55dac5f2ea71]
.build/x86_64-unknown-linux/release/crashtest(+0x1429) [0x55dac5f2e429]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7ff45b033890]
/usr/lib/swift/linux/libswiftCore.so(+0x321aba) [0x7ff45aa1faba]
/usr/lib/swift/linux/libswiftCore.so(+0x1485e9) [0x7ff45a8465e9]
.build/x86_64-unknown-linux/release/crashtest(+0x14b8) [0x55dac5f2e4b8]
.build/x86_64-unknown-linux/release/crashtest(+0x1445) [0x55dac5f2e445]
.build/x86_64-unknown-linux/release/crashtest(+0x1480) [0x55dac5f2e480]
.build/x86_64-unknown-linux/release/crashtest(+0x1465) [0x55dac5f2e465]
.build/x86_64-unknown-linux/release/crashtest(+0x140b) [0x55dac5f2e40b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7ff4599efb97]
.build/x86_64-unknown-linux/release/crashtest(+0x115a) [0x55dac5f2e15a]
Illegal instruction

into a file called /tmp/stacktrace we can run this beautiful bash script:

cat /tmp/stacktrace | tr '()' '  ' | while read bin addr junk; do addr2line -e "$bin" -a "$addr" -ipf; done | swift demangle

aaand

0x0000000000001a71: function signature specialization <Arg[0] = Dead> of closure #1 (Swift.Int32) -> () in crashtest at /tmp/crashtest/Sources/crashtest/main.swift:22
0x0000000000001429: @objc closure #1 (Swift.Int32) -> () in crashtest at /tmp/crashtest/<compiler-generated>:?
0x0000000000012890: __restore_rt at ??:?
0x0000000000321aba: function signature specialization <Arg[0] = Exploded, Arg[1] = Exploded> of Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never at crtstuff.c:?
0x00000000001485e9: Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never at ??:?
0x00000000000014b8: merged crashtest.recurse2(n: Swift.Int) -> () at main.swift.o:?
0x0000000000001445: crashtest.recurse2(n: Swift.Int) -> () at ??:?
0x0000000000001480: merged crashtest.recurse2(n: Swift.Int) -> () at main.swift.o:?
0x0000000000001465: crashtest.recurse(n: Swift.Int) -> () at ??:?
0x000000000000140b: main at /tmp/crashtest/Sources/crashtest/main.swift:48
0x0000000000000000: ?? ??:0
0x000000000000115a: _start at ??:?

which is basically what we wanted, wohoo! I don't exactly know why addr2line (which is in the binutils package) can't find the file/line information but it's probably a hell of a lot better than no stacktraces at all :).

Appendix

Should this be so hard? NO, this should be fixed and high-quality stack traces should be printed.

Can we get higher quality stack traces? Yes! With the help of lldb, the symbolicate-linux-fatal script in the Swift repository can get you really high-quality stack traces. The problem: Right now it tries to parse the output that Swift programs only output in debug mode. But it would be not too hard to combine steps 1 to 3 with symbolicate-linux-fatal and make it be able to produce those stack traces from the output from step 3.

Sorry for typos etc, I need to run...

6 Likes