How to profile heap allocations? (valgrind not working with _dispatch_sigsuspend)

Server Development
1

how do you profile heap allocations from a swift binary with massif and valgrind? i’m running into the “the 'impossible' happened” error:

==19637== Massif, a heap profiler
==19637== Copyright (C) 2003-2017, and GNU GPL'd, by Nicholas Nethercote
==19637== Using Valgrind-3.19.0.GIT and LibVEX; rerun with -h for copyright info
==19637== Command: .build/release/swiftinit
==19637== 
--19637-- VALGRIND INTERNAL ERROR: Valgrind received a signal 11 (SIGSEGV) - exiting
--19637-- si_code=2;  Faulting address: 0x66954A8;  sp: 0x100426dd68

valgrind: the 'impossible' happened:
   Killed by fatal signal

host stacktrace:
==19637==    at 0x5801D327: vgPlain_sigdelset (m_libcsignal.c:159)
==19637==    by 0x5806B47E: vgPlain_client_syscall (syswrap-main.c:2234)
==19637==    by 0x5806834D: handle_syscall (scheduler.c:1211)
==19637==    by 0x58069BE0: vgPlain_scheduler (scheduler.c:1529)
==19637==    by 0x580C8E76: thread_wrapper (syswrap-linux.c:101)
==19637==    by 0x580C8E76: run_a_thread_NORETURN (syswrap-linux.c:154)

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable syscall 130 (lwpid 19637)
==19637==    at 0x8C93FEE: sigsuspend (in /usr/lib64/libc-2.26.so)
==19637==    by 0x668BD87: _dispatch_sigsuspend (in /home/ec2-user/.swiftenv/versions/DEVELOPMENT-SNAPSHOT-2022-02-03-a/usr/lib/swift/linux/libdispatch.so)
==19637==    by 0x6689015: _dispatch_sig_thread (in /home/ec2-user/.swiftenv/versions/DEVELOPMENT-SNAPSHOT-2022-02-03-a/usr/lib/swift/linux/libdispatch.so)
==19637==    by 0x578E62F: __nptl_deallocate_tsd.part.5 (in /usr/lib64/libpthread-2.26.so)
==19637==    by 0x8C81156: (below main) (in /usr/lib64/libc-2.26.so)
client stack range: [0x1FFEFFE000 0x1FFF000FFF] client SP: 0x1FFF000A90
valgrind stack range: [0x100416E000 0x100426DFFF] top usage: 7568 of 1048576

Thread 2: status = VgTs_Init (lwpid 0)
==19637==    at 0x8D4D401: clone (in /usr/lib64/libc-2.26.so)
==19637==    by 0x578F36F: ??? (in /usr/lib64/libpthread-2.26.so)
==19637==    by 0xCA1B6FF: ???
client stack range: [0xC21C000 0xCA1AFFF] client SP: 0xCA1AEB0
valgrind stack range: [0x100747C000 0x100757BFFF] top usage: 312 of 1048576

has anyone seen this before?

2

You might want to try heaptrack- hints here:

2 Likes
3

And of course more about heaptrack here:

I found it working quite well and definitely liked the ability to diff multiple runs easily - awesome to track regressions. Also liked the ability to analyze transient allocations separately from leaks easily. Unlike valgrind it worked for me with reasonable performance on Ubuntu 20. Hope it helps.

3 Likes
4

unfortunately, i am not getting any useful information out of heaptrack at all, as it’s attributing everything to swift_slowAlloc:

swift_slowAlloc
  in .swiftenv/versions/DEVELOPMENT-SNAPSHOT-2022-02-03-a/usr/lib/swift/linux/libswiftCore.so
5

Release mode built or?

6

This sounds like heaptrack itself (or maybe it's just not compiled with DWARF support) doesn't do the stack unwinding using the DWARF info.
If you unwind Swift stacks purely with the frame pointer, then that can easily happen unfortunately. That's why in perf you get pretty bad results with just -g (or --call-graph fp) but really good ones with --call-graph dwarf,[...].

I think getting to a place where heaptrack can be made to work reliably for Swift would really be awesome and it'd be even more awesome to write it up as a guide in the SSWG's guides repo.

In case you can't get heaptrack or similar tools to work or you're in a bind and need to figure out a problem right now, then the following workarounds/tricks may help:

If you're purely after looking where (and how many) allocations happen, then you can use the SSWG's allocations debugging guide which uses just perf to achieve that job. It will be able to tell you where allocations happened but not how many un-freed allocs there are. You could however perform that job yourself by instrumenting both malloc (and friends) and free (and friends) and then manually tallying up (likely it won't be perfect because realloc can be used to allocate, re-allocate and free memory. So purely from the knowledge that realloc was called you can't deduce much. Hopefully that won't be your problem because apart from NIO's ByteBuffer I haven't seen anything in Swift-land that uses realloc. And because ByteBuffer is a value type it can't leak on its own, if leaked, there's always something else that would leak too so you should be able to find it)

FWIW, SwiftNIO uses its allocation counter tests to write tests that check that no leaks occurred as well as that the number of allocations didn't increase. Sadly, I never got to properly productise it, the bits can be found here and there.

2 Likes
7

I had it working successfully when troubleshooting io_uring for SwiftNIO (Ubuntu 20.04/20.10 IIRC)- thus the section quoted above on usage (which did get proper stacks) - heaptrack does/should support DWARF.

@taylorswift which OS are you trying on? How did you install it?

sudo apt-get install heaptrack

or something else?

Unfortunately I don't have a Linux setup available right now and the bandwidth to test at the moment, but agree it would be nice to document it properly - will try to circle back later if no one else gets there first.

2 Likes