Test for memory leaks in CI

Using Swift
1

Hi! I maintain a Swift library which wraps a C library. A user recently found a memory leak in our library by running their application with Xcode's leak checker.

This leak has been fixed, but ideally I'd like to try to scour the library for any others, and prevent new ones from popping up in the future. So two questions:

  1. Is there any way to run my library's entire test suite with leak detection on? I'm only seeing the option to do it for an executable.
  2. Relatedly, is there any way to do that leak detection (either via instruments or via something like valgrind) from the command line / via SwiftPM? It would be great to be able to set up a CI job for this.

Thanks!

1 Like
2

Instruments has a command line interface. See this answer on StackOverflow: c - Can Instruments be used using the command line? - Stack Overflow

The project’s test suite gets built as an executable (.build/debug/MyLibraryPackageTests.xctest), so you can run that using the instruments template.

1 Like
3

This doesn't appear to work for xctests. I can run command pointing to my compiled test suite and it asks for auth but nothing happens.

4

For SwiftNIO, we have an integration testing tool that we call the "allocation counter tests". The way they work is the following. You write a small test in the form of

import Module1YouNeedAvailableInTheTests
import Module2YouNeedAvailableInTheTests

func run(identifier: String) {
   // setup
   measure(identifier: identifier) {
       // what you want tested
   }
   // tear down
}

in a .swift file, for example our test_read_10000_chunks_from_file.swift test.

The main goal of the allocation counter tests is to count the number of allocations made during a test :slight_smile:. But as a side-effect, we also know the number of unfreed allocations, and those are the memory leaks. It measures everything that goes on in the whole program (all threads, all modules, C code, Swift code, ...) whilst the measure { ... } block is running.

The framework is pretty crude and there's not that much documentation (some is available in the debugging allocations and the general allocation counter docs.

Our CI then runs a terribly ugly script that runs all the allocation counter tests and then parses their output to verify a few things:

  • compare the number of allocations against the configured limits
  • check that we have no unfreed allocations (memory leaks)

The output of each of the allocation counter tests looks like:

test_future_lots_of_callbacks.remaining_allocations: 0
test_future_lots_of_callbacks.total_allocations: 75001
test_future_lots_of_callbacks.total_allocated_bytes: 4138056

So for each test we log

  • total_allocations (total number of allocations, usually to be divided by 1000 because we usually do 1000 iterations per test)
  • remaining_allocations: number of mallocs - number of frees = number of memory leaks
  • total_allocated_bytes: total amount of bytes allocated during this test

The framework is implemented in a mix of bash (glue and SwiftPM project generation), C (hooking malloc, free, and friends), and Swift. It's pretty crude, there's not too much documentation but it's written in a way that it can support pretty much any SwiftPM project (I'd hope). If this sounds interesting, please reach out to us, we're very happy to help with more info.

The basic usage (sorry, it's ugly) is:

"/path/to/checkout/of/swift-nio/IntegrationTests/allocation-counter-tests-framework/run-allocation-counter.sh" \
    -p "/path/to/your/package" \
    -m Module1YouNeedAvailableInTheTests \
    -m Module2YouNeedAvailableInTheTests \
    -d <( echo '.package(url: "https://some/dependency/that/you/have.git", from: "1.0.0"),' ) \
   test_my_first_alloc_counter_test.swift test_my_second_alloc_counter_test.swift

This is how NIO's HTTP/2 module runs its alloc counter tests: swift-nio-http2/run-nio-http2-alloc-counter-tests.sh at main · apple/swift-nio-http2 · GitHub .

6 Likes
5

On Linux ASAN does check for leaks. Source:

final class Klass {}

func main() {
  for _ in 0..<4096 {
    let k = Klass()
    let k2 = Unmanaged.passUnretained(k)
    k2.retain()
  }
}

main()

Output:

leak.swift:8:8: warning: result of call to 'retain()' is unused
    k2.retain()
       ^     ~~
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_extra_spill_area'
# ./leak 

=================================================================
==27==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 65520 byte(s) in 4095 object(s) allocated from:
    #0 0x56480be37c7d  (/code/leak+0x95c7d)
    #1 0x7f53dc0246b1  (/usr/lib/swift/linux/libswiftCore.so+0x3cb6b1)
    #2 0x56480be67eae  (/code/leak+0xc5eae)
    #3 0x56480be674b3  (/code/leak+0xc54b3)
    #4 0x7f53daca8b96  (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

SUMMARY: AddressSanitizer: 65520 byte(s) leaked in 4095 allocation(s).

On macOS, I would use the leaks tool from the command line. Specifically:

leaks -atExit -- $INSERT_MY_CMDLINE

It basically sets up an atexit handler that runs instruments on the process on tear down. I haven’t used it that much and would be interested in feedback if you try it.

In the runtime itself we have a simple leaks checker that we run with our benchmarks to make sure we don’t introduce leaks. It isn’t for public consumption though.

2 Likes
6

Any idea how you would run this against tests? As stated it seems like it would only work against an executable. The produced xctest directory looks like a framework but there doesn't seem to be a way to run it directly.

7

I don't know. That being said, I was actually trying to answer question (2), not (1). That is I was answering specifically how to test from the command line, not how to do it for a non-executable.

8

Relatedly, is there any way to do that leak detection … from the
command line / via SwiftPM?

On macOS there’s a leaks command line tool. See its man page for details. This uses the same core that powers the Leaks instrument.

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

1 Like
9

It does indeed, and so does valgrind and for developing I think it's great. However, both ASan and valgrind aren't suitable for CI because they will give you many false positives.
For example, they will often report that Strings or anything else that contains custom bit packing or an enum with an associated value of a reference type as leaked. The problem seems to be that ASan/valgrind check for a reference by trying to find the exact pointer value in memory but if you use the spare bits in the pointer representation to store other information that doesn't work anymore.
Whilst in development, that's totally cool because you can manually filter those out but in CI you need something that correctly identifies if there are leaks or not. We found the malloc/free counting much more reliable and therefore use that in CI and the Instruments/leaks/ASan/valgrind/dtrace to debug what is leaked after the CI tells us that something is leaked.

2 Likes
10

Good point. I was blindly assuming Linux but if the CI is on macOS, leaks can indeed be used :slight_smile:

11

How can leaks be used in CI? I could see how an executable could work, by compiling a version with stack logging, then running it separately, but how leaks work for testing?

12

Like @Michael_Gottesman proposed, you could register an atexit handler in your unit tests that then runs leaks on your own pid (assuming XCTest infrastructure doesn't leak)?

13

How would that work? Wouldn't the atExit be attached to SPM that's running my tests? Otherwise how do you run tests independently?

14

The tests are run in a separate binary, on Darwin it's xctest and on Linux it's just the .build/debug/my-packageTests.xctest binary. SwiftPM just spawns those binaries, so atexit should run in the right process.

15

I wasn't able to get this working with Xcode's compiled xctest bundle, but was able to run the SPM build version with:

leaks --atExit -- xcrun xctest /Users/jshier/Desktop/Code/Alamofire/.build/x86_64-apple-macosx/debug/AlamofirePackageTests.xctest

However, this doesn't produce any leaks output, just test run output. Does that mean there were no leaks?

16

I meant using the atexit function in the program to then run leaks on getpid() :slight_smile:

17

So calling atexit and launching leaks \(getpid()) does work, but wow, that's pretty bad.

It turns out that instruments -t Leaks <path to xctest> ... also doesn't work—using @Michael_Gottesman's example above, it tracks the allocations but it doesn't consider any of them as leaks.

Even though the test code is in a separate bundle-type binary, the tests run in a single process—the xctest tool just loads the bundle and then calls into it through the Obj-C runtime. So I'm curious, what's going on that causes neither leaks --atExit -- xcrun xctest ... nor instruments to consider these leaks? Is it something that just doesn't work if the calling process unloads the bundle that made the allocations first?

18

Could you post an example? Where would I add the handler for it to work in a test target?

19

Please don't beat me up for the ugliness of this program but it demonstrates all the bits you'll need I think:

import Darwin
import Foundation

class C {
    var c: C? = nil
}

// leak some
for _ in 0..<10 {
    let f = C()
    f.c = f
    print(f)
}

atexit {
    @discardableResult
    func leaksTo(_ file: String) -> Process {
        let out = FileHandle(forWritingAtPath: file)!
        defer {
            try! out.close()
        }
        let p = Process()
        p.launchPath = "/usr/bin/leaks"
        p.arguments = [ "\(getpid())" ]
        p.standardOutput = out
        p.standardError = out
        p.launch()
        p.waitUntilExit()
        return p
    }
    let p = leaksTo("/dev/null")
    guard p.terminationReason == .exit && [0, 1].contains(p.terminationStatus) else {
        print("Weird, \(p.terminationReason): \(p.terminationStatus)")
        exit(255)
    }
    if p.terminationStatus == 1 {
        print("================")
        print("Oh no, we leaked")
        leaksTo("/dev/tty")
    }
    exit(p.terminationStatus)
}

Example output:

test.C
test.C
test.C
test.C
test.C
test.C
test.C
test.C
test.C
test.C
================
Oh no, we leaked
Process:         test [28074]
Path:            /private/tmp/test
Load Address:    0x1020bb000
Identifier:      test
Version:         ???
Code Type:       X86-64
Parent Process:  bash [21589]

Date/Time:       2020-05-18 22:22:07.277 +0100
Launch Time:     2020-05-18 22:22:05.598 +0100
OS Version:      Mac OS X 10.15.4 (19E287)
Report Version:  7
Analysis Tool:   /Applications/Xcode.app/Contents/Developer/usr/bin/leaks
Analysis Tool Version:  Xcode 11.4.1 (11E503a)
----

leaks Report Version: 4.0
Process 28074: 1349 nodes malloced for 160 KB
Process 28074: 10 leaks for 320 total leaked bytes.

    10 (320 bytes) << TOTAL >>
      1 (32 bytes) ROOT LEAK: <C 0x7fde2740d650> [32]
      1 (32 bytes) ROOT LEAK: <C 0x7fde27604460> [32]
      1 (32 bytes) ROOT LEAK: <C 0x7fde27604620> [32]
      1 (32 bytes) ROOT LEAK: <C 0x7fde27604740> [32]
      1 (32 bytes) ROOT LEAK: <C 0x7fde276053c0> [32]
      1 (32 bytes) ROOT LEAK: <C 0x7fde276053e0> [32]
      1 (32 bytes) ROOT LEAK: <C 0x7fde276054e0> [32]
      1 (32 bytes) ROOT LEAK: <C 0x7fde27605500> [32]
      1 (32 bytes) ROOT LEAK: <C 0x7fde27605520> [32]
      1 (32 bytes) ROOT LEAK: <C 0x7fde27605540> [32]
2 Likes
20

Thanks. I put this in a random test case and it seems to work. But either Alamofire is extremely leaky (16MB over an SPM test run), or tests aren't cleaned up like normal code execution.