Swift 5.x and Coroutines

robertjpayne · 2020-10-23T05:00:12.216Z

I managed to strip this back and discover this bug has nothing to do with coroutines at all. It appears on Swift Linux there's some ARC/memory issues when using while loops that run indefinitely.

On Darwin we can use a autoreleasepool to force releasing the memory but on Linux there doesn't seem to be anything we can use to force that?

I stripped back our project to the bare minimums using threads and blocking I/O with as little of code as possible.

Reproduction steps (Ubuntu 18.04 server)

Compile swift build --configuration=release
Run ./runner.sh
In another terminal on the server run ps aux | grep server and take note of memory usage
On another machine use wrk to benchmark the endpoint wrk -c 10 -t 1 -d 10s http://<ip address>:3000
Wait until wrk finishes and then a little bit of time, check logs to ensure all tcp connections are closed
Check memory usage again using ps aux | grep server

Expected results:

Memory usage should be similar to when starting

Actual results:

Memory usage is much higher it peaks based on number of concurrent connections (the -c 10 flag in wrk) ever connected. Sometimes it will reduce back but most times it'll stay at a peak.

In my observations if you re-run wrk the peak usage doesn't change if the connection limit remains. By triggering the loops again memory is reclaimed.

I don't believe this is strictly related to Data objects. In our actual server code all of our reference type semantic objects were appropriately deinit'd but value types seemed to be causing the growth.

You can download the sample code here:

Dropbox

Dropbox - File Deleted

Dropbox is a free service that lets you bring your photos, docs, and videos anywhere and share them easily. Never email yourself a file again!

lukasa · 2020-10-23T08:19:52.445Z

It would be very valuable to try to identify whether the objects in question are leaked or whether the memory was used by the process and never released back to the OS. Memory fragmentation limits the ability of the allocator to return memory to the OS, and so in some cases you will see memory usage behave this way: it will jump up under load, and then settle somewhat but your process will appear to still be consuming lots of memory. Internally the allocator has plenty of space to allocate, but it's fragmented and so the allocator cannot return pages to the OS.

Tools like valgrind or the address sanitiser should be capable of detecting true leaks. It may be worth running those tools to get a better idea of what's going on.

robertjpayne · 2020-10-23T08:32:22.965Z

Hi Lukasa,

As far as I can tell it's Swift structures not being released when it is expected in code. If you switch Data out for a UnsafeMutableRawBufferPointer with a deferred deallocation you see no memory growth.

Again what happens is because we have an endless loop it's like until the loop "breaks" for the next request the memory used in the previous request isn't deallocated. It's not permanently lost but it does mean if we get a burst of high-load traffic we get a crashed process due to this "phantom" memory being in play.

The only thing Valgrind reports on is a lot of 8 byte unsafe access around swift_retain and swift_release calls.

I can't seem to find any way to "fool" the compiler into properly release the Data object back to the OS and unfortunately I don't think it's strictly related to Data internals as in our production app we see this growth happening after our JSON parsing which results in a large amount of enum/dictionary/array objects being allocated.

Is there anything else I can do to try and inspect where the Swift compiler is putting in retain/release calls?

lukasa · 2020-10-23T08:36:23.040Z

robertjpayne:

Is there anything else I can do to try and inspect where the Swift compiler is putting in retain/release calls?

Sure, you can disassemble the binary, or you can hook the function calls.

robertjpayne:

As far as I can tell it's Swift structures not being released when it is expected in code. If you switch Data out for a UnsafeMutableRawBufferPointer with a deferred deallocation you see no memory growth.

What do you mean by "a deferred deallocation"? Are you slicing Data objects and storing them anywhere?

robertjpayne · 2020-10-23T08:40:44.872Z

I'm not the greatest with a disassembler but I'll give it a code.

This code shows the memory growth that isn't released back until next iteration of the while loop:

while true {
   var request = socket_read_request() // blocking IO
   var body: Data! = try! Data(contentsOf: URL(<10 mb file>))
   socket_write_data(body) // blocking IO
   body = nil
}

At the point the loop is waiting for socket IO for the next request you'd expect the body from the previous request to be deallocated but that is not the case.

However if you use a raw pointer with deferred deallocation:

while true {
   var request = socket_read_request() // blocking IO
   let body = UnsafeMutableRawBufferPointer.allocate(capacity: 10*1024*1024, alignment: MemoryLayout<UInt8>.alignment)
  defer { body.deallocate() }
   socket_write_data(body) // blocking IO
}

The memory is freed immediately before the process idles waiting for the next request.

lukasa · 2020-10-23T08:42:13.562Z

Can you try allocating a large repeated empty data, something like Data(repeating: 0, count: 10 * 1024 * 1024)? Data(contentsOf:) is a huge, complex method that does an enormous number of allocations, and it's quite unlike allocating and freeing a pointer.

robertjpayne · 2020-10-23T08:47:37.257Z

Same issue using Data(repeating: 0, count: 10*1024*1024)

lukasa · 2020-10-23T08:49:01.278Z

That's very surprising. I'd be very curious if socket_write_data is storing a slice of the Data anywhere.

robertjpayne · 2020-10-23T08:51:16.086Z

You can check the sample code I put in, it's extremely small (4 files) as I tried to extrapolate a working example to it's most simple parts.

The Data object is used with withUnsafeBytes to actually write to the raw socket using C api's

robertjpayne · 2020-10-23T08:57:57.607Z

@lukasa Actually managed to simplify this even further:

func main() throws {
    let semaphore = DispatchSemaphore(value: 0)
    for _ in 0..<100 {
        Thread.detachNewThread {
            var data: Data! = Data(repeating: 0, count: 10*1024*1024)
//            let data = UnsafeMutableRawBufferPointer.allocate(byteCount: 10*1024*1024, alignment: MemoryLayout<UInt8>.alignment)
            print("data count: \(data.count)")
            data = nil
//            data.deallocate()
            semaphore.signal()
            Thread.sleep(forTimeInterval: 600)
        }
        semaphore.wait()
    }

    print("Idle now...")
    Thread.sleep(forTimeInterval: 40)
    print("Exiting")
}
try! main()

If you swap between Data and the UnsafeRawBufferPointer allocation methods you can see the memory usage during the processes idle phase.

Raw pointers in ps aux show no memory usage outside of baseline, while Data continues to show several hundred megabytes.

lukasa · 2020-10-23T08:58:44.264Z

And to confirm this is running on Linux, right? I just want to be absolutely sure we're on the same page.

robertjpayne · 2020-10-23T08:59:09.085Z

Yup on Linux -- Ubuntu 18.04 server with Swift 5.3 toolchain

robertjpayne · 2020-10-23T09:06:21.275Z

It's weird because Valgrind states nothing is amiss at all, but ps aux shows a lot of ram usage still.

For us since we use Docker ps aux or whatever underlying OS methods calculate ps aux is what Docker uses to determine if a container is using more ram than it should and kill it.

I might try and test with creating a few hundred thousand structs instead of Data and see if any difference.

robertjpayne · 2020-10-23T09:11:04.156Z

Yea so using normal structs instead of Data can cause it as well.

Modified the main.swift file a bit with commented out bits for the 3 types of test runs and reduced timers so it can run in Valgrind fully.

import Foundation

// at least 32 bytes
struct MyAwesomeStruct {
    let num0: Int64 // 8 bytes
    let num1: Int64 // 8 bytes
    let num2: Int64 // 8 bytes
    let num3: Int64 // 8 bytes
}

let semaphore = DispatchSemaphore(value: 0)
for _ in 0..<100 {
    Thread.detachNewThread {
        // structs
        var data: [MyAwesomeStruct]! = [MyAwesomeStruct].init(repeating: MyAwesomeStruct(num0: 0, num1: 1, num2: 2, num3: 3), count: 100_000)
        print("data count: \(data.count)")
        data = nil

//        // data
//        var data: Data! = Data(repeating: 0, count: 10*1024*1024)
//        print("data count: \(data.count)")
//        data = nil

//        // pointer
//        let data = UnsafeMutableRawBufferPointer.allocate(byteCount: 10*1024*1024, alignment: MemoryLayout<UInt8>.alignment)
//        print("data count: \(data.count)")
//        data.deallocate()

        semaphore.signal()
        Thread.sleep(forTimeInterval: 10)
    }
    semaphore.wait()
}

print("Idle now...")
Thread.sleep(forTimeInterval: 15)
print("All threads should be be gone...")
Thread.sleep(forTimeInterval: 5)

lukasa · 2020-10-23T09:57:33.885Z

So small examples are very helpful because they allow us to do really detailed analysis. In the case above we can pretty safely compile, repro, and look at both the ASM and the SIL.

In both cases, the underlying heap allocation for the Data is unequivocally being released. The memory seems to be being kept alive by the thread closure itself. If you remove the Thread.sleep at the end to allow the thread to exit, you find the memory usage does drop back to where you want it to be. So it's not that the data is not being freed, it's just not being freed when we want it to.

As a curiosity: what do you see if you allocate slightly more data? Try changing 10*1024*1024 to 100*1024*1024.

robertjpayne · 2020-10-23T10:38:58.087Z

If I allocate 100mb it doesn't exhibit the issue.

Interesting so though -- when you say it's not being freed when we want it to is this a fault of the compiler and where it's retaining the memory and/or releasing it or is it the way we've coded this sample?

We can't really let the threads exit just whenever as we use a Thread per TCP connection and it will continue to serve requests until the client closes it or it times out.

robertjpayne · 2020-10-23T10:56:05.650Z

@lukasa

For what it's worth I tried not using a closure based Thread but instead a subclass to see if it helps but it doesn't seem to make a difference. Also if we allow the thread to exit immediately without the sleep it sometimes results in less memory usage and sometimes it does not, seems a bit like a race condition determines that.

But that really doesn't make heaps of sense either, these Threads are actually exiting before the program is finished and thus I can't see why if they sleep first the memory usage is then "stuck"?

import Foundation

// at least 32 bytes
struct MyAwesomeStruct {
    let num0: Int64 // 8 bytes
    let num1: Int64 // 8 bytes
    let num2: Int64 // 8 bytes
    let num3: Int64 // 8 bytes
}

class Executor : Thread {
    override func main() {
        super.main()
        self.dataTest()
    }

    private func structTest() {
        var data: [MyAwesomeStruct]! = [MyAwesomeStruct].init(repeating: MyAwesomeStruct(num0: 0, num1: 1, num2: 2, num3: 3), count: 100_000)
        print("data count: \(data.count)")
        data = nil
    }

    private func dataTest() {
        var data: Data! = Data(repeating: 0, count: 10*1024*1024)
        print("data count: \(data.count)")
        data = nil
    }

    private func rawPointerTest() {
        let data = UnsafeMutableRawBufferPointer.allocate(byteCount: 10*1024*1024, alignment: MemoryLayout<UInt8>.alignment)
        print("data count: \(data.count)")
        data.deallocate()
    }

    deinit {
        print("Executor deinit")
    }
}

for _ in 0..<100 {
    let thread = Executor()
    thread.start()
}
Thread.sleep(forTimeInterval: 5)
print("All threads should be be gone...")
Thread.sleep(forTimeInterval: 15)

lukasa · 2020-10-23T11:34:21.676Z

The fact that allocating larger objects makes the problem go away points at the allocator, not the compiler. If I had to guess I'd say that 10MB allocations are not being served directly from mmap, and the allocator is choosing to keep hold of the allocated pages rather than return them to the OS when the data is freed. Again, I believe the pointer is being freed, the memory just isn't returning to the OS.

robertjpayne · 2020-10-23T11:36:46.200Z

@lukasa

Right that makes more sense, is there any info about how Swift allocates Structs/Data? I'd really like to be able to replicate the issue using raw C allocations because using raw memory pointers via UnsafeMtuableRawBufferPointer the ram is always released back to the OS

robertjpayne · 2020-10-23T13:07:51.398Z

Interestingly if I swap the memory allocator out to jemalloc the issue goes away entirely. @lukasa thanks heaps for your help here it looks like it's simply the allocator not releasing the pages back to the OS.

I wonder if my manual allocations via UnsafeRawBufferPointer are not kept because they are not aligned to the memory page sizing… hard to know.

EDIT:

A bit more education on memory allocators and this is definitely the reason. I suppose a lot of Swift's allocations are small and those tend to never get released back to the OS while larger ones do and the largest of allocations are released back immediately which is why your 100mb test works!