Using Backtrace API on linux with Swift 5.10

Server
1

I've been looking for a way to enrich the error logs from my backend by a stack trace and discovered the Backtrace API.

I thought it might be useful to share the progress/differences I've observed trying to use this API with Swift 5.9 and Swift 5.10 running my Vapor backend in a docker container on DigitalOcean using their App Platform $10.00 / month thing. Here is the Dockerfile I'm using Dockerfile · GitHub

Here is my test routes


    import _Backtracing

    app.get("teststacktracep") { req async throws in
        
        let now = Date.now
        let trace = try Backtrace.capture(algorithm: .precise).symbolicated()
        
        return "\(Date.now.timeIntervalSince1970 - now.timeIntervalSince1970) | \(trace?.frames)"
    }
    
    app.get("teststacktracef") { req async throws in
        
        let now = Date.now
        let trace = try Backtrace.capture(algorithm: .fast).symbolicated()
        
        return "\(Date.now.timeIntervalSince1970 - now.timeIntervalSince1970) | \(trace?.frames)"
    }

What I've observed is that precise symbolication seems to take around 15 seconds, fast symbolication seems to take around 7 seconds on average. This seems to be roughly 2x faster than my previous attempts using Swift 5.9.

The 15 seconds seems like an acceptable time to me to collect a symbolicated stack trace when my process crashes before it's reset. The 7 seconds when using fast symbolication is still unfortunately way too slow for me use case where I'd like to attach a stack trace to an error log being sent to Sentry (uses standard Logging framework with sentry as a destination), so I'll probably need to look into my options uploading unsymbolicated stack trace and performing the symbolication asynchronously. Any guidance here would be appreciated.

Furthermore, what I've been observing with the limited analytics DigitalOcean App Platform provides, there seems to be a significant spike (leak???) in memory every time I trigger the API (call the vapor route) that does not drop back over time. Please refer to the screenshot bellow

Screenshot 2024-04-02 at 15.56.46
Screenshot 2024-04-02 at 15.56.462438×1286 181 KB

I believe that the total amount of available RAM is 1GB.

This matches my experience using Swift 5.9, the memory footprint (leak???) just also seems lower from what I remember.

Hope this helps!

3 Likes
2

I just want to note that this is not necessarily a memory leak. When triggering a backtrace a lot of memory is allocated to store the stack information and potentially any symbolic information that was loaded. This memory should be freed after the backtrace has been deallocated; however, it doesn't mean that your process allocator returned the memory to the system. Most allocators retain freed memory for future allocations to improve performance.

Out of curiosity, could you share the system that you are running this on? I am curious how much CPU is available here to produce the back trace.

3

DigitalOcean is quite opaque about the specs, it's the $10/month VM with 1 gig of RAM App Platform Pricing | DigitalOcean.

I'm not 100% sure it's a memory leak either, I have not tried to invoke out of memory scenario yet, will need to spin up a dedicated instance for it. Running the backtrace for the 2nd time seems to allocate the same chunk of memory again instead of reusing the previously allocated one though.

Might be worth mentioning that I'm using jmalloc, as you can see in the dockerfile Dockerfile · GitHub

4

To be clear, the stuff in _Backtracing isn't actually API yet, hence the underscore. It's very similar to but not identical to the final form that will end up in the Runtime module. So if you rely on this, we may break you at some point; at the very least the _Backtracing module will go away, and the final API is going to look like the Swift Evolution proposal, SE-0419. You'll probably be OK if you put everything inside a #if canImport(_Backtracing). Probably. But no promises.

We don't yet have any tooling to perform the out-of-band symbolication; that's something that needs doing as part of the work to rearrange things and turn this into API. You could, however, put something together that just uses addr2line on the unsymbolicated output — the output is designed to be machine as well as human readable.

On a performance front, I'm pretty certain there's scope for further optimizations in the symbolication code, so I think the amount of time taken is likely to fall again in future releases, but for cases where you really need to be fast, the unsymbolicated mode is the thing to use.

4 Likes