Measuring, avoiding, and diagnosing a stack memory overflow crash

I investigated a stack overflow issue earlier this year. In my case none of the existing mechanisms (macOS stack guard, address/thread sanitizers, etc) were able to detect the issue. I summarized what happened under the hood here.

I found it was very helpful to estimate stack size using vmmap. See my approach here. How to apply it in practice, however, might be tricky. In my case when stack overflow occurred, the main thread silently used another thread's stack, so vmmap output was't accurate. To avoid this, you can set a large custom stack size (as you mentioned) when using this approach.

There are many reasons (including compiler bugs) that could lead to stack overflow. I think the key is to identify the code that caused the issue. The most straightforward approach is to look at assembly code (it doesn't require being familiar with low level details like register usage, but just to get basic information like what func it is and stack size allocated). In my case I used this approach to identify the issue occurred in enum accessor. EDIT: I'm aware that the code that crashed might not be the code that caused the issue. For example, in my case it's completely possible that main thread messed up background thread's stack and caused the background thread crashed. So this approach has its limitation, though it should be helpful usually..

2 Likes