Symbolicating Minidumps on Windows

Development Windows
,
1

I have a Sentry minidump of a crash from a user (truncated raw stack trace below). Using swift-5.9-RELEASE

A little complicated, but it's just an Electron app that loads a native module built from Swift. Don't worry, I'm packing the swift runtime dlls + vc redist + using a SxS manifest in the module I'm loading. I guess I can't totally ensure the user doesn't have Swift installed locally, but I'd really hope the SxS manifest would always prefer the specified side-by-side DLLs. The crash happens when we begin loading the native module, and the crash seems to not even touch our native Swift code it seems.

Thousands of users use this app daily without any issue, but this one user seems to be having issues. There are some locale-specific things going on here regarding the user (Chinese characters in paths, etc.) but we have other users in China successfully using the app, so that may be a red herring.

My question is, how do I symbolicate this? I have access to the suspect Swift runtime dlls in question, but when I try to upload them to Sentry, the sentry-cli doesn't detect any debug info. Just curious if anyone has had success using Sentry to debug the Windows Swift Runtime. I could simply bump the Swift runtime to a newer version and see if that does anything, but it's much better to not grasp at straws and just get to the bottom of what's going on.

OS Version: Windows 10.0.19045 (3636)
Report Version: 104
Crashed Thread: 11376

Application Specific Information:
Fatal Error: EXCEPTION_ILLEGAL_INSTRUCTION / 0x7ffd4088acec

Thread 11376 Crashed:
0   swiftCore.dll                   0x7ffd4088acec      <unknown>
1   swiftCore.dll                   0x7ffd40c1e072      <unknown>
2   swiftCore.dll                   0x7ffd4098ffdf      <unknown>
3   ucrtbase.dll                    0x7ffdb59bf05a      free_base
2

Unfortunately, we do not have PDBs for the runtime yet. The problem here is that we need to figure out how to get those onto a symbol server. I do not wish to include them in the (offline) installer as that will grow the size of the installer - the installer is close to ~500 MiB now, and adding debug information would push it to ~20 GiB. This has a massive impact on experience, resource usage, and CI as well as I have been unable to convince GH to really include Swift as part of the default image.

That said, it should be able to symbolicate for anything in the export symbol list as that should still be present.

The best option right now might be to build the toolchain (see GitHub - compnerd/swift-build: Alternate Swift Builds for an initial GHA definition or Azure Pipelines definition).

With online installation, we could add Debug Information as an optional fetch, but that requires getting the offline installer setup as well (CC: @mishal_shah).

3

Thanks @compnerd for your response!

OK, so I think I do want to spend the time and figure out how to do this. Please assume I have no idea what I'm doing here :slight_smile:.

Are you saying that I could build the 5.9 toolchain for x86 x64 Windows and get debug symbols that would be compatible with this existing minidump from the user?

Assuming I'm using the default non-developer Powershell, what would that command look like to build the toolchain, and where would the PDB files be?

4

I assume you mean x64; x86 is a WIP as there are known places where SIL breaks down. It might be (I think that you would need to match the compiler version exactly).

Assuming that you are using https://github.com/compnerd/swift-build/blob/main/docs/WindowsQuickStart.md ...

.\swift-build\build.ps1 -BuildType RelWithDebInfo -CDebugFormat codeview -SwiftDebugFormat codeview -SkipPackaging

on the release/5.9.1 branch:

repo init -u https://github.com/compnerd/swift-build -b release/5.9.1

is the tweak to the quick start that you would need (the addition of the branch).

5

Oops yes corrected.

Thanks so much for the help! I will take a stab at this and report back if I end up getting anything working.