Trying to get crash backtracing to work on Windows

Using Swift
,
1

I am trying to get the Swift runtime's crash backtracing to work for my Swift (Windows App SDK) GUI app (as a starting point for building some crash reporting service around that later on).

I am using a very recent Swift 6.4 Windows toolchain on an Intel machine running Windows 11. My build setup is CMake-based, not SPM-based.

I have read the Backtracing support in Swift documentation, and using the following configuration

set SWIFT_BACKTRACE=interactive=no,output-to=C:\tmp\CrashLogs

I also tried including more flags, like swift-backtrace=C:\Swift\Runtimes\6.4.0\usr\libexec\swift\swift-backtrace.exe, or symbolicate=off, also with quotes around the paths, but none of that made a difference.

To simulate a crash, I am calling fatalError() in my app, I hope that's sufficient; it certainly does crash the app, and Windows Error Reporting kicks in (see below), so I suppose it is.

The behaviour I am seing is that no crash logs are written to the folder specified in the environment variable. I first tried a relwithdebinfo build and later also a debug build, with the same outcome.

Note that my app doesn't have a console window / stdout; I am specifying

target_link_options(Compositor PRIVATE
    -Xlinker /SUBSYSTEM:WINDOWS
)

to make this a GUI app (with winMain entry point instead of main) that launches without opening a terminal window (which would be unacceptable for my use case).

After getting no crash logs, I recorded a short app session using Windows Performance Recorder to see what happens around crash time. Windows Error Reporting (WerFault.exe, wermgr.exe) seem to kick in when the crash happens:

image
image2569×1315 262 KB

And here's what I assume are corresponding logs in the Windows Event Viewer:

image
image1920×1114 356 KB

image
image1920×1114 300 KB

But I don't know if that's helpful for diagnosing Swift backtracing.

I will be in tomorrow's Windows Workgroup meeting anyway, but I wanted to write this up beforehand and hopefully give useful information for further diagnosis.

thanks,
Karl

Edit: swift-backtrace.exe does not show up in the list of processes around the crash time; it looks like the runtime wasn't able to launch it (I understand it runs as a process external to the crashing app):

image
image2604×1443 455 KB

2

I'm sorry to hear it @ktraunmueller ... we chatted a bit on the Windows Office Hours call so I'm repeating a bit for anyone else who is reading this post or interested in the native swift runtime crash logging/backtracing on Windows.

I think the top line thing I'd say is generally it should (and mostly does!) work on straight out of the box installations. It certainly has for me. So I don't think you're doing anything wrong.

When I get a chance I'll try installing the latest Swift 6.4 toolchain and just check that it seems to be fine. We have a fairly extensive suite of Backtracer tests that run on Windows CI machines that you can see on the usual swift.org CI pages.

If you can post your project source code or (much better) a minimum reproducer of the issue on here that would help, as I can try to reproduce the issue on my Windows 11 PC with the latest swift 6.4 Windows installer package.

If you look at this example unit test from the swift repository:

You can see a sort of classic simple program that you can build using the swift on windows toolchain and when you run it you should see a nice crash log on the console. If that doesn't work then something is odd on your installation.

If you're able to get Windows crash logs from "normal"/simple swift programs built using the toolchain from the swift 6.4 installer on your Windows PC, then I guess it's something specific about your project, and we'll have to dig into the source code.

I hope this helps as a first few steps. I'm very keen to help get you up and running if we can.

Carl

p.s. Unfortunately I have to go on annual leave for the rest of this week, however don't let that stop you from replying. Quite likely other readers of the forums can chip in with help if you post a reply. Good luck!

p.p.s. [edit] One final note, if you do get a backtrace from a test program or your own program but it is not symbolicated, you might need to make sure that debug symbols are being emitted in your build. See the documentation in swift/docs/Backtracing.rst at main · swiftlang/swift · GitHub and the forum post detailing the backtracer written when it was launched: On-Crash Backtraces in Swift | Swift.org for more help on that or, as always, ask for help on here!

1 Like
3

Thanks Carl @carlpeto

Using this little test program

func kablam() {
  kerpow()
}

func kerpow() {
  whap()
}

func whap() {
  zonk()
}

func zonk() {
  splat()
}

func splat() {
  pow()
}

func pow() {
  fatalError("Kablam!")
}

@main
struct SymbolicatedBacktrace {
  static func main() {
    kablam()
  }
}

and compiling and running it on my (Intel) machine

C:\tmp>%SWIFT_TOOLCHAIN_PATH%\bin\swiftc -O -g -debug-info-format=codeview -Xlinker /DEBUG -parse-as-library crasher.swift -o crasher.exe
   Creating library crasher.lib and object crasher.exp

C:\tmp>crasher.exe
crasher/crasher.swift:22: Fatal error: Kablam!

💣 Program crashed: Illegal instruction at 0x00007ff95644be7e

Platform: x86_64 Windows 11.0 build 26200

Thread 0 crashed:

0 0x00007ff95644be7e <unknown> in swiftCore.dll
1 0x00007ff6006b182d main + 92 in crasher.exe
2 0x00007ff9a43fccb7 <unknown> in KERNEL32.DLL
3 0x00007ff9a608ad6c <unknown> in ntdll.dll

Backtrace took 0.04s

I am getting a backtrace on the console, so that's looking good.

As suggested in the last Windows workgroup meeting by @al45tair, I recorded a (simulated) crash in my installed app with procmon - the log is available here, if that's of any help (the process is Compositor.exe, a RelWithDebInfo x86_64 build).

Thanks,
Karl

4

It looks like it's looking for swift-backtrace.exe in C:\Program Files\WindowsApps\Compositor_0.8.2.0_x64__b202aj40gcyaw\swift-backtrace.exe (i.e. alongside your program). I don't understand why it is ignoring the content of the SWIFT_BACKTRACE environment variable, mind — you'd specified that it should be in a different location to that.

5

Ah, I forgot (in my installer script) that I will need to bundle swift-backtrace.exe alongside the other Swift runtime libraries anyway:

# Copy binaries
Copy-Item -Path "$buildOutputFolder\Compositor.exe" -Destination "$installerStagingFolder" -Force

# Copy supplementary Windows App SDK binaries
Copy-Item -Path "$buildOutputFolder\swift-windowsappsdk_CWinAppSDK.resources" -Destination "$installerStagingFolder" -Recurse -ErrorAction SilentlyContinue
Copy-Item -Path "$buildOutputFolder\Microsoft.Graphics.Canvas.dll" -Destination "$installerStagingFolder" -Force -Recurse -ErrorAction SilentlyContinue

# Copy Swift binaries
Copy-Item -Path "$swiftRuntimeFolder\bin\swiftCRT.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\swiftCore.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\swiftDispatch.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\swiftWinSDK.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\swift_Concurrency.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\swift_RegexParser.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\swift_StringProcessing.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\swiftObservation.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\swiftSynchronization.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\BlocksRuntime.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\dispatch.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\Foundation.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\FoundationEssentials.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\FoundationInternationalization.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\FoundationNetworking.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\bin\_FoundationICU.dll" -Destination "$installerStagingFolder" -Force
Copy-Item -Path "$swiftRuntimeFolder\libexec\swift\swift-backtrace.exe" -Destination "$installerStagingFolder" -Force  # <-- this had been missing

Then I can remove the path argument from the SWIFT_BACKTRACE environment variable again:
enable=yes,interactive=no,symbolicate=off,output-to=C:\tmp\CrashLogs

But still nothing in C:\tmp\CrashLogs after a simulated crash.
Here's another log with the backtracer now in the app installation folder.

6

OK, I don't see anything especially obvious in that log, though I do note that we aren't trying to create the swift-backtrace.exe process, which suggests that something has altered the Windows exception handling set-up (which is how we trap errors).

1 Like
7

Complete tangent: use of the MSM would allow you to ignore the "Copy Swift binaries" segment - including swift-backtrace.exe.

8

Yeah, I really need to change the runtime distribution setup.

9

@carlpeto @al45tair Sorry for taking so long to follow up.

I have prepared a reproducer project at GitHub - ktraunmueller/backtrace-reproducer: Reproducer project for Swift Windows backtracing · GitHub. It is modeled after the project setup, Windows App SDK integration, and build system of my own app, to create something that is as close as possible to my scenario without carrying too much distracting baggage.

The repository's Readme should hopefully have enough detail and instructions to quickly be up and running with a binary that should reproduce the issue. The app is entirely self-contained and does not depend on a Windows App Runtime installation.

I have tested the reproducer on my Intel-based Microsoft Surface Laptop with a recent Swift 6.4.x development snapshot toolchain. The Readme has detailed version information of all the tools and components I am using.

Thanks in advance,
Karl

CC @kellychan-bcny I promised to provide this reproducer project in the September 2 Windows workgroup meeting, so there might be a ToDo with my name on it on your list to tick off.

10

Thanks @ktraunmueller looks good ... I'm afraid with an upcoming office move I can't start properly on this until Tuesday. I'll try to reproduce the issue on my machine then. Sorry for the delay.

11

@carlpeto No worries, there is absolutely no rush to this. I just hope I could provide you with something useful.

12

Hi @ktraunmueller ,

I thought I'd give you a short progress update.

I've managed to reproduce your issue, building a simple WinUI program on Windows on my PC now things have settled after the office move. The backtrace doesn't show, you're quite right. I did a bit of digging and I've got some clues.

One thing I found is if I crash immediately in WinMain before starting the UI, then the backtrace handler does run...

@_cdecl("wWinMain")
public func wWinMain(
_ hInstance: HINSTANCE?,
_ hPrevInstance: HINSTANCE?,
_ lpCmdLine: PWSTR?,
_ nCmdShow: INT32
) -> INT32 {
fatalError("Intentional AT STARTUP crash for Swift backtrace testing")
BacktraceReproducerApp.main()
return 0
}

...so I think we might be getting some interference from within WinUI or some Microsoft framework. Perhaps it has a _try/_except handler or something similar that's catching the error before it reaches the swift program's top level handler.

Note, one small gotcha. When I ran my version with the immediate fatal error in WinMain, I was slightly confused for a minute because I could see backtracing start and a console window popped up briefly. Subsequent investigation showed that window is swift-backtrace.exe. I think there's a couple of things going on here...

Because we have the -Xlinker /SUBSYSTEM:WINDOWS linker flag, the program doesn't have a console or standard handles. When it creates the process for swift-backtrace.exe, because of the next issue (see below), Windows sees this is a console program and creates a console for it. Also you don't see any console output from the backtracer in the runtime stub as you normally always would, for the same reason (in my tests I think you can get powershell to store the standard output to file using something like Start-Process -Wait -FilePath .\build\relwithdebinfo-x64\SwiftBacktraceReproducer.exe -RedirectStandardError C:\tmp\err.txt -RedirectStandardOutput C:\tmp\out.txt).

If you change your Windows Terminal to stay open after programs exit* then you'll see the backtrace. But really what we want is what you configured in your README... the crash log to go to a file, which makes much more sense for a UI app. That is failing because...

SWIFT_BACKTRACE is not being read correctly from the environment at process launch (again).

We've seen this issue before and I thought I'd fixed it. Tomorrow I'll go back and have a look over the details. Last time I dealt with it, it was to do with one of those slightly tiresome issues where the environment is in a "narrow character environment variable" and we're reading the "wide character environment variable" or vice versa, something like that. Good news is it's fixable. Bad news is it might mean a patch to the runtime. I'll get back to you with details tomorrow and a workaround if I can figure one out.

Anyway, I think we are moving towards an answer.

I'll also take a look tomorrow to see if there's a way to work around whatever exception handling WinUI (or some other Microsoft framework) is doing. We might need to have a bit of a think about it and about what's appropriate for UI programs?

I'm open to anyone's thoughts of course!

Cheers,

Carl

(*) If you use Windows Terminal, go to Settings → Startup → Default terminal application and choose Windows Terminal. Then under Defaults → Advanced, set Profile termination behaviour to Never close automatically ("closeOnExit": "never"). The swift-backtrace.exe window will then stay open after it exits, with its output.

1 Like
13

Extra little wrinkle... if you're building this project using a just-built compiler (following instructions like swift/docs/WindowsBuild.md at main · swiftlang/swift · GitHub ), I'd suggest updating variables such as...

$S = 'S:\Program Files\Swift'
$env:SWIFT_TOOLCHAIN_PATH = "$S\Toolchains\0.0.0+Asserts\usr"
$env:SWIFT_RUNTIME_PATH = "$S\Runtimes\0.0.0\usr"
$env:SDKROOT = "$S\Platforms\Windows.platform\Developer\SDKs\Windows.sdk"

Before running cmake with presets, then cmake --build.

14

@ktraunmueller Not a huge amount to add apart from my summary to everyone in the Windows Office Hours meeting just now.

The SWIFT_ environment variable reading bug typically does not affect Console programs built by Swift on Windows but affects most UI programs built using Swift on Windows.

The PR to fix it is: [Windows] [Runtime] Read SWIFT_ environment variables directly using Win32 API by carlpeto · Pull Request #93124 · swiftlang/swift · GitHub which is getting near completion but even once that PR lands, it will likely not be cherry-picked to 6.4.x, so you will not be able to use SWIFT_BACKTRACE to control the Backtracer on Windows UI programs unless you download a nightly build.

As discussed, with the default setup, backtraces will go to a popup console window, but it's often short lived, so you lose the backtrace there.

Given that the console window is a bit odd for UI programs, I've created a draft PR to give a bit more of a Windows UI friendly approach: [Windows] [Backtracing] Add Windows UI app improvements for crash logs. by carlpeto · Pull Request #93130 · swiftlang/swift · GitHub ... this does things like suppress the console popup by default for UI programs that don't have a console and, if no other destination is specified, fall back on logging something to the Windows Event Application Log at least. (Note that WER usually also logs an unhandled exception in the application log, although in this case that seems a slightly unfair description, given that our unhandled exception handler was not allowed to run!)

The other more pertinent issue in this case is WinUI programs seem to be capturing unhandled exceptions and converting them into fail fast errors or something similar, so the Swift Runtime unhandled exception handler is still installed but it is not getting to see the exception. Windows is gracefully closing the program before then.

We might be able to intercept this using the vectored exception handler, which is still being triggered I think, but we might need to watch for unexpected side effects. I'll keep investigating a bit.

short update: https://github.com/swiftlang/swift/pull/93124 is now merged, so SWIFT_BACKTRACE should now work correctly on Windows subsystems linked programs on MS Windows in the next nightly build of the installer.

1 Like