[Pitch] Debugging Swift Tests with LLDB

Evolution Pitches
,
1

Hello,

swift run supports interactive debugging via the --debugger flag, which builds the executable and launches it inside the LLDB that is bundled with the Swift Toolchain.

However, there is no equivalent flag for swift test . Developers who want to step through a failing test today have to figure out where the test binary is built, invoke LLDB by hand, and pass the right arguments to the built testing binary. They then have to repeat the process for each testing library and each test product in their package.

This pitch proposes a --debugger flag for swift test that builds the test products and drops the developer into an LLDB session pre-configured for both XCTest and Swift Testing, with target switching for multi-product and multi-test-library packages.

Motivation

Debugging a failing Swift test from the command line is currently a manual, error-prone exercise. The user must:

  1. Build the test product (swift build --build-tests).
  2. Locate the correct test bundle or test executable inside .build/.
  3. Construct the right argument list: --test-bundle-path, --testing-library swift-testing, --enable-swift-testing, XCTest filter selectors, etc.
  4. Launch LLDB, set breakpoints, and finally run.

This process is difficult and poorly documented. Editors like Xcode and VS Code paper over this for IDE-driven debugging, but there is no simple method for debugging on the command line.

swift run already solved this problem for executables with swift run --debugger , which builds the product and exec s LLDB attached to it. Bringing the same ability to swift test is the natural extension, and unblocks a workflow that today requires either an IDE or a deep understanding of SwiftPM internals.

Proposed solution

We propose adding a new --debugger flag to swift test . When set, SwiftPM builds the test products as it normally would, then exec s LLDB instead of running the test process. The LLDB session is pre-configured with one target per (test product, testing library) pair, with the right executable and run-arguments wired up for each. In addition a failbreak command alias is registered that sets breakpoints on the symbols called for XCTest and/or Swift Testing test failures.

$ swift test --debugger
# ... build output ...
(lldb) failbreak
(lldb) run

The flag composes with existing test selection options: --filter , --skip , --enable-swift-testing , --enable-xctest , etc. all narrow which targets are registered in the LLDB session, so a developer who runs swift test --debugger --filter MyFailingSuite only sees the relevant binary in LLDB.

Detailed design

Configuring LLDB targets

A swift test --debugger session is described by a DebuggableTestSession containing one entry per (test product, testing library) pair that survives the user's --filter and --skip selectors. Before exec ing LLDB, DebugTestRunner writes a temporary command file that, for each entry, issues:

  • target create -l "<productName> (<library>)" "<executable>" — registers a new LLDB target and gives it a human-readable label so the user can tell LibATests (XCTest) from LibATests (Swift Testing) in target list output.
  • settings clear target.run-args followed by one settings append target.run-args "<arg>" per argument. This uses the same launch arguments swift test would have used to run that product (--test-bundle-path, --testing-library, XCTest selectors, etc.).
  • target modules add "<binaryPath>". This explicitly loads the test binary's debug symbols. On macOS this matters because the active executable is xctest or swift-testing-helper, not the test binary itself, so symbols would otherwise only be picked up after the dynamic loader maps the bundle.

LLDB is then invoked with -s <command-file> , which replays these commands at startup. The user lands at an (lldb) prompt with every target already registered, run-args set, and symbols loaded — ready to failbreak and run .
All path components and product names are escaped before being interpolated into the quoted LLDB commands so that Windows paths, names containing spaces, and names containing \ or " cannot break the command file or inject additional commands.

Target switching

When a session contains more than one target, a Python helper script (target_switcher.py) is written to the same scratch directory and loaded into the LLDB session via command script import . The script:

  • Registers an event listener on each target's process and watches for exit events. When a target's process exits because test run completed naturally the script advances the active target via target select <next> so the next run drives the next test binary.
  • Mirrors user-set breakpoints across every registered target. A breakpoint set against MyTests.swift:42 while the XCTest target is active is replayed against the Swift Testing target before its first run, and vice versa, so the user does not have to remember which target is active when setting breakpoints. The script de-duplicates by file/line and by symbol name so re-syncing is idempotent.

For single-target sessions the script is not loaded.

Per-platform executable selection

The executable handed to LLDB depends on both the testing library and the host platform:

Library macOS Linux Windows
XCTest xctest <bundle> test binary test binary
Swift Testing swift-testing-helper --test-bundle-path <binary> test binary test binary

These are the same launch shapes that swift test uses today when running tests normally.

Multi-target sessions

A package can produce multiple test products (e.g. LibATests and LibBTests ), and a single product can host both XCTest and Swift Testing tests, which build into separate executables. To support all of these in one debugging session, DebugTestRunner registers each (product, library) pair as a distinct LLDB target and then loads a small Python helper script into the LLDB session. The script:

  • Listens for the active target's process to exit (e.g. when a test run ends or the user types exit).
  • Switches LLDB to the next registered target so the user can drive the next batch of tests without leaving the session.
  • Synchronizes user-set breakpoints across targets so that a breakpoint set while debugging the XCTest target also fires when the Swift Testing target runs, and vice versa.

For single-target sessions (the common case) the Python script is not loaded, keeping startup minimal.

Failure breakpoints

A failbreak command alias is registered in the LLDB session that sets breakpoints on the appropriate failure entry points for whichever testing libraries are present:

Platform XCTest Swift Testing
macOS _XCTFailureBreakpoint Testing.failureBreakpoint()
Linux XCTest.XCTestCase.recordFailure Testing.failureBreakpoint
Windows XCTest.XCTestCase.recordFailure Testing.failureBreakpoint() in Testing.dll

If the session contains both libraries, failbreak sets both. The user must run failbreak first themselves then run, and LLDB will pause when test assertion fails. The debugger will pause on the symbol defined in the table above, and the user can then display the stack trace and walk up to their failing assertion.

Process replacement

DebugTestRunner.run replaces the current swift-test process with LLDB via execv . LLDB inherits the test environment SwiftPM would have used to run the tests directly. Using execv means that the SwiftPM process doesn't have to act as an orchestrator that forwards signals and stdin/out/err back and forth between LLDB and the user.

Argument validation

--debugger is incompatible with several other flags. These are validated up-front so the user gets a clear error before any build work happens:

  • --configuration release: debugging requires debug symbols.
  • --parallel and --num-workers: debugging requires a single process to attach to, and --parallel will spawn --num-workers processes.
  • --list-tests: swift test list already exists for this.
  • --show-codecov-path: coverage path printing is incompatible with an interactive session.

The validation logic is extracted as a static validateLLDBCompatibility function so it can be exercised directly by tests without spinning up a full command pipeline.

Impact on existing packages

This proposal is purely additive. Behaviour of swift test without --debugger is unchanged. No existing flags, subcommands, or output formats are modified. Packages that do not opt in see no difference. The flag has no effect on Package.swift authoring or on how tests are written.

Alternatives considered

A subcommand (swift test debug)

A subcommand was considered, mirroring swift test list . We chose a flag for parity with swift run --debugger.

Spawning LLDB as a child process

Rather than exec ing LLDB, SwiftPM could spawn it as a child process. This was rejected for the same reasons as swift run --debugger : child-process LLDB sessions have second-class terminal control, awkward signal handling, and a confusing process tree. exec matches the existing precedent of swift run and gives LLDB full ownership of the TTY.

Try it out

You can try the in progress implementation:

Future directions

Pre-set breakpoints

Today the user can type failbreak then run . A future iteration could modify the debugger flag to be an option that allows for failure breakpoints to be automatically attached, i.e --debugger break-on-failure (or similar). Additionally we could walk up the stack to user code automatically instead of pausing on the internal XCTest/Swift Testing symbols.

Test selection inside LLDB

Once the session is open, the user is on their own to re-issue run with different target.run-args. A small set of LLDB command aliases for filtering or re-running individual tests would smooth this out, e.g. a runtest <name> alias that rewrites target.run-args for the active target.

13 Likes
2

SUPER excited to see this feature - this is something that I've wanted to use for a long time and attaching LLDB to unit tests has been quite painful without this. Thanks for pushing this forward!

One question that I had that I don't think was addressed in the pitch: how does this interact with swift-testing exit tests which spawns a child process and/or unit tests that spawn a child process via an API like Subprocess or Process? Is there some form of integration (natively from LLDB or as part of this change) that allows for LLDB to attach to the child process as well, or will the child processes run without the debugger attached. I've had cases where I want to debug child processes (both launched manually and launched via an exit test expectation) but I can also understand why attaching to every exit test process may not be beneficial since many are expected to crash, but I don't know what the expected behavior with this feature would be for those processes. I believe Xcode does have some support for debugging child processes when debugging with Xcode, so I wasn't certain if that behavior would carry over here or if that is an Xcode IDE-specific feature.

2 Likes
3

You raise a great point. Right now there isn't any support for debugging exit tests in the pitch. Since swift-testing spawns the code on the exit test block in a new process and breakpoints you set on code in these blocks is skipped over.

It looks like as of 2 weeks ago, swift-testing respects a new env var called SWT_START_CHILD_PROCESSES_SUSPENDED. This might give the LLDB Python script the ability to forward any breakpoints to the launched process before resuming the process. One caveat however is it doesn't work on Linux today.

1 Like
4

Ok thanks for confirming. Definitely not worth holding this pitch over that functionality - the value as it stands is still enormous. But, maybe it'd be a good thing to mention as a future direction for investigation in the proposal? It's probably not that common that you'd need it, but I suspect it'd be common enough that having that documented somewhere would be good.

Logistically, I'm sure it would also be even harder for processes spawned via a custom mechanism like Process/Subprocess where we don't have a tight coupling between SwiftPM and the process launching mechanism so I figured at a minimum that wouldn't be supported for now (cc @icharleshu as an FYI in case you have any thoughts on that aspect)

2 Likes
5

This is really fantastic, can't wait! :slight_smile: I'm often helping people adopt swift for the first time and lack of easy dropping into debugger from tests has been a hurdle quite often, so can't wait to see this improved :partying_face:

I like the spelling too, --debugger as an option sounds good :+1:

1 Like