XCTest xUnit Experimental failure messages

Development Package Manager
, ,
1

Hello folks,

Currently, when running swift test --parallel --xunit-output <myfile> , Swift Package Manager can generate up to two files, one for XML for XCTest and another for Swift Testing.

If there are any failed tests, the Swift Testing XML output contains a failure message that accurately represents the test failure. However, the XCTest equivalent simply displays failed , which is redundant with the test status.

The xUnit XML story for XCTest is not the greatest, and we have plans to properly fix this in the future. But in the interim, to try to improve the XCTest XML failure message ever so slightly, an experimental, and hidden, feature flag was introduced to the main nightly toolchain, via this PR, which only impacts XCTest xUnit behaviour. When enabled, instead of displaying the hardcoded failed message, an XML escaped output of the test is used as the failure message. This output may contain some content that is irrelevant to the test, and the content of the "failure" message is subject to change in the future, as this is an experimental flag. The XCTest experimental failure message captures stdout and stderr.

I want to emphasize this is experimental because a proper solution will take some time as we want to do it right. This is for folks that do not want to wait for the proper solution, and are fine with not having to rely on the contents of the failure message.

When enabled, the feature:

  • does not change any logic around when the XML is created.
  • for XCTest only, updates the existing "failed" message in the xUnit XML to a manually XML escaped test output contents.

Here is an example of a before

    <failure message="failed"></failure>

and after

    <failure message="Test Suite 'Selected tests' started at 2025-01-13 13:15:49.732.&#10;Test Suite 'TestFailuresPackageTests.xctest' started at 2025-01-13 13:15:49.733.&#10;Test Suite 'TestFailuresTests' started at 2025-01-13 13:15:49.733.&#10;Test Case '-[TestFailuresTests.TestFailuresTests testExample]' started.&#10;/Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/Fixtures/Miscellaneous/TestSingleFailureXCTest/Tests/TestFailuresTests/TestFailuresTests.swift:5: error: -[TestFailuresTests.TestFailuresTests testExample] : XCTAssertFalse failed - Purposely failing &amp; validating XML espace &quot;'&lt;&gt;&#10;Test Case '-[TestFailuresTests.TestFailuresTests testExample]' failed (0.084 seconds).&#10;Test Suite 'TestFailuresTests' failed at 2025-01-13 13:15:49.816.&#10; Executed 1 test, with 1 failure (0 unexpected) in 0.084 (0.084) seconds&#10;Test Suite 'TestFailuresPackageTests.xctest' failed at 2025-01-13 13:15:49.816.&#10; Executed 1 test, with 1 failure (0 unexpected) in 0.084 (0.084) seconds&#10;Test Suite 'Selected tests' failed at 2025-01-13 13:15:49.816.&#10; Executed 1 test, with 1 failure (0 unexpected) in 0.084 (0.085) seconds&#10;"></failure>

If you would like to try this experimental flag:

  1. Download the latest nightly toolchain from branch main (any toolchain after, and including, February 15, 2024, should contain this feature)
  2. Ensure the toolchain contains this feature by running swift test --help-hidden and ensure --experimental-xunit-message-failure is a supported argument.
  3. Provide --experimental-xunit-message-failure and --xunit-output <file> to the swift test command line argument

WARNING: This is an experimental flag and I cannot stress the importance of not relying on the output of the failure message because the failure message output can change at any time.

4 Likes