-retry-tests-on-failure retries @Test that have already passed

Using Swift
1

Raised FB20922425 as it's an Xcode feature but asking here also.

When using an xctestplan and enabling the -retry-tests-on-failure only failing XCTests are retried, but all Swift Testing @Test within the current target are retried, even if they have already passed.

struct SwiftTestingA {
  @Test
  func success(){
    #expect(Bool(true))
  }

 @Test
  func failure() {
    #expect(Bool(false))
  }
}

struct SwiftTestingB {
  @Test
  func success() async throws {
    #expect(Bool(true))
  }
}

final class XCTestA: XCTestCase {

  func testSuccess() {
    XCTAssertTrue(Bool(true))
  }

  func testFailure() {
    XCTAssertTrue(Bool(false))
  }
}

I see this behaviour in both xcodebuild and the UI app

retries
retries1168×1350 101 KB

2 Likes
2

Swift Testing currently has more limited support for "retry" functionality than XCTest: it behaves as you describe, by rerunning all tests if any of them fail when "retry on failure" is enabled by Xcode or xcodebuild.

I think it would be a good idea to enhance the Swift Testing library to make this more granular and only retry the specific tests which failed to avoid unnecessary work. Can you file a GitHub issue in the swift-testing repo to track that? It has come up before anecdotally but AFAIK there's not yet an issue tracking it.

Note that if such a change were made, it would be a separate task for the (closed-source) Xcode product to adopt such functionality, and that's not something I can discuss on this forum. However, the Feedback you already filed (FB20922425) can presumably be used to track that request.

1 Like
3