Xcode does also include succeeded tests in failed test section

I am using Xcode's test plans with the option 'Retry until failure' and some repetitions. When a test fails but succeeds, it will still show up in the failed test section when another test is actually consistently failing. This is my code:

import XCTest

class SomeTests: XCTestCase {
    func testExampleA() throws {
        let random = Int.random(in: 0...10)
        
        // Will always fail
        if random != -1 {
            XCTFail("fail")
        }
    }

    func testExampleB() throws {
        let random = Int.random(in: 0...3)
        
        if random != 1 {
            XCTFail("fail")
        }
    }
}

And I run the test command:

xcodebuild test -project uitest.xcodeproj -scheme unittest -destination 'platform=iOS Simulator,name=iPhone 12,OS=15.5' -only-testing SomeTests

This is the output:

/Users/jaspervisser/Desktop/uitest/dfsdfsff/dfsdfsff.swift:16: error: -[dfsdfsff.SomeTests testExampleA] : failed - fail
Test Case '-[dfsdfsff.SomeTests testExampleA]' failed (0.005 seconds).
Test Case '-[dfsdfsff.SomeTests testExampleA]' started (Iteration 20 of 20).
/Users/jaspervisser/Desktop/uitest/dfsdfsff/dfsdfsff.swift:16: error: -[dfsdfsff.SomeTests testExampleA] : failed - fail
Test Case '-[dfsdfsff.SomeTests testExampleA]' failed (0.005 seconds).
Test Case '-[dfsdfsff.SomeTests testExampleB]' started (Iteration 1 of 20).
/Users/jaspervisser/Desktop/uitest/dfsdfsff/dfsdfsff.swift:24: error: -[dfsdfsff.SomeTests testExampleB] : failed - fail
Test Case '-[dfsdfsff.SomeTests testExampleB]' failed (0.005 seconds).
Test Case '-[dfsdfsff.SomeTests testExampleB]' started (Iteration 2 of 20).
/Users/jaspervisser/Desktop/uitest/dfsdfsff/dfsdfsff.swift:24: error: -[dfsdfsff.SomeTests testExampleB] : failed - fail
Test Case '-[dfsdfsff.SomeTests testExampleB]' failed (0.005 seconds).
Test Case '-[dfsdfsff.SomeTests testExampleB]' started (Iteration 3 of 20).
Test Case '-[dfsdfsff.SomeTests testExampleB]' passed (0.019 seconds).
Test Suite 'SomeTests' failed at 2022-07-04 17:55:03.367.
         Executed 23 tests, with 22 failures (0 unexpected) in 0.184 (0.202) seconds
Test Suite 'dfsdfsff.xctest' failed at 2022-07-04 17:55:03.373.
         Executed 23 tests, with 22 failures (0 unexpected) in 0.184 (0.209) seconds
Test Suite 'All tests' failed at 2022-07-04 17:55:03.377.
         Executed 23 tests, with 22 failures (0 unexpected) in 0.184 (0.217) seconds
2022-07-04 17:55:28.752 xcodebuild[77552:4967879] [MT] IDETestOperationsObserverDebug: 31.789 elapsed -- Testing started completed.
2022-07-04 17:55:28.752 xcodebuild[77552:4967879] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2022-07-04 17:55:28.752 xcodebuild[77552:4967879] [MT] IDETestOperationsObserverDebug: 31.789 sec, +31.789 sec -- end

Test session results, code coverage, and logs:
        /Users/jaspervisser/Library/Developer/Xcode/DerivedData/uitest-dthpivieuzigdfgvtdrzgailclme/Logs/Test/Test-unittest-2022.07.04_17-54-56-+0200.xcresult

Failing tests:
        dfsdfsff:
                SomeTests.testExampleA()
                SomeTests.testExampleB()

** TEST FAILED **

You can see that testExampleB has succeeded, but it still pops up in the failing tests section. It takes me so much time in the 'real' CI to find actual consistently failing tests, I don't care about tests that fails and succeeds, I just need to kind consistently failing tests. Is there a way to identify them? Can I filter out tests that succeeds?

1 Like