Pitch: Make test repetition per-test-case instead of global

Evolution Pitches
1

Hi folks! I’ve been working on a pitch to fix some suboptimal (and in some cases, surprising) behavior in Swift Testing. When setting up test repetition to repeat tests when they fail (or until they fail), Swift Testing re-runs the whole test plan instead of just the test cases that failed.

This pitch reworks test repetition to repeat only those test cases which fail, and provides an iteration counter in the JSON event schema.

I’d love some feedback on this pitch, especially if you’ve been surprised by this behavior or if you prefer this behavior.

The pitch is here:

9 Likes
2

Makes a ton more sense! +1

1 Like
3

I found it crazy that this just happened without any warning, and don’t think it’s mentioned in the documentation. Messed up our CI when transitioning to Swift Testing from XCTest. So +100

The proposal says “if only one test case of a large parameterized test suite fails” but it’s also the case with non parameterised tests right?

4

Yes, non-parameterized tests have a single synthesized test case, that example was mostly illustrative of how much work needs to be redone.

5

Evolution review for this pitch has now begun here!

1 Like