Capture all issues in a block without reports, for property-based testing

Development Swift Testing
1

Hi,

I'm working on a library for property-based testing (a.k.a. QuickCheck) designed for the new Swift Testing framework, as an up-to-date replacement for SwiftCheck. It currently supports running tests with random inputs, while also being able to supply a specific random seed to reproduce failing cases.

One aspect of property-based testing that Swift Testing is currently not suited for is shrinking, which involves calling a failing test repeatedly with different inputs to find a minimal failing case. The closest function is withKnownIssue(), but this will still report every suppressed issue as a "known issue". Finding the actual Issue I want to display will also add around 20 "known" Issues which are irrelevant to the output.

I'd like Swift Testing to add a way to suppress all issues being recorded inside a block, while also returning which issues were suppressed (or at least the amount of issues). I could then use it to run all tests, and in the case of failures, run them again while changing the inputs. Finally, run it without the suppression to actually record all issues.

7 Likes
2

Before I make a pitch for a new top-level function in Swift Testing that (selectively) allows for issues to be completely erased: What's a good way to persuade developers to avoid this new function in favor of withKnownIssue, unless it's absolutely necessary?

Hiding it behind an @_spi attribute isn't going to work for third-party packages, since no SPI's are shipped with the Testing package included in the Swift framework.

3

Hi Lennard,

Would the proposed [Pitch] Test Issue Warnings be able to help you here?

There is an implementation you could try out.

4

It doesn't look like this fits my use case, because I want to be able to modify all issues before they get recorded (regardless of their severity).

Here's an example of how I plan to use it, with most implementation details omitted:

func testBlock(input: Int) {
    #expect(input < 10000) // contrived example that uses regular #expect or #require macros
}

let runCount = 100
for _ in 0..<runCount {
    let input: Int.random() // Repeatable RNG omitted here for brevity
    var hasFailed = false // should actually be Mutex
    suppressAllIssues(in: {
        testBlock(input: input)
    }, onIssue: { issue in
        // issue must NOT be recorded here or appear in the test output.
        hasFailed = true
    }

    if hasFailed {
        let shrunkValue = input.shrink(using: testBlock) // custom function that also uses suppressAllIssues()
        // shrunkValue is now 10000, or slightly above this value
        
        // Run it once more without suppressing issues, to report the actual test failure
        testBlock(input: shrunkValue)
        break
    }
}
5

By coincidence, earlier today I posted [Pitch] Issue Handling Traits. Reviewing your use case here, it's possible you may be able to combine that feature with ST-0007: Test Scoping Traits and achieve the goal you're looking for.

I'd encourage you to have a look at proposal and chime in on the pitch thread to say whether or not you think it could help!

1 Like