Exhaust: Cross-platform property-based testing, stateful testing and fuzzing

Community Showcase
, , , ,
1

Exhaust is a feature-rich, macro-driven testing library that works with both Swift Testing and XCTest. Its aim is to empower you to test the values you didn't think of.

It is the library I've wanted to have in Swift for years, ever since I started using SwiftCheck: one that matches the ergonomics and test case reduction of Python's Hypothesis with the performance of Swift. A library where you can get started with three lines of code and scale that all the way to coverage-guided fuzzing.

Exhaust enables what's known as "property-based testing" (PBT), where you can test your code against generated values and receive minimal, simple-to-understand counterexamples.

Getting started

The simplest test to get started with is something like this: does my code throw or crash? You'd be surprised.

#exhaust(#gen(.string())) { input in
    var parser = Parser(input)
    var count = 0
    // Walk forward
    while parser.reachedEnd() == false {
        try parser.advance()
        count += 1
    }
    // Walk backward
    while (try? parser.retreat()) != nil {
        count -= 1
    }
    #expect(count == 0)
}

Stateful testing

Stateful systems can be tested using declarative state machine tests. These support sequential, thread-based and task-based modes, enabling you to interrogate the consistency of your code under sequential and concurrent access.

@Test("Atomic dictionary under concurrent access")
func testAtomicDictionary() async {
    await #execute(
      AtomicDictionarySpec.self, 
      mode: .threads, 
      .parallelize(lanes: .two)
    )
}

@StateMachine
final class AtomicDictionarySpec {
    static let idGen = #gen(.int(in: 0...5))

    @SystemUnderTest
    var dict: AtomicDict = .init()

    @Equivalence
    func matches(sequential: AtomicDict) -> Bool {
        // Underlying dictionaries
        dict.store == sequential.store
    }

    @Command(idGen)
    func get(key: Int) -> String? {
        dict[key]
    }

    @Command(idGen)
    func set(key: Int) {
        dict[key] = key.description
    }
    
    func failureDescription() -> String? { … }
}

Coverage-guided fuzzing

Generators and state machine specs can both be tested using coverage-guided fuzzing with #explore(…, time:) and the appropriate build flags. Unless specified with .failFast, #explore will continue after a failure and reduce and cluster faults by their counterexamples and suspected file location.

// With generator
await #explore(#gen(…), time: .hours(5)) {
    …
}
// With a @StateMachine spec
await #explore(AnySpec.self, mode: .tasks, time: .hours(5))

Test fixtures on tap

Generators can be written by hand, or synthesised from any Decodable instance. #example can be used to create any number of instances of the generator's output.

let personGen = #gen(.uint(in: 0...130), .string(), .string()) {
    Person(age: $0, first: $1, last: $2)
}

let person = try #example(personGen)
let throng = try #example(personGen, count: 50_000)

// …Or synthesise one via `Decodable` instance
let person = Person(age: 43, first: "C", last: "K")
let synthGen = try #gen(from: person)

Give it a go!

There's a lot more detail available in the documentation to help you get started.

If you try it and it breaks, or if the API fights you, I'd love to hear about it.

5 Likes