Swift-test-kit: Composable property-based testing, stateful testing, structured diffs, and more for Swift Testing and XCTest

Community Showcase
, , ,
1

swift-test-kit extends Swift Testing and XCTest with composable test evaluators, advanced assertions, structural diffs, and expression capture. The package offers two libraries with identical APIs: SwiftTestKit and XCTestKit. SwiftTestKit integrates with Swift Testing, and XCTestKit integrates with XCTest, so they can easily be added to existing test suites.

Features

  • Advanced assertions with customizable path-based diffs and expression capture.

  • Atomic testing: Group assertions into a single atomic evaluation and verify that all assertions pass within a single execution.

  • Performance testing: Measure execution time and physical memory footprint across multiple runs, and verify that median values stay within configurable limits.

  • Temporal testing: Poll assertions over a configurable duration to verify continuous invariants or eventual convergence.

  • Property-based testing: Describe properties that must hold for any given value, and swift-test-kit will generate random test cases automatically.

  • Stateful testing: Extend property-based testing to systems with mutable state. Rather than testing individual values against a property, stateful testing generates random sequences of commands and executes them against both a simplified model and the real system, verifying consistency at each step.

  • Composable tests and assertions.

  • Built-in conformance for many standard library types.

  • Macros to automatically synthesize conformance for property-based and stateful testing.

Structured Diffs

swift-test-kit assertion failures produce path-based diff output providing clear insight into where values differ within complex data structures.

Comparing Structs

struct Inner: Equatable
{
    let id      : Int
    let value   : Int
    let label   : String
}

struct Outer: Equatable
{
    let tag     : String
    let inner   : Inner
}

let expected    = Outer(tag: "a", inner: Inner(id: 1, value: 100, label: "b"))
let actual      = Outer(tag: "a", inner: Inner(id: 1, value: 200, label: "b"))

XCTKAssertEqual(expected, actual)

// XCTKAssertEqual failed
//
// Outer differs at:
//
//     .inner.value
//         Expected:   100
//         Actual:     200

Comparing Strings

let expected    = "Line 1\nLine 2\nLine 3"
let actual      = "Line 1\nLine X\nLine 3"

XCTKAssertEqual(expected, actual)

// XCTKAssertEqual failed
//
// String differs at:
//
//     line 2
//         Expected:   "Line 2"
//         Actual:     "Line X"
//         Changed:    character 6 ("2" → "X")

Diffs support any Equatable type, including recursive/indirect types and generics. Types that conform to CustomDiffRepresentable and CustomDiffStringConvertible can customize their diff representation.

Expression Capture

Every swift-test-kit assertion has a macro equivalent. Macro assertions capture the literal source text of expressions for use in failure output, which can be helpful when reviewing CI logs without immediate access to the source code.

For boolean macro assertions, compound expressions using && and || are decomposed to show the value of each sub-expression and identify which caused the assertion failure, respecting short-circuit evaluation so only evaluated operands appear in the output.

#XCTKAssertFalse((a || b) && (c || d))
// where a == true, b == false, c == true, d == false

// #XCTKAssertFalse failed
// 
// Expression: (a || b) && (c || d)
// 
//     a = true ←
//     c = true ←
// 
//     (2 expressions not evaluated)

Property-Based Testing

Describe properties that must hold for any given value, and swift-test-kit will generate random test cases automatically.

When a value causes a property to fail, swift-test-kit will shrink the value to the smallest value that still fails the property (the minimal counterexample).

swift-test-kit assertions are automatically intercepted inside property bodies, so counterexamples include the same diff output, expression capture, and formatting used by standalone assertions.

The counterexample is reported along with the seed used for generation. To deterministically reproduce the failure, set PropertyOptions.seed or the TEST_KIT_SEED environment variable. The failing values can also be pinned as examples to prevent regressions.

func customSort(_ array: [Int]) -> [Int] { /* ... */ }

XCTKForAll
{
    (array: [Int]) in

    // Assert that a custom sort function is working correctly.
    XCTKAssertSorted(customSort(array), by: >=)
}

// XCTKForAll failed after 4 iterations (shrunk in 2 steps)
// 
// Counterexample:
//     Array<Int> = [1, 0]
// 
// XCTKAssertSorted failed
// 
// Collection count: 2
// 
// Not sorted at:
// 
//     [0]: 0
//     [1]: 1
// 
// Seed: 2188239925673862914 (XCTKForAll)

Apply the @Arbitrary macro to a struct or enum to automatically synthesize Arbitrary conformance for custom types.

Generic parameters that appear in stored properties or associated values are automatically constrained to Arbitrary. Recursive and indirect enums are supported.

@Arbitrary
struct User<T>: Equatable where T : Equatable & Hashable
{
    let name    : String
    let id      : T
}

XCTKForAll
{
    (user: User<UUID>) in

    // Test properties that must hold for any user.
}

Property-based testing also includes:

  • Classification of properties.
  • A Generator to produce specific distributions of values, such as only positive integers, or only non-empty arrays.
  • Targeted property-based testing to guide generation toward values that maximize a target metric, testing edge cases and worst-case behavior that random generation alone is unlikely to reach.

Composable Testing

Property-based tests, stateful tests, temporal tests, performance tests, and atomic tests compose freely. Any evaluator may be nested inside any other evaluator, and all evaluators can wrap standalone assertions. Any failures propagate with the same rich output used by standalone assertions.

// Assert that any random array is correctly sorted within 50 milliseconds.
await XCTKForAll
{
    (array: [Int]) in

    await XCTKPerformance(timeLimit: .milliseconds(50))
    {
        XCTKAssertSorted(customSort(array), by: >=)
    }
}

let pipeline = Pipeline()
pipeline.start()

// Assert that all stages are eventually ready at the same time.
await XCTKEventually(timeout: .seconds(5))
{
    XCTKAtomic
    {
        XCTKAssertEqual(.ready, pipeline.stage1)
        XCTKAssertEqual(.ready, pipeline.stage2)
        XCTKAssertEqual(.ready, pipeline.stage3)
    }
}

See the documentation and README for the complete API reference. swift-test-kit is licensed under the Apache License, Version 2.0.