Probing a.k.a. breakpoints for Swift Testing - precise control over side effects and fully observable state transitions in asynchronous functions

Community Showcase
, ,
1

I’m thrilled to introduce Probing - my most ambitious project to date, and my first serious venture into the world of open source!

Excerpts from README:

What Problem Probing Solves?

Testing asynchronous code remains challenging, even with Swift Concurrency and Swift Testing. Some of the persistent difficulties include:

  • Unobservable state transitions: When invoking methods on objects, often with complex dependencies between them, it’s not enough to inspect just the final output of the function. Inspecting the internal state changes during execution, such as loading states in view models, is equally important but notoriously difficult.
  • Non-determinism: Task instances run concurrently and may complete in different orders each time, leading to unpredictable states. Even with full code coverage, there’s no guarantee that all execution paths have been reached, and it's' difficult to reason about what remains untested.
  • Limited runtime control: Once an asynchronous function is running, influencing its behavior becomes hard. This limitation pushes developers to rely on ahead-of-time setups, like intricate mocks, which add complexity and reduce clarity of the test.

Over the years, the Swift community has introduced a number of tools to address these challenges, each with its own strengths:

  • Quick/Nimble: Polling with the designated matchers allows checking changes to object state, but it can lead to flaky tests and is generally not concurrency-safe.
  • Combine/RxSwift: Reactive paradigms are powerful, but they can be difficult to set up and may introduce unnecessary abstraction, especially now that AsyncSequence covers many use cases natively.
  • ComposableArchitecture: Provides a robust approach for testing UI logic, but it’s tightly coupled to its own architectural patterns and isn’t suited for other application layers.

These tools have pushed the ecosystem forward and work well within their intended contexts. Still, none provide a lightweight, general-purpose way to tackle all of the listed problems that embraces the Swift Concurrency model.

That's why I have designed and developed Probing.

How Probing Works?

The Probing package consists of two main modules:

  • Probing, which you add as a dependency to the targets you want to test
  • ProbeTesting, which you add as a dependency to your test targets

With Probing, you can define probes - suspension points typically placed after a state change, conceptually similar to breakpoints, but accessible and targetable from your tests. You can also define effects, which make Task instances controllable and predictable.

Then, with the help of ProbeTesting, you write a sequence of dispatches that advance your program to a desired state. This flattens the execution tree of side effects, allowing you to write tests from the user’s perspective, as a clear and deterministic flow of events:

@Test
func testLoading() async throws {
    try await withProbing {
        await viewModel.load()
    } dispatchedBy: { dispatcher in
        #expect(viewModel.isLoading == false)
        #expect(viewModel.download == nil)

        try await dispatcher.runUpToProbe()
        #expect(viewModel.isLoading == true)
        #expect(viewModel.download == nil)

        downloaderMock.shouldFailDownload = false
        try await dispatcher.runUntilExitOfBody()
        #expect(viewModel.isLoading == false)
        #expect(viewModel.download != nil)

        #expect(viewModel.prefetchedData == nil)
        try await dispatcher.runUntilEffectCompleted("backgroundFetch")
        #expect(viewModel.prefetchedData != nil)
    }
}

ProbeTesting also includes robust error handling. It provides recovery suggestions for every error it throws, guiding you toward a solution and making it easier to get started with the API.

Examples

The CHANGED and ADDED comments highlight how the view model in the examples has been adapted to support testing with ProbeTesting. As you can see, the changes are minimal and don’t require any architectural shift.

Observing State Transitions

// ViewModel.swift

func uploadImage(_ item: ImageItem) async {
    do {
        uploadState = .uploading
        await #probe() // ADDED
        let image = try await item.loadImage()
        let processedImage = try await processor.processImage(image)
        try await uploader.uploadImage(processedImage)
        uploadState = .success
    } catch {
        uploadState = .error
    }

    await #probe() // ADDED
    try? await Task.sleep(for: .seconds(3))
    uploadState = nil
}

// ViewModelTests.swift

@Test
func testUploadingImage() async throws {
    try await withProbing {
        await viewModel.uploadImage(ImageMock())
    } dispatchedBy: { dispatcher in
        #expect(viewModel.uploadState == nil)

        try await dispatcher.runUpToProbe()
        #expect(uploader.uploadImageCallsCount == 0)
        #expect(viewModel.uploadState == .uploading)

        try await dispatcher.runUpToProbe()
        #expect(uploader.uploadImageCallsCount == 1)
        #expect(viewModel.uploadState == .success)

        try await dispatcher.runUntilExitOfBody()
        #expect(viewModel.uploadState == nil)
    }
}

Just-in-Time Mocking

// ViewModel.swift

func updateLocation() async {
    locationState = .unknown
    await #probe() // ADDED

    do {
        for try await update in locationProvider.getUpdates() {
            try Task.checkCancellation()

            if update.authorizationDenied {
                locationState = .error
            } else if let isNear = update.location?.isNearSanFrancisco() {
                locationState = isNear ? .near : .far
            } else {
                locationState = .unknown
            }
            await #probe() // ADDED
        }
    } catch {
        locationState = .error
    }
}

// ViewModelTests.swift

@Test
func testUpdatingLocation() async throws {
    try await withProbing {
        await viewModel.beginUpdatingLocation()
    } dispatchedBy: { dispatcher in
        #expect(viewModel.locationState == nil)

        locationProvider.continuation.yield(.init(location: .sanFrancisco))
        try await dispatcher.runUpToProbe()
        #expect(viewModel.locationState == .near)
        
        locationProvider.continuation.yield(.init(location: .init(latitude: 0, longitude: 0)))
        try await dispatcher.runUpToProbe()
        #expect(viewModel.locationState == .far)
        
        locationProvider.continuation.yield(.init(location: .sanFrancisco))
        try await dispatcher.runUpToProbe()
        #expect(viewModel.locationState == .near)
        
        locationProvider.continuation.yield(.init(location: nil, authorizationDenied: true))
        try await dispatcher.runUpToProbe()
        #expect(viewModel.locationState == .error)
        
        locationProvider.continuation.yield(.init(location: .sanFrancisco))
        try await dispatcher.runUpToProbe()
        #expect(viewModel.locationState == .near)

        locationProvider.continuation.finish(throwing: ErrorMock())
        try await dispatcher.runUntilExitOfBody()
        #expect(viewModel.locationState == .error)
    }
}

Controlling Side Effects

// ViewModel.swift

private var downloadImageEffects = [ImageQuality: any Effect<Void>]() // CHANGED

func downloadImage() {
    downloadImageEffects.values.forEach { $0.cancel() }
    downloadImageEffects.removeAll()
    downloadState = .downloading

    downloadImage(withQuality: .low)
    downloadImage(withQuality: .high)
}

private func downloadImage(withQuality quality: ImageQuality) {
    downloadImageEffects[quality] = #Effect("\(quality)") { // CHANGED
        defer {
            downloadImageEffects[quality] = nil
        }

        do {
            let image = try await downloader.downloadImage(withQuality: quality)
            try Task.checkCancellation()
            imageDownloadSucceeded(with: image, quality: quality)
        } catch is CancellationError {
            return
        } catch {
            imageDownloadFailed()
        }
    }
}

// ViewModelTests.swift

@Test
func testDownloadingImage() async throws {
    try await withProbing {
        await viewModel.downloadImage()
    } dispatchedBy: { dispatcher in
        await #expect(viewModel.downloadState == nil)

        try await dispatcher.runUntilExitOfBody()
        #expect(viewModel.downloadState?.isDownloading == true)

        try await dispatcher.runUntilEffectCompleted("low")
        #expect(viewModel.downloadState?.quality == .low)

        try await dispatcher.runUntilEffectCompleted("high")
        #expect(viewModel.downloadState?.quality == .high)
    }
}

@Test
func testDownloadingImageWhenHighQualityDownloadSucceedsFirst() async throws { ... }

@Test
func testDownloadingImageWhenHighQualityDownloadFailsAfterLowQualityDownloadSucceeds() async throws { ... }

@Test
func testDownloadingImageRepeatedly() async throws { ... }

// ...

I believe there’s a lot of potential in this idea. That said, I also understand that introducing dedicated test-support code into applications can be polarizing. Fortunately, Swift Macros go a long way toward making it more digestible, by stripping it away in non-debug builds (by default).

Still, I think the examples speak for themselves - they show just how clean and sequential reasoning about asynchronous execution becomes with Probing and ProbeTesting.

If you’re curious, I encourage you to check it out on GitHub:

If you have any questions or suggestions, I’m here for them!

15 Likes
2

While working on Probing, I had a chance to explore some of the less obvious behaviors of the new Mutex type - much of which was discussed across this forum. If anyone finds it useful, I’ve compiled my findings into a blog post - Peculiarities of Mutex - NSFatalError.

6 Likes
3

@NSFatalError this is excellent and much needed work, thank you for building this!

2 Likes
4

Thank you so much for this comment and all the reactions, that's very encouraging :smiley:. This is the starting point - my vision is to essentially enable TCA-levels of testability, but for every app layer, regardless of the architecture used.

Apple’s sample code usually demonstrates the most natural way to work with their technologies, but typically skips comprehensive test coverage. I feel that in the "real" world, that clarity can get lost when developers adopt complex architectures - and I believe testability often becomes the main driver of architectural choices.

You can already find the groundwork for upcoming features in the Probing repository:

  • With the DeeplyCopyable and EquatableObject macros, I’ll be implementing a feature called object expectations (or object probes — name still TBD). It will work similarly to exhaustivity checks in TCA. In your tests, you’d mutate only the test object, and ProbeTesting would verify at key points whether those mutations fully represent the changes made to the original reference (e.g. a view model).
  • I’m also planning macros that would generate proprietary mocks, specifically tailored for use with Probing. This means you’ll rarely need to declare probes manually - most will be embedded in the mocks for you.
  • I want to add better support for debugging effect hierarchies, enabling you to print the entire execution tree and inspect the state of individual nodes.
  • And finally, I’ll keep working on documentation - and I’m considering recording some video content as well.

Stay tuned!

2 Likes
5

@NSFatalError This is a very exciting project! I think there is a lot of potential in this area.

I've been toying with a similar proof-of-concept, but focusing more on controlled side effects and just-in-time mocking, as you call it (great name, btw!). Essentially, the idea is to express effects as Free Monads (glorified enums with Continuations as associated values) and bridge them into normal async function calls using Macros. I haven't got time to flesh out the implementation yet, and I've been struggling with some synchronization challenges, so I'm excited to dig into your code and learn from you, as you seem to have good solutions.

I've outlined this concept in my two recent talks, in case you're interested in more details:

I think there needs to be a stronger community push behind this line of research, as I'm convinced that we need to move away from effect systems that are based on Unidirectional Data Flow - and suffer greatly from message ping-ponging - to effect systems that are based on interruptible and resumable direct-style async function calls (Algebraic effects, etc).

I'll be eagerly following your work in this area!

Best, Alex

2 Likes
6

Thanks for the support, @Alex-Ozun! I watched your conference talk, and it seems we’re aligned on many goals in this space.

In your presentation, you're capturing very well how powerful testing becomes when we can simulate runtime conditions inline with the test. That’s exactly the direction I’m heading with Probing and the upcoming mocking functionality I've mentioned. Async functions naturally provide a way to inject continuations to the tested code and await callbacks from tests, as you've showed yourself. Here’s the direction I’m heading (pseudocode):

try await withProbing {
    await viewModel.asyncFunction()
} dispatchedBy: { dispatcher in
    #expect(viewModel.someState == ...)
    try await dispatcher.runThrough(
        someMock.someFunctionThatIsExpected(arg: 1, arg2: "foo", returning: …)
    )

    #expect(viewModel.someState == ...)
    try await dispatcher.runThrough(
        someOtherMock.someOtherFunctionThatIsExpected(...)
    )

    // ...
}

This syntax addresses several pain points with typical mocking:

  • Keeps test logic inline with the flow of the code
  • Verifies call order to dependencies
  • Ensures each method is called exactly once
  • Asserts passed arguments automatically

While Probing primarily focuses on async testing, I also want to support mocking for synchronous code, as using continuations there won't be possible. Making a function async just to enable testing is a no-go in my book. I believe developers should only need to do the bare minimum to make their code testable. Ideally, the testing infrastructure should be flexible enough that developers structure their code purely for readability and maintainability of production code, not to satisfy the test framework. Changing a function’s signature just for testing crosses that line for me.

For sync calls, result builders are best I could think of right now (pseudocode):

withExpectations {
    someMock.someFunctionThatIsExpected(arg: 1, arg2: "foo", returning: …)
    someOtherMock.someOtherFunctionThatIsExpected(...)
} perform: {
    viewModel.syncFunction()
}

// Or, as part of async operation

try await withProbing {
    await viewModel.asyncFunction()
} dispatchedBy: { dispatcher in
    try await dispatcher.runThrough(
        someMock.someFunctionThatIsExpected(arg: 1, arg2: "foo", returning: …),
        someOtherMock.someOtherFunctionThatIsExpected(...) // stop after that
    )

    // ...
}

I recognize that verifying the order of calls to dependencies isn’t always necessary, so I want to support those use cases as well. But at the same time, I want to make it easy to enable that check when maintaining call order may be important.

This is still early-stage, so any insights, concerns, or use cases you’d like to see supported are very welcome!

7

Congratulations. This will be a valuable tool for concurrency testing. I developed a similar tool for TOPS-20 (in the 80s) for testing simultaneous client updates for our relational database product.

We found the need for a couple of features that were critical for our testing. I will describe them using your terminology.

The first feature you already defined with probe identifiers. The other may be more challenging, or you may have a method to support. We could continue a specific probe while keeping the others paused, wait for another probe from that path, and then release the others to verify the result. We supported various permutations of this control flow, enabling us to debug complex situations effectively. I encounter similar situations in asynchronous code due to reentrancy, as well as when testing actor-to-actor interactions. In the future, this may also be helpful when testing server-client situations and distributed actors.

I look forward to using your package. Thanks.

1 Like
8

Thanks, @Dlemex! I believe the kind of scenarios you’re referring to are already testable with Probing, though I haven’t highlighted them directly in the examples yet.

Each effect is independently controllable while all others remain suspended exactly where they left off. That means you can simulate reentrancy and interleaving by explicitly progressing some effects while holding others. For example:

// MyActor.swift

actor MyActor {
    func someReenteringMethod() async {
        someState = ...
        await #probe()
        await someOtherActor.someMethod()
        someState = ...
    }
}

// MyActorTests.swift

try await withProbing {
    #Effect("first") {
        await myActor.someReenteringMethod()
    }
    #Effect("second") {
        await myActor.someReenteringMethod()
        // Does not have to be the same method
    }
} dispatchedBy: { dispatcher in
    try await dispatcher.runUpToProbe(inEffect: "first")
    #expect(...)
    try await dispatcher.runUpToProbe(inEffect: "second")
    #expect(...)
    try await dispatcher.runUntilEffectCompleted("first") // or "second"
    #expect(...)
    try await dispatcher.runUntilEffectCompleted("second") // or "first"
    #expect(...)
}

By creating multiple effects like this, you can test any permutation you need. Let me know if you’d like help with a specific example.

2 Likes
9

This is great. This keeps the bulk of the support inside the test. I believe I also saw that probes could be named as well. I envision putting a named probe in each state of my state machine and then running up to the state I want to perform tests on.

This package will be great. Thanks.

10

Absolutely! By naming probes, you can advance execution to a desired state in a single call:

try await withProbing {
    await #probe("1") // id: "1"
    print("1")
    await #probe("2") // id: "2" <- SUSPENDED
    print("Not called until dispatch is given.")
} dispatchedBy: { dispatcher in
    try await dispatcher.runUpToProbe("2")
    // Always prints:
    // 1
}

Named probes can also be placed inside effects. When runUpToProbe is called for such a probe, execution progresses until all parent effects are created along the path and suspends them at next available probe (or keeps them in their previous state if the effect has been created before by some other dispatch):

try await withProbing {
    await #probe("1") // id: "1"
    print("1")
    #Effect("first") { // <- SUSPENDED
        print("Not called until dispatch is given.")
    }
    #Effect("second") {
        await #probe("1") // id: "second.1"
        print("second.1")
        await #probe("2") // id: "second.2" <- SUSPENDED
        print("Not called until dispatch is given.")
    }
    await #probe("2") // id: "2" <- SUSPENDED
    print("Not called until dispatch is given.")
} dispatchedBy: { dispatcher in
    try await dispatcher.runUpToProbe("second.2")
    // Always prints:
    // 1
    // second.1
}

You can find more examples like these in the documentation of individual ProbingDispatcher methods.

1 Like