Testing Closure-based asynchronous APIs

Related Projects swift-testing
1

I've been trying to migrate my existing XCTest based suites to Testing and whenever I do swift concurrency dependent tests, everything seems fine. However, if I need to test some closure based APIs, then I can't find a way.

Let's imagine we have a ViewModel class with a sync interface function isolated to the MainActor that looks like this (for the sake of the simplicity of the example):

final class MyViewModel {

    let analyticsTracker: AnalyticsTracker
    init(analyticsTracker: AnalyticsTracker) {
        self.analyticsTracker = analyticsTracker
    }

    @MainActor
    func start() {
        Task {
            // something here
            await somethingAsync()
            // something here
            analyticsTracker.track(event: .init(name: "SomeEvent", params: [:]))
        }
    }

    private func somethingAsync() async {}
}

And I have a behavioral mock for the analytics tracker that tracks all the invocations of it's functions.

And my XCTest looks like this:

@MainActor
func test_example() async throws {
    // create sut here
    let expectation = expectation(description: "Async expectation")
    env.analyticsTracker.trackEventMock.whenCalled { _ in
            expectation.fulfill()
    }
    sut.start()
    await fulfillment(of: [expectation], timeout: 0.001)
// do the checks here
    XCTAssert(...)
}

I can't get my head around how I can translate the same to Swift Testing.

confirmation doesn't seem to work as I expected (or maybe I had a wrong expectation)

// I can't add `MainActor` here, because confirmation will not compile then
func test_example() async throws {
// create sut here ...
    await confirmation { confirmed in
            env.analytics.trackEventMock.whenCalled { _ in
                confirmed()
            }
            // this should be awaited now as we're not on the main actor
            await sut.start()
    }
    #expect(env.analytics.trackErrorMock.calledOnce)
}

In this case the confirmation never gets fired.
Is this a known issue or am I making wrong assumptions on how to test asynchronous closure-based code?

I'm on Xcode beta 4 now and I'm using built-in Testing framework (not fetching it as a swift package from Github).

5 Likes
2

Hi @SergeyPetrachkov, this code looks like the correct usage of confirmation to me. If you put a breakpoint in the track endpoint is it definitely being called? Could there be something wrong with the whenCalled helper you have? Are you able to make a small, standalone repro of the problem you are seeing to share?

One thing that stands out, and this may just be from you copy-pasting code, but your test is not marked with @Test, which is necessary to run the test. Is it definitely annotated with the macro in your test suite?

2 Likes
3

Oh yes, it's marked with the Test macro indeed. It's just a copy-paste mistake.
I'll create a sample repo and attach it here.

2 Likes
4

Are you getting a compiler error when you try this? Can you post the specific error? That may or may not be related to the core issue you’re seeing but I’d like to understand why that isn’t working, at least.

2 Likes
5

When given a nonzero timeout, the fulfillment(of:timeout:) API from XCTest spins the run loop and waits (up to the specified timeout) for the XCTestExpectation to become fulfilled. In contrast, the confirmation() API from Swift Testing does not wait; it requires its Confirmation to be confirmed before its closure returns.

In your example, the start() function creates an unstructured Task, but doesn't await its value. That's a fine pattern to use, but it means that when start() returns, the work performed asynchronously in the body of that Task may not have executed. To guarantee that, you would need to modify the code to ensure that the confirmation() closure only returns once that Task has finished running. One possible way you could do that would be to return the Task from start():

@MainActor
@discardableResult
func start() -> Task<...> {
    Task {
        // ...
    }
}

and then wait for it to finish in the confirmation closure:

await confirmation { confirmed in
    env.analytics.trackEventMock.whenCalled { _ in
        confirmed()
    }
    let startingTask = await sut.start()
    await startingTask.value
}
4 Likes
6

If I put MainActor on top of the @Test function I get "Sending main actor-isolated value of type (Confirmation) async -> () with later accesses to nonisolated context risks causing data races"

7

Nice insight about XCTest spinning the run loop!

Unstructured Task usage was a deliberate choice for the architectural approach that I'm using (though I understand that it may not be the best). I wonder if it's possible to adopt the Swift Testing without having to rewrite the project :)

Here's the sample code that I am hoping to get to work:

8

You can find the isolated example here GitHub - SergeyPetrachkov/SwiftTestingAndClosureIssueSample

9

In this example I am firing the confirmation when my analytics tracker gets called (and it's in the end of the Task execution). So, in this case it should work, but for some reason it doesn't. I can get to the breakpoint where the confirmation is called, but the counter doesn't increment for some reason and the test fails.

10

One other potential POV that might work here is for a package version of the start to accept a TaskGroup. Your test code can create a TaskGroup and wait on that:

func doSomething() -> Task<Void, Never> {
  Task {
    await withTaskGroup(of: Void.self) { group in
      doSomething(with: &group)
    }
  }
}

func doSomething(with group: inout TaskGroup<Void>) {
  group.addTask {
    print("doSomething")
  }
}

@Test func test() async throws {
  let task = doSomething()
  await task.value
}

@Test func testWithGroup() async throws {
  await withTaskGroup(of: Void.self) { group in
    doSomething(with: &group)
    await group.waitForAll()
  }
}
11

This is theoretical, but in reality I am limited by the necessity of creating a ViewModel that is concurrency-model agnostic. In other words:
the class has an interface which is synchronous and is only accessed from View layer. View sends signals to the ViewModel via this interface (func start() in the very simple case). ViewModel starts a Task, or if I have to work with older parts of the app, it uses Combine or even good old closure-based APIs. When the result of an async operation is there, it updates the state and propagates it to the View layer via @Published property or Observation framework.
So, with XCTest it was easy to create an expectation for something inside my ViewModel/Interactor/whatever, wait for it and then assert the results.
I expected confirmation of Swift Testing to work in a similar way, but I cannot get it to work unfortunately, which makes the adoption of the framework hard.
So, I wonder if there's a way. Otherwise many engineers will have to consider the Testing only for the newer projects where they can still tweak architectural approaches to be able to use the framework.

P.S. Please take a look at the link I posted above if you have time. It pretty much describes the case.

1 Like
12

Why can’t you modify at the view level to create unstructured task there? It should be almost no difference for the view (just wrapping call in a Task), and would allow you to test just async methods on view model.

13

It's not a single function of course that we're talking about :) So this will bring a massive change to the project that consists of hundreds modules and hundreds of thousands LoC :) Not an easy migration for the sake of using a new testing framework :)

1 Like
14

If you really want to "block" in a way similar to XCTest… you could probably try and roll your own blocking logic on CheckedContinuation. Just keep in mind that you have to be careful with continuations not to deadlock (but this is test code so a deadlock would just timeout a test instead of crash your app in production).

15

Good idea, I wonder though if that can become a part of the framework, as I am 99.999% sure I am not the only one who needs this :D

16

There is no one "right" answer here… but I am of the opinion that production code that needs a continuation (or a task sleep) to pass tests should (when possible) be refactored in a way that test code has more control of the concurrency model. IMO the test framework should not encourage blocking on continuation with a first-party API (but engineers can choose use that technique by-hand if that's really their best option).

1 Like
17

Yeah, I agree that changing architecture just for the sake of tests doesn’t sounds good. But if the goal is to migrate, in a large project that would still be an iterative work.

As for this case, I find tests that require fulfillments (in case of XCTest) more complex and tend to avoid, that’s why altering interface seems preferable for me. But I understand that this is still a burden when that won’t give any benefits to the app.

You can try wrapping your closure-based API in (don’t know if there more suitable one) AsyncStream, then await for it after the start called. Haven’t checked, but this might work. It seems to work:

struct SwifyTests {
    // ...

    @Test("The issue")
    @MainActor
    func testExample() async {
        let env = Environment()
        let sut = env.makeSUT()
        env.analyticsTracker.trackEventMock.returns()

        let stream = AsyncStream { continuation in
            env.analyticsTracker.trackEventMock.whenCalled { _ in
                continuation.yield()
            }
        }
        sut.start()
        var iterator = stream.makeAsyncIterator()
        await iterator.next()

        #expect(env.analyticsTracker.trackEventMock.calledOnce)
        #expect(sut.viewState.title == "Loaded!")
    }
}

No warnings either for @MainActor or last line that was commented out.

1 Like
18

When we put it this way, the migration won't probably become a goal :) I really tried to evaluate the new framework, what it can and what it can't do before I bring it on for discussion at the company, so we could pioneer in this. But the cost of such migration will be either too high in the moment, or will take too long. Especially when we don't have issues with XCTest.

I don't find expectations-based tests too hard to read and write. It's a common pattern when unit-testing VIP/VIPER/MVVM layers' behaviors. Every company I worked for and every team used those.
Of course when it comes to testing pure business logic, we don't need those, as we can either fully grasp the swift concurrency model, or flatten the closures.
Unfortunately, it's not the case here :(

19

Oh, thanks for the code sample! I'll check it out in the morning, when I get to my laptop. The AsyncStream seems like an elegant solution

20

We believe this is a compiler issue, however it may end up being something we need to resolve on Swift Testing's side. We're investigating this in Swift-Testing Swift 6 Strict Concurrency Legit Build Failure? · Issue #74882 · swiftlang/swift · GitHub.

2 Likes