Revisiting reliable testing of concurrent code

Using Swift
,
1

Hello everyone!

I'd like to revisit the topic of reliably testing concurrent Swift code and gather your opinions. The original discussion is a few years old now, the follow-up on task executors is also no longer new, and quite a lot has landed since. I think it is worth taking stock of what is possible today, what still seems difficult, and whether the available tools cover concurrency testing well enough.

The testing problem

A common pattern in async tests is that we want to check an ordering like this:

  1. Start some async work.
  2. Wait until it reaches some point.
  3. Trigger a relevant dependency.
  4. Assert what happened before and after.

In practice, tests often approximate step 2 with Task.yield() or Task.sleep(). Those tests may pass locally and then fail under load, on CI, or after unrelated runtime changes.

The Point-Free team gave a great demonstration of this problem in their series of videos "Testing async code" back in 2023.

For context, let's start with a deliberately simple ProfileProvider:

struct Profile: Codable, Equatable {
    let name: String
}

struct ProfileProvider {
    let httpClient: HTTPClient

    func loadProfile() async throws -> Profile {
        try await httpClient.get("/profile", as: Profile.self)
    }
}

There is nothing difficult to test here. A simple HTTPClient stub can return a profile immediately (or throw), and the test can await loadProfile() and assert the result.

Now let's add a common piece of concurrent behavior:

Given a profile request is already in progress,
When another caller requests the profile before the first request completes,
Then only one HTTP request is performed and both callers receive its result.

The interesting part is the test: the two loadProfile() calls must overlap. Whichever call reaches the provider first needs to remain suspended until both calls have had a chance to run. Otherwise, the first call may finish before the second one starts, so we cannot use a classic stub with data prepared in advance.

It is easy to end up writing something timing-based. Before implementing request sharing, I ran the following test against the original ProfileProvider. It failed in all 10,000 repetitions, as expected, with no false passes:

@Test func `Loads share the same process`() async throws {
    let expectedProfile = Profile(name: "test")
    let (clientResult, clientResultContinuation) = AsyncStream.makeStream(
        of: Profile.self,
        bufferingPolicy: .bufferingNewest(0),
    )
    let getHandlerCount = Mutex(0)
    let httpClient = HTTPClientDouble(getHandler: { _ in
        if getHandlerCount.withLock({ $0 += 1; return $0 == 1 }) {
            try #require(await clientResult.first(where: { _ in true }))
        } else {
            Profile(name: "unexpected")
        }
    })
    let provider = ProfileProvider(httpClient: httpClient)
    
    async let taskOne = provider.loadProfile()
    async let taskTwo = provider.loadProfile()
    
    await Task.yield() // hope both calls have started
    
    clientResultContinuation.yield(expectedProfile)
    clientResultContinuation.finish()
    
    let (profileOne, profileTwo) = try await (taskOne, taskTwo)
    
    #expect(getHandlerCount.withLock(\.self) == 1)
    #expect(profileOne == expectedProfile)
    #expect(profileOne == profileTwo)
}

The clientResult stream should keep the first loadProfile() call suspended, whether that is taskOne or taskTwo. HTTPClientDouble is a more general-purpose test double that delegates the result of get to getHandler. This lets the test suspend the first loadProfile() call and stub any subsequent calls. I use the getHandlerCount mutex because confirmation { _ in } would not provide the same flexibility here.

Here is the implementation intended to satisfy this test:

actor ProfileProvider: Sendable {
    enum State {
        case pending
        case inProgress(Task<Profile, any Error>)
    }
    
    private let httpClient: HTTPClient
    private var state: State = .pending
    
    init(httpClient: HTTPClient) {
        self.httpClient = httpClient
    }
    
    func loadProfile() async throws -> Profile {
        if case .inProgress(let task) = state {
            return try await task.value
        }
        
        let task = Task { try await httpClient.get("/profile", as: Profile.self) }
        state = .inProgress(task)
        
        let result = await task.result
        state = .pending
        
        return try result.get()
    }
}

In a few local experiments, running the test 10,000 times against the new implementation produced a success rate of about 95%. The Task.yield() above is unreliable: the test usually passes, but it is flaky.

There are two failure scenarios:

  1. Neither task has started by the time clientResult is finished because one Task.yield() was not enough. This accounts for most of the failures. The clientResult stream makes this clearly visible because it is configured as a pass-through sequence (.bufferingNewest(0)): if no consumer is waiting, the element is dropped. Otherwise the element can be buffered and the test may pass without exercising the intended behavior.

  2. The first task completes just as the second one starts. This is much rarer because whichever task runs first suspends while awaiting clientResult.

The test needs to represent a clear relationship between the two tasks, but it cannot do so reliably. Solving the first scenario is not difficult with additional manual coordination, such as an expectation. The second scenario is the bigger challenge because there is no side effect that tells us when the second task has started.

What we have now

In this section, I'll point to tools that improve the reliability of the test above.

ConcurrencyExtras helpers

  • Replacing Task.yield() with Task.megaYield() helps in practice, but repeatedly yielding still cannot prove that the second task reached the relevant state. It also adds measurable overhead.

  • Wrapping the test in withMainSerialExecutor {} and using a single Task.yield() worked better in this experiment and was faster:

Coordination Time for 10,000 repetitions
withMainSerialExecutor + Task.yield() ~2 seconds
Task.megaYield() ~6 seconds

These numbers are from a local experiment rather than a general benchmark, but they show that repeatedly yielding is not free.

withMainSerialExecutor has a more important cost, though. It temporarily overrides the global hook, swift_task_enqueueGlobal_hook. Swift Testing runs tests in parallel by default, and the hook is not scoped to the test. Jobs enqueued by unrelated tests during that window are also redirected to the main executor. I put together a small reproduction of this cross-test interference with more details.

TaskExecutor and SerialExecutor

TaskExecutor and SerialExecutor together make an interesting combination: you can route the work of the system under test through your own queue and observe it directly. For this test, much like with withMainSerialExecutor, we can apply withTaskExecutorPreference using NaiveQueueExecutor and a serial DispatchQueue, as follows:

@Test func `Loads share the same process`() async throws {
    let queue = DispatchQueue(label: #function)
    try await withTaskExecutorPreference(NaiveQueueExecutor(queue)) {
        // test content
    }
}

In my local experiment, this version completed 10,000 repetitions successfully in around one second. Unlike withMainSerialExecutor, the executor preference is test-scoped and no cross-test interference.

For this example, task executor preference with a serial executor seems like the best option. It provides test-scoped execution control, is faster than the other approaches, and makes the required ordering reliable.

What I am less sure about is how far this approach generalises. The clearest next boundary for me is code that combines execution ordering with time, such as retries or debouncing. Those tests need to control virtual time as well as the order in which work runs. Two limitations seem relevant here:

I also run into related coordination problems when testing AsyncSequence or systems with many dependencies. Swift Async Algorithms' AsyncSequenceValidation is interesting prior art, and I have experimented with AsyncStream of Sendable closures to describe more complex event sequences in tests. Those cases deserve concrete examples of their own, though, rather than being treated as the same problem in this opening post.

When working with Swift concurrency, I keep encountering the same trade-off: a test either relies on timing (Task.sleep(), Task.megaYield()), serialises jobs to control execution ordering, or explicitly coordinates each event and accumulates a lot of manual machinery.

In practice, I can only use or combine the latter two options because test speed and reliability are essential.

What I would like to hear

  1. Do you encounter tests like this? If so, do you see the same trade-offs, and what approaches do you use?
  2. Is a test-scoped serial TaskExecutor the approach you would use for this kind of test?
  3. Have I missed any relevant capabilities, prior art, or ongoing work?

Mostly I want to understand how far the tools available today can take us before discussing or proposing a particular API.

Thanks for reading.

8 Likes
2

Task.immediate solves some problems here, since it runs synchronously until the first suspension. So in your example, replacing

    async let taskOne = provider.loadProfile()
    async let taskTwo = provider.loadProfile()
    
    await Task.yield() // hope both calls have started

with

    let taskOne = Task.immediate { await provider.loadProfile() }
    let taskTwo = Task.immediate { await provider.loadProfile() }

Should probably resolve the issue?

3 Likes
3

Adopting nonisolated(nonsending) (or NonisolatedNonsendingByDefault) can also eliminate suspension points and the need to yield for tests.

As you pointed out, task executor preference does not seem to be fully completed (last time I checked task group tasks were still scheduled in a non-deterministic order with preference overridden in a serial manner).

3 Likes
4

Thanks for the suggestion, @KeithBauerANZ! I tried this approach:

let taskOne = Task.immediate { try await provider.loadProfile() }
let taskTwo = Task.immediate { try await provider.loadProfile() }

await Task.yield()
    
clientResultContinuation.yield(expectedProfile)
clientResultContinuation.finish()

let (profileOne, profileTwo) = try await (taskOne.value, taskTwo.value)

Across 10,000 repetitions, the test passed 97% of the time. In the failing runs, neither task had reached the point where the HTTP client was awaiting clientResult before the stream was finished. This is similar to the first failure scenario described in the original post.

I think the important distinction is between the immediate task starting and the actor-isolated loadProfile() call starting. Task.immediate starts executing on the caller’s context until it suspends. However, try await provider.loadProfile() crosses the actor boundary, so the task can suspend before the body of loadProfile begins.

I’m curious whether a different implementation could help, for example, by moving away from the actor and using nonisolated(nonsending) where appropriate. I haven’t had much luck with that so far.

1 Like
5

We have had good luck with a "sendable shell, non-sendable core"[1] design, where an actor isolates a large amount of state and business logic, all of which is wrapped in a nonisolated class with nonisolated(nonsending) endpoints, and where all unstructured async work is kicked off using Task.immediate. The shell actor can be @MainActor or arbitrary by overriding the executor. Tests can be almost 100% synchronous (with a few exceptions still being worked on in the Standard Library, like clock APIs that aren't yet nonisolated(nonsending)).

Because provider is an actor and I assume loadProfile is isolated to it, I can think of 2 ways to close the gap:

  • Allow the actor's unownedExecutor to be overridden to match the actor of the test. This will force serialization and potentially result in Task.yield() deterministically cycling to the next scheduled job
  • Once you override the unownedExecutor you can synchronously enter the actor itself via assumeIsolated, and that could eliminate the need for the yield entirely.

  1. Functional Core, Imperative Shell ↩︎

2 Likes
6

Thank you for the ideas, @stephencelis! I made several adjustments to the test and implementation, and they allowed me to remove await Task.yield().

ProfileProvider remains an actor, but its unownedExecutor can be supplied externally. The state and core logic have been moved into a non-Sendable ProfileProviderCore class.

My testing SPM package enables Approachable Concurrency and uses nonisolated as its default isolation. Consequently, ProfileProviderCore is nonisolated and its asynchronous loadProfile() method is implicitly nonisolated(nonsending).

The implementation also uses Task.immediate when starting the HTTP request:

struct Profile: Codable, Equatable {
    let name: String
}

enum ProfileProviderState {
    case pending
    case inProgress(Task<Profile, any Error>)
}

actor ProfileProvider {
    
    nonisolated let unownedExecutor: UnownedSerialExecutor
    
    private let core: ProfileProviderCore
    
    private var state: ProfileProviderState {
        core.state
    }
    
    init(
        httpClient: HTTPClient,
        unownedExecutor: UnownedSerialExecutor,
    ) {
        self.core = .init(httpClient: httpClient)
        self.unownedExecutor = unownedExecutor
    }

    func loadProfile() async throws -> Profile {
        try await core.loadProfile()
    }
}

private final class ProfileProviderCore {
    
    let httpClient: HTTPClient
    var state: ProfileProviderState = .pending
    
    init(httpClient: HTTPClient) {
        self.httpClient = httpClient
    }
    
    func loadProfile() async throws -> Profile {
        if case .inProgress(let task) = state {
            return try await task.value
        }
        
        let task = Task.immediate { [httpClient] in
            try await httpClient.get("/profile", as: Profile.self)
        }
        state = .inProgress(task)
        
        let result = await task.result
        state = .pending
        
        return try result.get()
    }
}

The test now uses Task.immediate for both calls to loadProfile(). It runs on a custom global actor, TestActor, and ProfileProvider is configured to use TestActor.sharedUnownedExecutor, so, the test and ProfileProvider share the same serial executor:

@TestActor
@Test func `Loads share the same process`() async throws {
    let expectedProfile = Profile(name: "test")
    let (clientResult, clientResultContinuation) = AsyncStream.makeStream(
        of: Profile.self,
        bufferingPolicy: .bufferingNewest(0),
    )
    let getHandlerCount = Mutex(0)
    
    let httpClient = HTTPClientDouble(getHandler: { _ in
        return if getHandlerCount.withLock({ $0 += 1; return $0 == 1 }) {
            try #require(await clientResult.first(where: { _ in true }))
        } else {
            Profile(name: "unexpected")
        }
    })
    let provider = ProfileProvider(
        httpClient: httpClient, 
        unownedExecutor: TestActor.sharedUnownedExecutor,
    )
    
    let taskOne = Task.immediate {
        try await provider.loadProfile()
    }
    let taskTwo = Task.immediate {
        try await provider.loadProfile()
    }
    
    clientResultContinuation.yield(expectedProfile)
    clientResultContinuation.finish()
    
    let (profileOne, profileTwo) = try await (taskOne.value, taskTwo.value)
    
    #expect(getHandlerCount.withLock(\.self) == 1)
    #expect(profileOne == expectedProfile)
    #expect(profileOne == profileTwo)
}

I ran this test repeatedly 100,000 times without observing a failure.

The following parts appear to be important for the stability of this particular test:

  • The test and the system under test share the same serial executor, such as the executor belonging to TestActor or MainActor.
  • Task.immediate is used both by the test and when starting the HTTP request. Replacing either use with Task, or replacing the test tasks with async let, reintroduces nondeterministic scheduling.
  • The core operation is exposed through a nonisolated(nonsending) asynchronous method, so it continues executing on the caller’s isolation rather than introducing another executor hop.

My understanding of the resulting execution order is:

  1. The first immediate task enters ProfileProvider and starts the HTTP operation.
  2. The HTTP operation reaches the controlled suspension on clientResult.
  3. The second immediate task enters ProfileProvider and finds the existing in-progress task.
  4. Only after both immediate tasks have reached their suspension points does control return to the test.
  5. The test supplies the HTTP result.

Therefore, the test no longer needs to yield and hope that both calls have started.

Some broader observations:

  • Once an actor explicitly provides unownedExecutor, I do not see a way to fall back to the actor’s synthesized default executor. This makes executor injection a meaningful production-design decision rather than a test-only override.
  • withTaskExecutorPreference with NaiveQueueExecutor did not replace the TestActor plus custom actor executor combination. I'm still looking into it, the following didn't work so far:
@Test func `Loads share the same process`() async throws {
    try await withTaskExecutorPreference(executor) {
        // ...
        let provider = ProfileProvider(
            httpClient: httpClient,
            unownedExecutor: executor.asUnownedSerialExecutor(),
        )
        // ...
    }
}
  • The relationship between these mechanisms is still fairly subtle. Although the execution order above now has a concrete steps, the amount of implementation structure required raises questions about how well the approach scales to larger systems, particularly those involving several asynchronous dependencies.

Overall, it is encouraging that this test can now be written without timing-based coordination or substantial manual plumbing.

1 Like
7

It's actually not safe for ProfileProvider to hold onto the unownedExecutor directly. For a global actor that may be fine, since they are alive for the duration of the process, but generally you would want to hold onto the actor that provides the executor instead. An "unowned" executor can go away when the owner goes away and crash your application.

 actor ProfileProvider {
-  nonisolated let unownedExecutor: UnownedSerialExecutor
+  let isolation: any Actor
+
+  nonisolated var unownedExecutor: UnownedSerialExecutor {
+    isolation.unownedExecutor
+  }

Once you do that, you can take the actor in the init, and fall back to a trivial actor for default execution:

 init(
   httpClient: HTTPClient,
-  unownedExecutor: UnownedSerialExecutor
+  isolation: any Actor = DefaultIsolation()
 ) {
   self.core = .init(httpClient: httpClient)
-  self.unownedExecutor = unownedExecutor
+  self.isolation = isolation
 }
+
+private actor DefaultIsolation {}
1 Like