Async/await and XCTest

Using Swift
,
1

Of course, we're still far from getting async/await into release (as well as actors, structured concurrency etc), but there is one thing I would love to have early on just to try on what we have already — ability to test async code.

Sure, async syntax might change, and therefore it's not reasonable to update XCTest right away, but what about a temporary hack to enable us testing stuff as usual?

I tried this, but got an expected compile error:

func asyncTest<T>(closure: () async throws -> T) throws -> T {
    var result: Result<T, Error>

    let group = DispatchGroup()
    group.enter()

    _ = Task.runDetached {
        do {
            result = .success(try await closure()) // Mutation of captured var 'result' in concurrently-executing code
        } catch {
            result = .failure(error) // Mutation of captured var 'result' in concurrently-executing code
        }
        group.leave()
    }

    group.wait()

    switch result {
    case let .success(res): return res
    case let .failure(err): throw err
    }
}

Of course, this check is reasonable, but in this particular case I know what I'm doing (or do I? vsauce.mp3).

Is there any way of force-syncing an async piece of code?

2

The async toolchains have a temporary runAsyncAndBlock global function you can use to test things, you just need to put everything inside the closure.

1 Like
3

Hm. TBH I didn't think of using runAsyncAndBlock for that purpose... Maybe because it's not throwing (_runAsyncMain is though). I'm afraid it might break XCTest runtime. Will try, however. Thank you.

4

You are supposed to use either waitForExpectations or XCTWaiter instead of DispatchGroup. You don't even need async/await for it.

5

This isn’t about generally testing async APIs but working with the new async APIs, for which you need an async context.

6

Async/await is just syntactic sugar for completion block, that's how the Objective C API will be generated: swift-evolution/NNNN-concurrency-objc.md at concurrency-objc · DougGregor/swift-evolution (github.com)

What I'm saying is that XCTest already has tools to "wait until async function is completed (or timeout reached)", so if you have a completion based API (that you will switch to async/await), you can already test that just like how you would test async/await the same way OP wrote their example code but instead of DispatchGroup, they should just use the proper XCTest API that has been existed for a long time.

7

Nothing from XCTest is automatically imported in a way that's useful for async APIs. So in the short term, using the async context workaround with runAsyncAndBlock works fine.

8

OK so _runAsyncMain doesn't work at all in tests, runAsyncAndBlock only allows non-throwing closures, force trying functions isn't good, and XCTAssertNoThrow doesn't work with async functions, obviously. Sigh.

9

Maybe it’s time for SwiftTest - a new Swift only Unit testing Framework?

Actually, I like Catch2 very much with its SECTIONS - which are much simpler and more powerful than classical setup/ teardown fixtures.

1 Like
10

What you did in the original post is almost right. You just need to use the proper, already existing, XCTest functions that I linked yesterday.

11

I kinda got it working eventually, but of course it's far from even acceptable:

public extension XCTestCase {
    func asyncTest(
        expectationDescription: String? = nil,
        timeout: TimeInterval = 3,
        file: StaticString = #file,
        line: Int = #line,
        closure: @escaping () async throws -> ()
    ) {
        let expectation = self.expectation(description: expectationDescription ?? "Async operation")

        Task.runDetached {
            do {
                try await closure()
                expectation.fulfill()
            } catch {
                XCTFail("Error thrown while executing async function @ \(file):\(line): \(error)")
            }
        }

        self.wait(for: [expectation], timeout: timeout)
    }
}

FYI: [SR-14403] async/await support in XCTest · Issue #347 · apple/swift-corelibs-xctest · GitHub

12

IIRC calling XCTFail on another thread can cause some other issues, so you will have to go with your previous solution. (But with XCTExpectation instead of DispatchGroup)

(But I still don't understand how you "kinda got it working eventually" when you almost had a working solution and I told you exactly how to make it work)

So something like this:

var result: Result<T, Error>?
let expectation = self.expectation(description: expectationDescription ?? "Async operation")
Task.runDetached {
    do {
        result = .success(try await closure())
    } catch {
        result = .failure(error)
    }
    expectation.fulfill()
}
self.wait(for: [expectation], timeout: timeout)
switch result {
case let .success(result)
    return result
case let .failure(error):
    throw error
case nil:
    throw "Async function timed out"
}
13

Have you tried it? It won't compile. Maybe wrapping Swift.Result in a class will work, but still it's extremely ugly, not to mention I'm too lazy to create async versions of assertion functions.

14

You need an initial value before you capture it. It works with var result: Result<T, Error>?

15

sorry to resurrect this thread, but the solutions here don't seem to work any more:

  • runAsyncAndBlock is gone
  • detach executes the contents on an arbitrary thread, so it's unsafe to use XCTAssert within it I think?
  • It's now an error to mutate a captured variable from concurrent code (there's lots of solutions once you're already in an async context, but I don't see how to get data out of detach at the moment without writing my own unsafe locking code?

It really feels like XCTest needs an update to handle the new async stuff. Ideally you could just write async func testBlah() { ... XCTAssert(...) } and have XCTest itself manage all the details (using a task-local for XCTAssert to contribute to the results of the correct test, for example).

2 Likes
16

It's an error when you are trying to capture the variable like this:

let result: Result<T, Error>

It works when you capture it like this (because this gets initialized to nil):

var result: Result<T, Error>?

Waiting for expectations makes sure that there won't be any concurrent accesses (if you only access it before fulfilling the expectation and continueOnFailure is set to false)

17

I understand that it's (probably) dynamically safe, but the compiler won't allow it any longer.

18

I've just found out about this issue myself. This is bothersome.

I have solved this issue for now by disabling async tests, I can't merge to main something that may or may not work in future without a previous notice.

IMHO the XCTest needs to be updated.

1 Like
19

I started working on it myself (out of curiosity, with no intent of creating a PR; either way, I'm sure it's already in work somewhere inside Apple), but I have no clue how to integrate updated XCTest into toolchain (Xcode?)

20

Xcode uses the Objective-C version of XCTest that has been incorporated as part of Xcode that pre-dates Swift and can be used with the C-style languages as well as Swift. The XCTest that is part of the Swift repository is a version of XCTest for Swift only that is intended for Linux, Windows, FreeBSD, etc., so that testing on those platforms mimics the XCTest package on MacOS as much as possible, given that the Objective-C run-time is not available on those platforms. I think you can use that version on Darwin, but, usage has to be outside of the Xcode environment, in particular, xcodebuild does not use it, as far as I know.