Reliably testing code that adopts Swift Concurrency?

[Joe_Groff]

@stephencelis

Are you suggesting this as a workaround for things today, or are you suggesting that this is how async-await code should be written?

It feels really, really unfortunate to need to introduce ad hoc, test-only code and interleave it through a feature’s suspension points just to get a reliably passing test. This is also much more onerous than typical dependency injection, which has a minimal footprint on the logical reasoning of code, and is useful for more than just testing (Xcode previews, demos, offline testing, etc.). The sole purpose of the code you’ve added to the feature is for testing, and because async-await doesn’t provide a reliable testing experience by default.

I picked AsyncChannel because it's a readily available tool that didn't require much rearchitecting of the example code you had written, but I think that in general, robustly observing the execution of async code is going to need some explicit coordination. I don't think you'd need to, and you really shouldn't, try to instrument every suspend point, only the points where an event that's interesting to test occurred. And if an event is interesting to test, then it's likely also interesting for logging, analytics, tracing, or other instrumentation too. I may have added the code for testing, but you could use the channel to dependency-inject any other tracing you might want to do for this code, so it doesn't seem so different in nature to using DI in synchronous code to both improve testability and get all the other benefits of doing so like you said. But where traditional DI often requires a lot of restructuring and refactoring to get good results, the impact of the instrumentation I added is relatively minimal to the structure of your original code, on the level of inserting print or os_log statements.

@stephencelis

I also don’t quite understand how this code guarantees a passing test. The .didSubscribeToScreenshots progress point is sent to the channel before the actual subscription starts.

I'll have to defer to @Philippe_Hausler here. If the subscription isn't guaranteed to actually be in effect before notifications(named:) returns, then I agree that being able to get that guarantee seems like something Foundation should provide (and it should be documented that the subscription may be delayed in the API docs).

@stephencelis

Shouldn’t async-await aspire to give us these probe points “for free” as well, either with a single-threaded/serial mode, executor overrides, or otherwise? I find it odd that in order to test a simple async-await feature I need to introduce test-only code and thread it between each line that might have an async hop.

Trying to test every single suspension point is overkill for most application code. Whether and how often most code suspends is an implementation detail that'll be constantly changing, and even if you try to minimize the nondeterminism in the system with a mock execution environment, you'll be fighting library, compiler, and runtime changes that move otherwise uninteresting suspension points around.

@stephencelis

As a side note, this line does not currently compile:

async let _ = vm.task()

The variable must be named for some reason, which gives us an unused variable warning, which we can’t even silence:

async let task = vm.task()
_ = task // 🛑 Expression is 'async' but is not marked with 'await'

Should I file a bug?

Ah, that's too bad async let _ gets rejected; being able to fire-and-forget a scoped child task seems useful enough that we should let that work. _ = await task would be the correct way to explicitly await the task at the end of its scope, though.