Expectation failure messages are less useful with `await`

Related Projects swift-testing
1

If you write something like

#expect(status.current == .attached)

then you’ll get a failure message of

Expectation failed: (current → .detached) == .attached

that is, it shows you information about current which helps you to
understand why the expectation failed.

But if, on the other hand, you want to test an asynchronous property, and you write

#expect(await status.current == .attached)

then you’ll get a failure message of

Expectation failed: await status.current == .detached

which does not help you to understand why the expectation failed. Is this a known issue, and are there plans to improve it?

2

It is a known issue, and we do not anticipate making a change here at this time. #expect() is a macro and macros in Swift can see syntax, but can't see type or effects information. In order to correctly expand an await or try expression, we need to know exactly which subexpressions require those keywords so that we can reapply them in the right places in the macro's expanded form.

For more discussion, see Value not shown in error message when calling async variant of #expect · Issue #162 · swiftlang/swift-testing · GitHub.

1 Like
4

I see, thank you!

5

(For those who come across this in the future, the linked GitHub issue made me realise that in my case I can actually just move the await outside the #expect and this solves it.)

2 Likes
6

That will break in other ways depending on the expression you're writing, because macros cannot see syntax nodes (e.g. keywords) outside of themselves, so we don't know anything's going to be awaitable. :slight_smile: Tread with caution.

7

Do you think you could give an example of a usage that might "break in other ways", please? It might help me understand whether I should switch back to the previous way I was doing things :slight_smile:

1 Like
8

If I write something like await #expect(true), then I get a (helpful) compiler warning of "No 'async' operations occur within 'await' expression", which I would expect. Is that what you're referring to or am I way off the mark?

9

Consider this trivial example test:

func f() async -> Bool {
  true
}

@Test func g() async {
  await #expect(f())
}

This function fails to compile with a warning and an error:

:warning: No 'async' operations occur within 'await' expression
:stop_sign: 'async' call in a function that does not support concurrency

By default, the #expect() macro tries to break down condition expressions in order to get the complex failure messages that we all love so dearly. Because Swift Testing cannot tell that g() needs an await keyword when called, it doesn't emit one in the expansion of await #expect(f()), so it ends up calling f() without an await keyword (generating the error) while the expanded form of the macro is not itself async (generating the warning.)

1 Like
10

I see. So, my understanding is that:

  • trying to use await or try before #expect may or may not cause a compiler error, and it's hard for an average user to predict whether it will for a given use case
  • but if in a particular use case it doesn’t cause a compiler error, then there is no harm in doing so (and will give better failure messages)

Is that right?

11

The behaviour is undefined; in the future, we expect to change it to always behave as if the keyword were inside the parentheses (and skip expanding the expression) so at best you're looking at a temporary quirk of the library here, not a permanent feature.

If you want to get the full expansion reliably, consider moving await outside of #expect():

let currentStatus = await status.current
#expect(currentStatus == .attached)
1 Like
12

Got it, thank you!