# \[Mini-Vision\] Rethinking expectation capture in Swift Testing

**URL:** <https://forums.swift.org/t/mini-vision-rethinking-expectation-capture-in-swift-testing/77075>\
**Category:** Swift Testing\
**Tags:** swift-syntax, macros, swift-testing\
**Created:** [January 8, 2025, 8:16pm UTC](https://forums.swift.org/t/mini-vision-rethinking-expectation-capture-in-swift-testing/77075 "2025-01-08T20:16:58Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![grynspan](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/grynspan/32/15315_2.png) [@grynspan](https://forums.swift.org/u/grynspan)\
**Post date:** [January 8, 2025, 8:16pm UTC](https://forums.swift.org/t/mini-vision-rethinking-expectation-capture-in-swift-testing/77075/1 "2025-01-08T20:16:58Z")

</div>

Happy New Year, my festive Swifty fellows! Have I got a late Hanukkah present\[1\] for you!

Since we announced Swift Testing and shipped it to developers with the Swift 6 toolchain, we've been looking at how we could improve the library. Some changes we hope to introduce, like exit tests and attachments, are major new features. Others are smaller quality-of-life changes (bug fixes, performance improvements, and so on.)

### ℹ Be advised

This post assumes some knowledge and understanding of Swift macros and how they work. In particular, you should know that when the Swift compiler encounters an _expression_ of the form `#m(a, b, c)`, it passes that expression's _abstract syntax tree_ (AST) to a compiler plugin that then replaces it with an equivalent AST known as its _expansion_.

## Our expectations for expectations

One area we knew we'd want to revisit was the `#expect()` and `#require()` macros and how they work. `#expect()` and `#require()` are more than just functions: they're _expression macros_. We wrote them as expression macros so that we could give them some special powers.

When you use one macro or the other, it examines the AST of its condition argument and looks for one of several known kinds of expression (e.g. binary operators, member function calls, or `is`/`as?` casts). If found, the macro rewrites the expression in a form that Swift Testing can examine at runtime. Then, if the expectation fails, Swift Testing can produce better diagnostics than it could if it just treated the condition expression as a boolean value.

For example, if you write this expectation:

```swift
#expect(f() < g())

```

… Swift Testing is able to tell you the results of `f()` and `g()` in addition to the overall expression `f() < g()`.

### Effectively ineffective macros

This works fairly well for simple expressions like that one, but it doesn't have the flexibility needed to tell a test author what goes wrong when a more complex expression is used. For example, this expression:

```swift
#expect(x && y && !z)

```

… is subject to a transformation called "operator folding" that takes place prior to macro expansion, and the AST represents it as a binary operator whose left-hand operand is _another_ binary operator. Swift Testing doesn't know how to recursively expand the nested binary operator, so it only produces values for `x && y` and `!z`.

We also run into trouble when effects (`try` and `await`) are in play. Swift Testing's implementation can't readily handle arbitrary combinations of subexpressions that may or may not need either keyword. As a result, it doesn't try to expand expressions that contain effects. If `f()` or `g()` is a throwing function:

```swift
#expect(try f() < g())

```

… Swift Testing will not attempt to do any further processing, and only the outermost expression (`try f() < g()`) will be captured.

## Looking forward

Now that Swift 6.1 has branched for its upcoming release, we can start to look at the _next_ Swift version and how we can improve Swift Testing to help make it the _Awesomest Swift Release Ever_. And we'd like to start by revisiting how we've implemented these macros.

We've been working on [a branch](https://github.com/swiftlang/swift-testing/tree/jgrynspan/162-redesign-value-capture) of Swift Testing (with a corresponding [draft PR](https://github.com/swiftlang/swift-testing/pull/840)) that completely redesigns the implementation of the `#expect()` and `#require()` macros. Instead of trying to sniff out an "interesting" expression to expand, the code on this branch walks the AST of the condition expression and rewrites _all_ interesting subexpressions.

This expectation:

```swift
#expect(x && y && !z)

```

Can now be fully expanded and will provide a full breakdown of the condition at runtime if it fails:

```swift
◇ Test example() started.
✘ Test example() recorded an issue at Example.swift:1:2: Expectation failed: x && y && !z → false
↳ x && y && !z → false
↳ x && y → true
↳ x → true
↳ y → true
↳ !z → false
↳ z → true
✘ Test example() failed after 0.002 seconds with 1 issue.

```

## How you can help

Before we merge this PR and enable these changes in prerelease Swift toolchains, we’d love it if you could try it out! These changes are a major change for Swift Testing and the more feedback we can get, the better. To try out the changes:

1. _Temporarily_ add an explicit package dependency on Swift Testing and point Swift Package Manager or Xcode to the branch. In your Package.swift file, add this dependency:

2. Once you’ve added the package dependency, clean your package (`swift package clean`) or project ( **Product** → **Clean Build Folder…** ), then build and run your tests.

3. Let us know how your experience goes, especially if you run into problems. You can reach me by sending me [a forum DM](https://forums.swift.org/u/grynspan/summary) or by commenting on [this PR](https://github.com/swiftlang/swift-testing/pull/840).

## Here be caveats

There are some expressions that Swift Testing could previously successfully expand that will cause problems with this new implementation:

- Expectations with effects where the effect keyword is to the left of the macro name:

- Expectations with particularly complex conditions can, after expansion, overwhelm the type checker and fail to compile:

- Type names are (syntactically speaking) indistinguishable from variable names. That means there may be some expressions that we _could_ expand further, but because we can't tell if a syntax node refers to a variable, a type, or a module, we don't try:

- The `==`, `!=`, `===`, and `!==` operators are special-cased so that we can use [`difference(from:)`](https://developer.apple.com/documentation/swift/bidirectionalcollection/difference(from:)) to compare operands where possible. The macro implementation assumes that these operators eagerly evaluate their arguments (unlike operators like `&&` that short-circuit their right-hand arguments using `@autoclosure`.) This assumption is true of all implementations of these operators in the Swift Standard Library, but we can't make any real guarantees about third-party code.

### Disabling expression expansion

If you have a condition expression that you don't want expanded at all (for instance, because the macro is incorrectly expanding it, or because its implementation might be affected by side effects from Swift Testing), you can cast the expression with `as Bool` or `as T?`:

```swift
let x = ...
let y = ...
#expect((x == y) as Bool)

let z: String?
let w = try #require(z as String?)

```

## Example expansions

Here (hidden behind disclosure triangles so as not to frighten children and pets) are some before-and-after examples of how Swift Testing expands the `#expect()` macro. I've cleaned up the whitespace to make it easier to read.

> **\#expect(f() \< g())**
>
> #### Before
> 
> ```swift
> Testing.__checkBinaryOperation(
> f(),
> { $0 < $1() },
> g(),
> expression: .__fromBinaryOperation(
> .__fromSyntaxNode("f()"),
> "<",
> .__fromSyntaxNode("g()")
> ),
> comments: [],
> isRequired: false,
> sourceLocation: Testing.SourceLocation.__here()
> ).__expected()
> 
> ```
> 
> #### After
> 
> ```swift
> Testing.__checkCondition(
> { ( __ec: inout Testing.__ ExpectationContext) -> Swift.Bool in
> __ec(__ec(f(), 0x2) < __ec(g(), 0x400), 0x0)
> },
> sourceCode: [
> 0x0: "f() < g()",
> 0x2: "f()",
> 0x400: "g()"
> ],
> comments: [],
> isRequired: false,
> sourceLocation: Testing.SourceLocation.__here()
> ).__expected()
> 
> ```

> **\#expect(x && y && !z)**
>
> #### Before
> 
> ```swift
> Testing.__checkBinaryOperation(
> x && y,
> { $0 && $1() },
> !z,
> expression: .__fromBinaryOperation(
> .__fromSyntaxNode("x && y"),
> "&&",
> .__fromSyntaxNode("!z")
> ),
> comments: [],
> isRequired: false,
> sourceLocation: Testing.SourceLocation.__here()
> ).__expected()
> 
> ```
> 
> #### After
> 
> ```swift
> Testing.__checkCondition(
> { ( __ec: inout Testing.__ ExpectationContext) -> Swift.Bool in
> __ec(__ec(__ec(x, 0x6) &&__ec(y, 0x42), 0x2) && __ec(!__ec(z, 0x1400), 0x400), 0x0)
> },
> sourceCode: [
> 0x0: "x && y && !z",
> 0x2: "x && y",
> 0x6: "x",
> 0x42: "y",
> 0x400: "!z",
> 0x1400: "z"
> ],
> comments: [],
> isRequired: false,
> sourceLocation: Testing.SourceLocation.__here()
> ).__expected()
> 
> ```

> **\#expect(try f() \< g())**
>
> #### Before
> 
> ```swift
> Testing.__checkValue(
> try f() < g(),
> expression: .__fromSyntaxNode("try f() < g()"),
> comments: [],
> isRequired: false,
> sourceLocation: Testing.SourceLocation.__here()
> ).__expected()
> 
> ```
> 
> #### After
> 
> ```swift
> try Testing.__checkCondition(
> { ( __ec: inout Testing.__ ExpectationContext) -> Swift.Bool in
> try __ec(__ec(f(), 0xc) < __ec(g(), 0x1004), 0x4)
> },
> sourceCode: [
> 0x4: "f() < g()",
> 0xc: "f()",
> 0x1004: "g()"
> ],
> comments: [],
> isRequired: false,
> sourceLocation: Testing.SourceLocation.__here()
> ).__expected()
> 
> ```

### What's with the hexadecimal?

You'll note that all calls to `__ec()` include an integer literal argument, and the `sourceCode` argument is a dictionary whose keys are integer literals too. These values represent the unique identifiers of each captured syntax node from the original AST. They uniquely encode the syntax nodes' positions in the tree so that we can reconstruct the (sparse) tree at runtime when a test fails.

I find this subtopic interesting enough to want to devote a whole forum thread to it, personally, but it's a bit arcane—for more information, see the implementation [here](https://github.com/swiftlang/swift-testing/blob/jgrynspan/162-redesign-value-capture/Sources/Testing/SourceAttribution/ExpressionID.swift) or feel free to reach out to me via forum DM.

* * *

1. Or, if you prefer, a late Christmas present! (Or, if you prefer, a very early [_Grunnlovsdagen_](https://en.wikipedia.org/wiki/Constitution_Day_(Norway)) present?) 

2. @smontgomery has opened [a PR](https://github.com/swiftlang/swift-syntax/pull/2724) against swift-syntax that we hope will help resolve this issue.