[Mini-Vision] Rethinking expectation capture in Swift Testing

Development Swift Testing
, ,
1

Happy New Year, my festive Swifty fellows! Have I got a late Hanukkah present[1] for you!

Since we announced Swift Testing and shipped it to developers with the Swift 6 toolchain, we've been looking at how we could improve the library. Some changes we hope to introduce, like exit tests and attachments, are major new features. Others are smaller quality-of-life changes (bug fixes, performance improvements, and so on.)

:information_source: Be advised

This post assumes some knowledge and understanding of Swift macros and how they work. In particular, you should know that when the Swift compiler encounters an expression of the form #m(a, b, c), it passes that expression's abstract syntax tree (AST) to a compiler plugin that then replaces it with an equivalent AST known as its expansion.

Our expectations for expectations

One area we knew we'd want to revisit was the #expect() and #require() macros and how they work. #expect() and #require() are more than just functions: they're expression macros. We wrote them as expression macros so that we could give them some special powers.

When you use one macro or the other, it examines the AST of its condition argument and looks for one of several known kinds of expression (e.g. binary operators, member function calls, or is/as? casts). If found, the macro rewrites the expression in a form that Swift Testing can examine at runtime. Then, if the expectation fails, Swift Testing can produce better diagnostics than it could if it just treated the condition expression as a boolean value.

For example, if you write this expectation:

#expect(f() < g())

… Swift Testing is able to tell you the results of f() and g() in addition to the overall expression f() < g().

Effectively ineffective macros

This works fairly well for simple expressions like that one, but it doesn't have the flexibility needed to tell a test author what goes wrong when a more complex expression is used. For example, this expression:

#expect(x && y && !z)

… is subject to a transformation called "operator folding" that takes place prior to macro expansion, and the AST represents it as a binary operator whose left-hand operand is another binary operator. Swift Testing doesn't know how to recursively expand the nested binary operator, so it only produces values for x && y and !z.

We also run into trouble when effects (try and await) are in play. Swift Testing's implementation can't readily handle arbitrary combinations of subexpressions that may or may not need either keyword. As a result, it doesn't try to expand expressions that contain effects. If f() or g() is a throwing function:

#expect(try f() < g())

… Swift Testing will not attempt to do any further processing, and only the outermost expression (try f() < g()) will be captured.

Looking forward

Now that Swift 6.1 has branched for its upcoming release, we can start to look at the next Swift version and how we can improve Swift Testing to help make it the Awesomest Swift Release Ever. And we'd like to start by revisiting how we've implemented these macros.

We've been working on a branch of Swift Testing (with a corresponding draft PR) that completely redesigns the implementation of the #expect() and #require() macros. Instead of trying to sniff out an "interesting" expression to expand, the code on this branch walks the AST of the condition expression and rewrites all interesting subexpressions.

This expectation:

#expect(x && y && !z)

Can now be fully expanded and will provide a full breakdown of the condition at runtime if it fails:

◇ Test example() started.
✘ Test example() recorded an issue at Example.swift:1:2: Expectation failed: x && y && !z → false
↳ x && y && !z → false
↳   x && y → true
↳     x → true
↳     y → true
↳   !z → false
↳     z → true
✘ Test example() failed after 0.002 seconds with 1 issue.

How you can help

Before we merge this PR and enable these changes in prerelease Swift toolchains, we’d love it if you could try it out! These changes are a major change for Swift Testing and the more feedback we can get, the better. To try out the changes:

  1. Temporarily add an explicit package dependency on Swift Testing and point Swift Package Manager or Xcode to the branch. In your Package.swift file, add this dependency:

    dependencies: [
  /* ... */ 
  .package(
    url: "https://github.com/swiftlang/swift-testing.git",
    branch: "jgrynspan/162-redesign-value-capture"
  ),
],

    And update your test target:

    .testTarget(
  name: "MyTests",
  dependencies: [
    /* ... */ 
    .productItem(name: "Testing", package: "swift-testing"),
  ]
)

    If you’re using an Xcode project, you can add a package dependency via the Package Dependencies tab in your project’s configuration. Add the Testing target as a dependency of your test target and click Trust & Enable to use the locally-built TestingMacros target.

  2. Once you’ve added the package dependency, clean your package (swift package clean) or project (Product → Clean Build Folder…), then build and run your tests.

    Swift Testing will be built from source along with swift-syntax, which may significantly increase your build times, so we don’t recommend doing this in production—this is an at-desk experiment. Swift Testing will be built with optimizations off by default, so runtime performance may be impacted, but that’s okay: we’re mostly concerned about correctness rather than raw performance measurements for the moment.

  3. Let us know how your experience goes, especially if you run into problems. You can reach me by sending me a forum DM or by commenting on this PR.

Here be caveats

There are some expressions that Swift Testing could previously successfully expand that will cause problems with this new implementation:

  • Expectations with effects where the effect keyword is to the left of the macro name:

    try #expect(h())

    Macros cannot currently "see" effect keywords in this position. The old implementation would often compile because the expansion didn't introduce a nested closure scope (which necessarily must repeat these keywords in other positions.) The new implementation does not know it needs to insert the try keyword anywhere in this case, resulting in some confusing diagnostics:

    :warning: No calls to throwing functions occur within 'try' expression

    :stop_sign: Call can throw, but it is not marked with 'try' and the error is not handled

    @smontgomery and I chatted with @Douglas_Gregor and @hborla recently about this issue; they're looking at the problem and seeing what sort of compiler-side support might be possible to help solve it[2].

    To avoid this issue, always place try and await within the argument list of #expect():

    #expect(try h())

    For #require(), the implementation knows that try must be present to the left of the macro.

  • Expectations with particularly complex conditions can, after expansion, overwhelm the type checker and fail to compile:

    :stop_sign: The compiler is unable to type-check this expression in reasonable time; try breaking up the expression into distinct sub-expressions

    Because macros have little-to-no type information, there aren't a lot of opportunities for us to provide any in the macro expansion. I've reached out to @hborla and her colleagues to see if there's room for us to improve our implementation in ways that help the type checker.

    If your expectation fails with this error, break up the expression as recommended and only include part of the expression in the macro's argument list:

    let x = ...
let y = ...
#expect(x == y)

  • Type names are (syntactically speaking) indistinguishable from variable names. That means there may be some expressions that we could expand further, but because we can't tell if a syntax node refers to a variable, a type, or a module, we don't try:

    #expect(a.b == c) // a may not be expressible in isolation

    Where we think we can expand such a syntax node, the macro expansion appends .self to the node in case it refers to a type. There may be cases where the macro expansion logic does not work as intended: please send us bug reports if you find them!

  • The ==, !=, ===, and !== operators are special-cased so that we can use difference(from:) to compare operands where possible. The macro implementation assumes that these operators eagerly evaluate their arguments (unlike operators like && that short-circuit their right-hand arguments using @autoclosure.) This assumption is true of all implementations of these operators in the Swift Standard Library, but we can't make any real guarantees about third-party code.

    We believe that this should not be a common issue in real-world code, but please reach out if Swift Testing is expanding these operators incorrectly in your code.

Disabling expression expansion

If you have a condition expression that you don't want expanded at all (for instance, because the macro is incorrectly expanding it, or because its implementation might be affected by side effects from Swift Testing), you can cast the expression with as Bool or as T?:

let x = ...
let y = ...
#expect((x == y) as Bool)

let z: String?
let w = try #require(z as String?)

Example expansions

Here (hidden behind disclosure triangles so as not to frighten children and pets) are some before-and-after examples of how Swift Testing expands the #expect() macro. I've cleaned up the whitespace to make it easier to read.

#expect(f() < g())

Before

Testing.__checkBinaryOperation(
  f(),
  { $0 < $1() },
  g(),
  expression: .__fromBinaryOperation(
    .__fromSyntaxNode("f()"),
    "<",
    .__fromSyntaxNode("g()")
  ),
  comments: [],
  isRequired: false,
  sourceLocation: Testing.SourceLocation.__here()
).__expected()

After

Testing.__checkCondition(
  { (__ec: inout Testing.__ExpectationContext) -> Swift.Bool in
    __ec(__ec(f(), 0x2) < __ec(g(), 0x400), 0x0)
  },
  sourceCode: [
    0x0: "f() < g()",
    0x2: "f()",
    0x400: "g()"
  ],
  comments: [],
  isRequired: false,
  sourceLocation: Testing.SourceLocation.__here()
).__expected()
#expect(x && y && !z)

Before

Testing.__checkBinaryOperation(
  x && y,
  { $0 && $1() },
  !z,
  expression: .__fromBinaryOperation(
    .__fromSyntaxNode("x && y"),
    "&&",
    .__fromSyntaxNode("!z")
  ),
  comments: [],
  isRequired: false,
  sourceLocation: Testing.SourceLocation.__here()
).__expected()

After

Testing.__checkCondition(
  { (__ec: inout Testing.__ExpectationContext) -> Swift.Bool in
    __ec(__ec(__ec(x, 0x6) && __ec(y, 0x42), 0x2) && __ec(!__ec(z, 0x1400), 0x400), 0x0)
  },
  sourceCode: [
    0x0: "x && y && !z",
    0x2: "x && y",
    0x6: "x",
    0x42: "y",
    0x400: "!z",
    0x1400: "z"
  ],
  comments: [],
  isRequired: false,
  sourceLocation: Testing.SourceLocation.__here()
).__expected()
#expect(try f() < g())

Before

Testing.__checkValue(
  try f() < g(),
  expression: .__fromSyntaxNode("try f() < g()"),
  comments: [],
  isRequired: false,
  sourceLocation: Testing.SourceLocation.__here()
).__expected()

After

try Testing.__checkCondition(
  { (__ec: inout Testing.__ExpectationContext) -> Swift.Bool in
    try __ec(__ec(f(), 0xc) < __ec(g(), 0x1004), 0x4)
  },
  sourceCode: [
    0x4: "f() < g()",
    0xc: "f()",
    0x1004: "g()"
  ],
  comments: [],
  isRequired: false,
  sourceLocation: Testing.SourceLocation.__here()
).__expected()

What's with the hexadecimal?

You'll note that all calls to __ec() include an integer literal argument, and the sourceCode argument is a dictionary whose keys are integer literals too. These values represent the unique identifiers of each captured syntax node from the original AST. They uniquely encode the syntax nodes' positions in the tree so that we can reconstruct the (sparse) tree at runtime when a test fails.

I find this subtopic interesting enough to want to devote a whole forum thread to it, personally, but it's a bit arcane—for more information, see the implementation here or feel free to reach out to me via forum DM.

  1. Or, if you prefer, a late Christmas present! (Or, if you prefer, a very early Grunnlovsdagen present?) ↩︎

  2. @smontgomery has opened a PR against swift-syntax that we hope will help resolve this issue. ↩︎

26 Likes