More flexible failure messaging for custom test helpers?

I'd like to build some custom helpers over Swift Testing, but the current public tools seem to be quite rigid in how the messages emit. The main API a custom helper might leverage is Issue.record, which prefixes failures with "Unconditionally failed". This is not only verbose, but can feel wrong when emitted from a helper that should only conditionally fail:

func assertDifference<T: Diffable>(
  lhs: T,
  rhs: T,
  sourceLocation: SourceLocation = #_sourceLocation
) {
  guard isDifferent(lhs, rhs) else {
    Issue.record(
      "No difference detected between \(lhs) and \(rhs)",
      sourceLocation: sourceLocation
    )
    return
  }
}

assertDifference(countBefore, countAfter)

:x: Unconditionally failed: No difference detected between 1 and 1

The alternative is to use #expect, which has a nicer "Expectation failed" prefix. While it feels strange to call a macro from library helper code, I tried anyhow, and unfortunately it is even less usable, because custom comments are hidden below the fold in Xcode's UI, and the primary comment is generated from the expression, which feels difficult to leverage:

func assertDifference<T: Diffable>(
  lhs: T,
  rhs: T,
  sourceLocation: SourceLocation = #_sourceLocation
) {
  guard isDifferent(lhs, rhs) else {
    #expect(
      Bool(false),
      "No difference detected between \(lhs) and \(rhs)",
      sourceLocation: sourceLocation
    )
    return
  }
}

assertDifference(countBefore, countAfter)

:x: Expectation failed: Bool(false) […]

Now one thing I can do is expand the macro-generated code and traverse the double-underscored Testing.__ world. This actually lets me achieve some nice failure messaging:

func assertDifference<T: Diffable>(
  lhs: T,
  rhs: T,
  sourceLocation: SourceLocation = #_sourceLocation
) {
  guard isDifferent(lhs, rhs) else {
    Testing
      .__checkValue(
        false,
        expression: __Expression.__fromStringLiteral(
          "No difference detected between \(lhs) and \(rhs)",
          ""
        ),
        comments: [],
        isRequired: false,
        sourceLocation: sourceLocation
      )
      .__expected()
    return
  }
}

assertDifference(countBefore, countAfter)

:x: Expectation failed: No difference detected between 1 and 1

But:

  1. I'm relying on an implementation detail of __Expression.__fromStringLiteral rendering the error message how I want. It feels like the wrong thing to do. Maybe the format will not be right in the future...
  2. I'm relying on __-prefixed methods and I'm not sure if Swift Testing treats these public functions as stable (even ABI-stable), even though they are "public" as far as macro-expansion is concerned.

So none of these options feels great. The third provides the best tool, but it would be nice to not rely on private, implementation details. Is there room for better public-facing APIs?

We're tracking this problem space with New Issue.Kind case for third party expectations. · Issue #490 · apple/swift-testing · GitHub. :slight_smile: I would strongly recommend against using double-underscored symbols as we will break you at some point in the future.

2 Likes