Customize difference output?

Development Swift Testing
1

Is it possible to customize the output of the difference of a expect/require comparison? When comparing larger structures or collections the current output makes it almost impossible to identify what is actually different.

I'm currently using GitHub - krzysztofzablocki/Difference: Simple way to identify what is different between 2 instances of any type. Must have for TDD. with XCTest, but I'm not sure how to use it or an equivalent with Swift Testing.

Can this be done with overloads (of __checkBinaryOperation?), traits, etc.?

The current output for collections seems to be implemented in swift-testing/Sources/Testing/Expectations/ExpectationChecking+Macro.swift at 0d85d8b32bc93afb30f52346b3612a86ebbb11f3 · swiftlang/swift-testing · GitHub.

1 Like
2

@grynspan @smontgomery is there currently any way to accomplish this? If not, what would be a good way to enable such customization? I’d be happy to discuss and contribute

3

First, sorry for overlooking your original post!

On the topic of expectation "diffing", I know this is an incredibly useful feature. Third-party libraries (such as Difference which you mentioned) do a pretty good job in the XCTest space. However, we'd like to eventually incorporate this kind of functionality directly into Swift Testing. Although those of us who maintain the library have talked about it often, and it has come up in Testing Workgroup meetings, I just realized we do not have a GitHub issue formally tracking the feature yet, so I went ahead and filed one here:

If you are inclined to experiment on your own, you might be able to leverage ST-0011: Issue Handling Traits to some benefit, however that's a fairly roundabout way to accomplish this workflow so it's only something I mention as a workaround.

1 Like
4

Thank you for the reply Stuart! I will have a look at the issue and ST-0011

5

We have a tool called expectNoDifference that is part of a few other tools for nicely printing the internals of complex types. It allows you to assert in both XCTest and Swift Testing that two values are equal, and when they are not a nicely formatted message is printed:

expectNoDifference(user, other)

expectNoDifference failed: …

  User(
    favoriteNumbers: […],
    id: 2,
-   name: "Blob"
+   name: "Blob!"
  )

(First: −, Second: +)

It tries to collapse arrays/dictionaries where there are no changes in the elements.

5 Likes
6

Nice! Thank you for the link, I’ll give this a try

7

Now that Swift 6.2 is released I tried out compactMapIssues.

I wrote a simple suite:

@Suite(.compactMapIssues { issue in
  print("Issue: \(issue)")
  return issue
})
struct FooTests {
    @Test
    func testFoo() {
        #expect(1 == 2)
    }
}

Which surprisingly results in:

◇ Test run started.
↳ Testing Library Version: 6.2 (3fdabe5392108d8)
↳ Target Platform: x86_64-unknown-linux-gnu
◇ Suite FooTests started.
◇ Test testFoo() started.
✘ Test testFoo() recorded an issue at FooTests.swift:10:9: Expectation failed: 1 == 2
Issue: Expectation failed: 1 == 2 (error)
Issue: Expectation failed: 1 == 2 (error)
✘ Test testFoo() failed after 0.001 seconds with 1 issue.
✘ Suite FooTests failed after 0.001 seconds with 1 issue.

It looks like:

  • The handler passed to compactMapIssues is run twice. Is this expected?
  • issue.error is nil. Is this expected? It seems like there is no data available about the values that were compared
8

The library looks great Brandon! I tried to use it on Linux, but it looks like that is not supported. Opened Support for non-Apple platforms · Issue #141 · pointfreeco/swift-custom-dump · GitHub

9

Yes, the handler is invoked for the test and then for the suite (because the trait is declared on the suite.) You can check Test.current?.isSuite to skip handling it the second time.

Yes. That property will only be non-nil for issues that correspond to thrown errors. In your code, you don't throw an error. Information about the difference between compared values is not exposed in Swift Testing's API at this time; eventually, for collection types, we would want to expose it as a property of type CollectionDifference on Issue, however this property wouldn't be available for trivial comparisons like the one you have in your example.

1 Like
10

Replied in the GitHub issue, but Linux is supported just fine (as well as Android and Windows) and we use it quite a bit in various Linux projects.

1 Like