Spawning my comments here into a new topic.
My experience with matchers mostly comes from GoogleTest in C++ and Truth in Java. Right off the bat, swift-testing provides a much nicer syntax for predicate-like assertions than those frameworks due to the fact that the language and macros let you just write what you mean.
#expect(x == y) is so much cleaner than
That's also the appeal of Nimble's operator overloads, although their syntax is slightly different since swift-testing has the advantage of being able to parse the actual expression and transform it. And Nimble also falls back to
traditional(matcher(syntax)) when you go outside those common operations.
So (as directed) I wanted to start this thread to brainstorm ideas for how swift-testing might tackle similar problems around customization while sticking as close as possible to the elegant
#expect(whatIWantToTest) that you've created. There are two related problems to tackle:
- How do I express a specific kind of test that isn't something innate like
- How do I provide meaningful feedback about the failure to the test runner?
Looking over some tests I've written recently, I've had to reach for the following things that aren't easily expressed as traditional binary operator expressions:
- expect that some array contains a specific subsequence
- expect that two collections have the same elements, disregarding order (the collections themselves may be ordered, like arrays)
- expect that some collection is a superset of some other collection, disregarding order
- expect that two protobuf messages are equal, ignoring some fields
Another complication with traditional matchers is that they can be composed. Most matcher frameworks let you write things like "expect that every element in this collection satisfies some matcher" or "expect that this collection is a superset of elements satisfying these matchers".
Many of the cases above could be written using various collection methods, but if all the testing framework ends up seeing is the Boolean result, all context is lost—"expected true, got false" is less helpful than, say, showing what the actual collection contained. The docs cite
#expect(x == y) as a case where the macro can pick apart the values and provide that helpful context.
Figuring out how to generalize what you've done there for other arbitrary expressions would be really powerful. If I write
#expect(x.hasPrefix(y)), I'd want the test output to tell me what
y are and that it was a prefix test that failed. You have that done already, which is lovely! But I wonder if we'd ever want an expectation's failure to have more context about the specific operation that was performed. That's easy to do when matchers have to be their own unique functions/types; the
EXPECT_THAT(x, SomeCondition(y)) can do whatever it wants without the testing framework having to be specially aware of it, but I'm curious how we could extend swift-testing to have hooks for that kind of additional context without twisting the expectation's call site.
I'm really excited to hear the swift-testing team's thoughts on this!