[Pitch] Range-based confirmations

Hmm… I'm not completely sure how testing patterns from other ecosystems handle situations like this by-convention (and I would like to hear if anyone has any ideas or context for that).

I'm thinking that there's a subtle shift in thinking implied here… where we are now giving a subset of tests (the confirmation) the opportunity to be "flaky" by-design. It's not necessarily a "bad" thing… but we already formally define withKnownIssue to indicate tests that can fail but do not lead to test failure runs. One way to look at this is multiple parallel "versions" of flakiness. Should a confirmation that might fire with different values be wrapped in a withKnownIssue or wrapped with a new range-based confirmation? Would we make any kind of opinionated statement about how these two different approaches work together?

Was there discussion of a parallel API on confirmation that transitions away from a expectedCount integer and toward something like an arbitrary closure that could be defined ad-hoc by the engineer building the tests? If an engineer truly wanted to build more flexible tests could that be a legit alternative?

1 Like