[Idea] Adding support DRY'ing of tests using Shared Behavior

Related Projects swift-testing
1

One common feature in BDD-style frameworks such as Quick is the idea of shared behavior/examples. One common use case for this, if you have code that takes a few branching paths, before ultimately converges into the same set of behavior, then it lets you write the tests for that set of behavior once and refer to them later on. For example, if you have a form with a bunch of textfields, you might want to save a draft every time the text in any textfield changes. Using the shared example feature in a BDD-like such as Quick would allow you to write the "it saves a draft that re-populates if the form is closed and opened again" tests once, saving you from having to copy/paste/edit a large number of tests, and also prevents errors when doing so. Another common use case for shared behaviors is for contract testing - ensuring that a type correctly implements a protocol, essentially.

I think that it would be great if Swift Testing offered support for this arbitrary DRYing of tests. Swift Testing already supports a subset of the capabilities of shared behaviors, through parameterized tests. But that doesn't go nearly as far - parameterized tests work really well for testing a range of inputs/outputs, but not so much for the examples here of contract tests, or when multiple paths leads to the same behavior.

2

I think anything that helps make test authoring easier and eliminates rote/boilerplate is worth our time.

Do you have any thoughts on how you'd surface this sort of functionality in a Swift Testing-y way? We don't generally rely on class hierarchies for our functionality, but we love a good protocol 'round here.

3

As far as defining these shared tests, definitely a protocol makes sense. A struct conforming to some protocol that defines a shared suite of tests makes sense. A macro might make more sense, considering how much of the commonly-used API from Swift Testing is available only as macros. Either way, the code for detecting test suites would need to be updated to not include types specifically marked as shared tests (and I think a build-time warning should be emitted if there are unused shared test suites detected).

The thing that trips me up is how to include those shared tests in another suite. At the simplest, in the init method for a Suite, you could include the shared tests somehow. Like this:

@Suite
struct SomeSuite {
    init() {
        include(SomeSharedTests())
    }
}

Which could work, but is clunky and not idiomatic at all.

A variation on the above idea might be a separate macro, that can be applied to a method/func in a suite to insert the contents of a shared test suite there. It's also somewhat clunky, but provides a pretty good user experience, and doesn't require an expansion to the swift macro system.

@Suite
struct SomeSuite {
    @WithSharedTests
    func thisNameDoesntMatter() -> SharedTests {
        SomeSharedTestSuite(...)
    }
}

With SharedTests being a public protocol from Swift Testing that all Shared Tests must conform to.

While better, it's still a bit clunky - having to define a method just to output the list of shared tests is weird.

As a third idea, there could be separate macro that automates the second idea. Something like this, maybe?

@Suite
@WithSharedTests { SomeSharedTests(value1) }
struct SomeSuite {
    let value1: ...
}

(I think it reads a bit better if adding shared tests is a separate macro, purely for the trailing closure syntax, but it ultimately doesn't matter).

However, I don't think it's possible for member macros to add arbitrary closures that implicitly reference self like what I just tried. You could certainly have it take in a Self, such that the code then becomes @WithSharedTests { SomeSharedTests($0.value1) }, but that's a bit clunky.

4

Suite seems to not support generics?

See this XCTest inheriting from a base test and specifying SUT as a generic to Test

Then I layer this and for each layer I constrain the generic type to a more specific protocol and putting test cases in each layer. XCTest has been great for this!

But I tried redoing it in with Swift Testing and I think I got hit by an error saying Suite did not support generics.

Any plan on adding support for it?

5

In order to support a generic type, we need to know in advance how to specialize it before we can call any test functions it contains. If we can't specialize it, it's not possible to call its test functions.

As well, a generic test suite is not always visible to the @Test macro applied to member functions of the suite type, which also prevents us from specializing the suite type.

Although we are aware of a desire to support generic suite types and test functions, we have no plans at this time to support them.

T code emitted by the @Test and @Suite macros currently injects information into the type metadata section used by the Swift runtime. Once we switch over to a new model based on @_section, we may be able to revisit this constraint because we won't need to specialize types manually if they are sufficiently well-declared at compile time.