Discovering and Registering Programmatically-Generated Tests with Swift Testing

Related Projects swift-testing
3

Fundamentally, Swift Testing relies on information about tests that's produced at compile time and discovered at runtime. It's intentional that no public initializer is offered for Test because just instantiating the type is not sufficient to make a test runnable.

A big part of Swift Testing's design philosophy is that we know what tests exist up front, before we start running any, so that we can plan out a test run (by skipping invalid tests, by serializing interdependent tests, etc. etc.) Having new instances of Test appear after a test plan is constructed is not something we want to support if we can avoid it.

I'm going to put aside the idea of adding a Test.register() function or similar, for the moment. I'm not ruling it out—just for the purposes of this forum post I'm ignoring it. :slight_smile:

In order for your test code to get discovered by Swift Testing with the implementation we have today, you need to create a type, give it a particular funky name, conform it to a private protocol, and add a static __tests property that enumerates your tests. This mechanism/interface for test discovery is not meant to be permanent.

Thoughts about a future test discovery ABI

Eventually, we intend to migrate to a dedicated runtime metadata section (swift5_tests). We're waiting for @kubamracek's @_section attribute to land in some form before we can do that.

We haven't defined the ABI for that section yet, but it's intended to be of a form that can be used by tools like Quick. It'll likely consist of a sequence of function pointers with signatures like:

@convention(thin) @Sendable () async -> [Test]

Since Quick still needs to be able to generate instances of Test for that to work, we could generalize it a bit and say "returns an instance of any TestGeneratorProtocolFactoryBeanWhatever" instead, and provide a public documented interface. This is appealing over simply providing Test.init because it's less likely to be accidentally used by a test author who thinks it'll cause a test to be dynamically run. You'd have to intentionally reach for it, meaning you probably have to read the documentation for it too!

Since the current solution (type with weird name and protocol conformance) is not permanent, we've prefixed all the public bits with double underscores to emphasize that developers shouldn't be using them. You could hypothetically write something within Quick that uses these symbols directly as a proof-of-concept.

Temporary hackish solution to the problem 
@usableFromInline
struct GlueForQuick__🟠$test_container__: __TestContainer {
  static var __tests: [Test] {
    get async {
      var result = [Test]()

      for testData in ... {
        result.append(
          .__function(
            named: testData.programmaticFunctionName,
            in: testData.containingTypeIfAny,
            xcTestCompatibleSelector: nil,
            displayName: testData.humanReadableTestName,
            traits: [...],
            sourceLocation: testData.sourceLocationOfTest,
            testFunction: testData.testFunction
          )
        )
      }

      return result
    }
  }
}

I want to again emphasize that this would not be a permanent solution, and at most would be an experimental implementation to allow you to move forward with your project. If you did write something like the above code, we'd ask that you be prepared to refactor it when we get our stable ABI at some point in the future, and to add lots of scary comments warning people not to copy the code because it will break someday.

9 Likes