RFC: In-Line Tests

Evolution Pitches
1

Hello all,

A notification I received in an old thread about furthering the hack we have to enable out-of-target unit testing with @testable imports got me thinking about what we could do to better the situation. I've drawn up this RFC to kick off a broader discussion about what we can do to get modern testing facilities into the language.

Note: This is not a proposal, this is not final, this is just food for thought.

In-Line Tests

Motivation

Testing is a critical part of the development cycle of reliable software. As languages evolved, they each came to support different modalities and feature sets to better enable testing that aligned with their unique philosophies. For Objective-C, the frameworks of choice for unit testing was SenTestingKit and OCUnit and their integrated test harness. Throughout Xcode's evolution, these frameworks saw little changes from their original APIs. Deeper IDE integration only came with Xcode 5 when SenTest was phased out in favor of XCTest - a nearly identical framework and API with deeper IDE integration.

Fast-forward to today when we have a completely new language, but the same tools. Swift, despite having a vastly different programming philosophy than Objective-C, has nonetheless inherited Objective-C's solution and tried to paint over the cracks to make it work like a bona-fide Swift framework. All of this makes unit testing in Swift feel like a second-class citizen. XCTest forces a number of undesirable patterns just to conform to its interaction model:

  • All tests must occur outside of the system under test in a separate testing target
  • All test groups must subclass XCTestCase
  • All tests must have method names beginning with the word "test"

To further support the first point, that tests occur in a separate testing bundle, Swift allows temporarily breaking the access control model by declaring an @testable import.

On non-Darwin platforms, because we lose the dynamic introspection that defines XCTest's test harness' discovery process, we have the additional rule:

  • All tests must be manually declared in XCTMain

This additional rule has lead to the current system of having each XCTestCase subclass define ad-hoc boilerplate to teach the test harness about where to find test functions

  #if !os(macOS)
  static var allTests = testCase([
    ("testFoo", testFoo),
  ])
  #endif

And a LinuxMain that must additionally manually specify all test cases. This quickly and easily leads to mismatches between macOS and Linux test suites: a common mistake encouraged by this situation is to simply forget to add a test suite or a test function to the list.

Solution

Swift deserves a modern Swift-first approach to testing, we believe that solution lies in a literate programming approach to testing. Literate programming advises us that keeping our tests close to the code it is testing leads to richer tests. Better locality of information and reasoning means less cognitive overhead when defining test cases. It also massively lowers the barriers to defining a test suite by not requiring a separate target.

We propose the addition of the @test attribute. @test on a function declaration transforms it into a unit testing function then and there. To better support the aggregation of these unit test functions into suites, we allow @test on classes, structs, and enums (with some caveats) so member functions may be transitively marked @test. Crucially, @test declarations are only compiled into Debug builds - they are explicitly excluded from Release builds†.

@test declarations come with some restrictions in order to keep the programming model logical:

  • @test functions cannot take arguments
  • @test functions are private to their containing aggregate and may not be called by non-@test code
  • @test structs, classes, and enums must have a no-arguments init() - this may be synthesized

Because @test functions can appear in-line with application and framework code, they naturally enable the testing of private and fileprivate declarations. In fact, they naturally enable the testing of internal declarations as well. To that end, we propose the deprecation of @testable import.

@test import XCTest

struct RawSauce {
  private let four = 4
  private let three = 3

  // quickMaffs is only available in a testing build and 
  // will be invoked by the test harness automatically.   
  @test func quickMaffs() {     
    XCTAssert(2 + 2 == four)
    XCTAssert(4 - 1 == three)
  }
}

This Has Been Done Before

  • Lagrangian is an experiment by Rob Rix to enable this directly in Cocoa.
  • D, Rust, Groovy, among others have language-level support for this literate testing paradigm. Other languages like JavaScript and Scala have third-party frameworks.

Open Questions

  • †You may want to test in a release build, maybe we need an orthogonal scheme for "include/don't include unit tests"
  • To better enforce the rules, should we make @test on aggregates a protocol conformance in addition/instead?
    • This may enable better third-party tools
  • In replacing/automating the old NSInvocation test-discovery process, can we better enable third party test harnesses too?
60 Likes
2

Swift certainly needs a Swifty testing framework, but combining production code and tests seems like it would be terrible. If it were available, a multitude of blog posts would follow, outlining how to separate your tests from your production code. Also, it's generally bad practice to need to access private state in order to test something, even if it's nice to occasionally be able to break that limitation with @testable.

In the short term, @test would be a good workaround for the Linux testing issue, which is a major pain in the ass.

8 Likes
3

This doesn't seem to be what's happened in any of the other languages that adopt this paradigm; D's design in particular is extremely well-received by it users. There's legitimate uses for both white-box and black-box testing, and the former is often as much part of the "documentation" of the code as comments and type signatures are.

17 Likes
4

Rust has doc-tests where code snippets in doc comments can get executed as tests. This is really nice because writing these gives you both docs and tests at the same time, and encourages small, tested examples in your documentation.

They obviously don‘t cover all testing need but they are pretty nice.

16 Likes
5

Good point, and I believe they borrowed that idea from Python, where it's also very popular.

5 Likes
6

While that's nice, none of the other languages mentioned inhabit the same niches Swift does as a multi-paradigm and multi-platform language. Different types of usage (and apps) being built with the same language would have a huge impact on the feasibility of such integrated tests. Having worked on mobile apps that have more lines of code in tests than in the app itself, integrated tests sounds nightmarish. And offering a feature in the language that's entirely infeasible for entire realms of usage seems undesirable.

3 Likes
7

I would gladly welcome this. My only doubt at this moment is: what would the preferred way of adding type-specific test utilities? As in: if tests on a function would like an utility method to make writing them nicer/more readable, how would you do it? Of course these shouldn't compile on production code... And by defining them as @test you clash with actually writing a test.
Maybe @test extension? But in that case you potentially lose access to private stuff.

3 Likes
8

In that case, keep your unit testing target. Nothing in here breaks that model.

7 Likes
9

Yes, doc-tests are a pretty useful feature. The only downside to them that I've seen is related to division between unit and doc-test and how they're used. I've seen codebases that try to put several tests in a doc-test, when you really only want a few, to both document how to call/use something, and as a quick sanity check.

10

Of course it does. You're creating a completely separate testing paradigm with different capabilities, bifurcating Swift's testing story with changes to the language that are largely unused on some platforms. That's generally a bad idea. Can't we come up with a Swifty testing framework that doesn't completely change everything, and then see how much further we need to go?

2 Likes
11

I don't see your argument that this would go unused in some paradigms. D, Rust, Python, and Groovy are all "multi-paradigm general purpose languages" too. Going with a "doctest"-style model like Rust or Python I think would allay your fear of code getting drowned out by tests, since the tests would be delimited like doc comments are and be similarly easy to skim past. This ideally wouldn't bifurcate the testing story, beyond existing test frameworks remaining for compatibility, since it's a more general feature that can subsume them.

8 Likes
12

A way to separate tests and the code they test in different files (but in the same module) would be really nice. The most intuitive way I think this can be enabled is to allow extensions to access private members in @test methods.

Now, why would I want that: at my work we have a linter rule that limits line count of files to encourage refactor and organization of code. This helps a lot to scale a codebase with large number of developers. With tests thrown in the mix, this benefit goes out of the window until we update the linter to have more intelligence for this type of things (which may be a better solution than changing the access control rules?).

I apologize for bringing up a taboo topic.

13

Allowing this freely has many of the same drawbacks as @testable does now, since it would subvert the normal visibility guarantees on private members and thereby require a special compilation mode, which in turn means that the thing you're testing is somewhat less likely to be similar the thing you release, especially regarding inlining and specialization performance impact. As an alternative, there could be a way to pair a test file with a specific implementation file, which would let the tests be built together as if they were part of that file in the same translation unit.

5 Likes
14

My main point is that some users of Swift, perhaps even entire platforms, would find doctests completely unsuitable, for a variety of reasons. In that case, you've bifurcated the language in a fundamental way, which, at the very least, is undesirable. I also disagree with your assertion that doctests would be easy to ignore. This list has had requests for the ability to separate documentation from code in a past, and such requests would only increase when even more vertical space is take with a few tests, especially of the complexity outlined in the OP.

1 Like
15

The pitch is very interesting but I already do not like the limitations like a required init(). Have you also considered first class unittest blocks like Joe Groff described in the other thread I created a couple month ago?

16

What? How is adding doc-tests bifurcating a language? Doc-tests are not, and should not, be used as a substitute for unit testing. Like I said upthread, doc-tests are useful for showing how to call/use something, and as a quick sanity check that happens while testing.

2 Likes
17

We've had 5+ years to come up with one and we still have XCTest (or StdlibUnitTest for those familiar with stdlib). I say this as the author of one such framework and a former user of Lagrangian, Quick, Specta, and Expecta under the old regime. I posit we have seen the best Cocoa-style frameworks can do for the situation by now.

5 Likes
18

I'm sorry, but you really shouldn't use Apple's investment into the language as an example of everything that can be done. As you just said, there are a variety of other testing frameworks out there which have been adopted to some extent or another, in addition to the official version. Obviously there are other options, it's just a matter of getting them adopted officially by Swift. And if you don't think it's possible, what is this thread for? Is it just an end run around creating such a framework?

19

To be honest IMO this issue only exisits because of what access control we have today. Without rehashing anything in this thread I think it wouldn't be an issue if private was type bound for the entire module and not the current file. In that sense current fileprivate really is (file)internal.

⚠️ Please do not unfold and read this spoiler if you don't want to see anything about access control that has been discussed 1000 times.

Here is how imagine ideal access control for Swift which would have solved every possible access control issues:

  • open (can subclass, can conform to outside the module)
  • public (cannot subclass or conform to outside the module)
  • internal
  • (file)internal (like current fileprivate)
  • private (type bound access from any file of the module)
  • (file)private (same as current private)

:shushing_face::zipper_mouth_face::ship:

1 Like
20

In this new testing word, what would XCTest look like? If we go with the Rust-like model where unit tests live in the same file as what's being tested, is there really a point to having XCTest? I would think we would just have a stdlib/test where most of the testing primitives would be defined.