RFC: In-Line Tests

I completely agree that something needs to happen with testing and Swift, but it seems there are lots of different opinions of what that should look like. I foresee an epic thread coming up here.

A few things that come to mind:

  • If we have the @test methods side by side with our code, the line-count could potentially explode for lots of files.
  • If we need separate frameworks for testing, we still have to have a separate test target? (I assume importing them with @test import would only import them when running the tests, but we still have to specify them for copying them into some bundle or for the linker, which is most likely oblivious of such conventions?)
  • If I need mock classes and maybe JSON file fixtures, those should of course also just be compiled/copied when I am testing, but how would I do that?

If the answer to some of those (maybe stupid or naïve) questions is that I can still use separate test-targets, then this would let me to believe that we are creating yet another parallel way of testing (the others being custom testing frameworks for example that exist now in parallel to XCTest) which wouldn't help the whole testing mess we have at the moment.

2 Likes

I'm happy my reply in that older thread sparked this discussion.

I find the following structure appealing:

// MyClass.swift
class MyClass {
    private func add(_ a:Int, to b: Int) -> Int {
        return a+b
    }
}

// MyClassUnitTests.swift
@test extension MyClass {
    func testAdd() {
         let four = self.add(2, to: 2)
         XCTAssertEquals(4, four)
    }
}

However, as has been stated above, separating the tests into an extension in another file wouldn't solve the problem of testing private methods. Again we fall into a case where this functionality requires rules to be broken only for tests.

2 Likes

Just like under Rust, there is still room for a testing framework to augment the usual pile of assertions. One of my goals in having this discussion is that we can come to some kind of consensus to expose the hooks to make that happen. In a world of literate testing, XCTest does not have to disappear. After all, there will still be Cocoa code, older test suites to contend with, and people that agree with @Jon_Shier in that your unit tests should be separate from your code base. In more practical terms, XCTest also exposes timing primitives, assertions for asynchronous code, and exception testing that are quite useful by themselves.

The underlying idea in the language changes is that the test discovery part of XCTest should be generalized out of XCTest itself, but that's really it.

5 Likes

This is certainly the primary criticism of literate testing paradigms. As a counter, I would invite you to explore what that looks like in practice, particularly for larger/more popular Rust projects. The consensus seems to be to use a mix of inline unit testing for smaller invariants/private members, and broader integration test suites to cover the boundaries between interacting modules. The idea here as there isn't all the tests are now inline the idea is relevant tests appear near relevant code. When you start to speak about testing systems instead of components, then you're talking about integration testing, and integration testing fundamentally cannot occur at the level of the individual component.

7 Likes

I understand and I think it would be pretty cool to do this sometimes. But this is exactly the point I think – it adds on top of the other cruft and doesn't solve that.

I really, really want to see a better solution for exposing code to tests. Thank you for starting this thread! It's especially exciting to see this topic brought up by someone with experience working on the compiler! :slight_smile:

I have worked with same-file tests in Ruby in the past. In some cases included tests in the same file (or even local scope!) as the code under test is really convenient, especially as the code is first being developed. However, as the code grows there are often better ways to organize the code. This is of course a value judgement, but it is one that is shared by a large number of programmers.

At the same time, I think there are advantages in placing test code in much closer proximity to production code than is usually the case today in Swift and Objective-C projects. Compiling tests along with production code in some build configuration(s) thus allowing tests to participate in existing scoping mechanisms is a good way to take advantage of closer proximity.

I do have some concerns about the details of this pitch. The first and most significant is that this pitch only makes progress on the visibility issue (relative to @testable) is if tests are placed in the same file as production code. As noted above, this is convenient when first writing code but it is an approach that doesn't scale well (at least in the opinion of many of us).

This would help a lot, but I'm not convinced it is ideal either. It couples the physical structure of our test code with the physical structure of our production code. It is not uncommon to have a test code living in one file that tests production code that residing in more than one file (and vice versa). The ideal solution to the testability problem would not force a tradeoff between suboptimal physical structure and suboptimal access control.

A submodule system would help alleviate some of the pressure of scope-based testability if it allowed us to form scopes that are larger than a file but smaller than the whole module. It would allow us define submodules which contain some production code and some test code within a "submodule internal" scope. private and fileprivate symbols still wouldn't be visible to test code, but at least the impact of bumping their visibility is limited to a submodule rather than the entire module.

Fortunately, the above approaches are not mutually exclusive. Some tests could begin life with the @test attribute in the same file as the production code they test. As code grows most of them could move to a test file that is "paired" with the file containing production code. Additional test code including test helpers can live in separate files within the same submodule and would have access to "submodule internal" symbols in the production code.

There is a lot to like about that direction, however, I don't think its potential is fulfilled without a suitable submodule system (which I believe is out of scope for Swift 5). It is hard to evaluate a design for scope-based testability when we don't yet know if Swift will ever have submodules and if so, what the design would look like.

In addition to the concerns listed above I would also want:

  1. The ability to have entire files considered test code without needing to annotate every declaration.
  2. The ability to exclude test code from debug builds when not running tests.
  3. The ability to have test-only external dependencies (perhaps the @test import XCTest syntax in the pitch is relevant here).
  4. The ability to have test-only resources (such as mock json files).
  5. There are probably other things we do today that take advantage of tests being in a separate target. I haven't had time to think through all of the implications of losing the ability to configure the test target.

This all seems reasonable, but on its own is not sufficient. We absolutely must have the ability to define test-only code that does not meet these requirements, but is used by test code that does meet them. For example, we must be able to write domain-specific test helpers that streamline our test code.

Having said all of the above, I think the discussion may be a bit premature. As mentioned above, I have some reservations about committing to a design prior to having the submodule discussion.

One alternative that would solve the conflict between access control and testability and might be viable in the Swift 5 timeframe would be to add @testinternal which would promote the visibility of a symbol to internal when building for tests. This may not be the ideal long-term solution but it would be a tangible improvement and would allow more time to think about the longer-term design, perhaps allowing us to tackle submodules first.

I am excited to see where this thread goes and whether we can converge on a solution that makes a genuine improvement in the Swift 5 timeframe.

11 Likes

Can you be specific about how this is a non-solution?

I would say that more fundamentally, from an access control standpoint, white-box implementation tests need to be treated as part of the same access scope as the thing being tested. Adding more access control levels is not going to sidestep that fundamental property, and adding backdoors to access control so tests can bypass it is treating the symptom rather than the problem.

7 Likes

Yes, sorry for being non-specific. I imagine when not doing the in-line tests, we still need separate test targets for the reasons I mentioned above. That's why I thought the in-line tests – while being a really good thing – are additive to what we have already – hence not a solution to the original problem.

Does that make sense? (I could be completely wrong of course and misunderstand)

I realized I should have elaborated a bit further here. While this on its own is not an ideal long-term solution @testinternal or something like it would continue to be useful in a scope-base test system. For example, it would allow us to expose a private members of a type to any test code. Without something like @testinternal test code living inside a @test class will not be able to see any private members declared on production types, thus forcing fileprivate or internal to be used instead.

1 Like

I very much appreciate hearing your opinions, especially on the current hypothetical, given that you have lots of experience with this. I agree with your points about better modularity, so I'm going to strategically duck the issue and focus on the criticisms

A question I have is: what does scaling mean here? Most codebases I've seen try to stick to one or two classes per file, or perhaps keep the main body of a class in one file and define specific behaviors in extensions in nearby files. The number of tests thus grows linearly in the number of regions of concerns and that scales quite well.

I was hoping that this would still be possible given the current outline. (I think the transitivity of @test on aggregates is the main impediment to that?). One might, for example, define a Formatter class that has a pack of test methods that are fed by a generator of random strings that is entirely local to the @test context.

class Formatter { /**/ }

@test class FormatterTests {
  let formatter = Formatter(/**/)
  let rng = StringGenerator(/**/)
 
  // @test-context makes this private private; does not appear in non-test builds.
  class StringGenerator {
    func nextString() -> String { /**/ }
  }

  @test func dontCrashOnGarbage() {
    XCTAssert(formatter.format(rng.nextString()) != nil)
  }
}

Absolutely. I want the discussion to happen now so I can gauge community reaction to something in the future. I have no plans of this making it into Swift any time soon.

2 Likes

Well I'm not proposing to add more access control since this ship has sailed and the topic has become a general pain point of everyone in the community. To me it seems that some do care about the line/file height and some don't. The folks that do care about it (including myself) will move parts of the code into different files, which potentially will break the intended code encapsulation due to access control limitations. The next issue I see in this thread is the lowest access modifier we currently have that become more or less default in a lot of projects. So the folks that would want to test these private functions and simultaneously would want to put those tests into a different file will hit the wall very quickly if they don't want to sacrifice the access level for tests. That said I think we wouldn't discuss this pitch that intensibly if we didn't had those access control issues and potentially could just go with a first class unittest blocks. If then someone would want to separate tests into different files it wouldn't be an issue if the current private was type bound in the module.

Let's be specific about the original problem, because it is manifold

  • XCTest is fundamentally not a Swift framework
  • Unit testing does not interact well with access control and is forcing @testable as a compromise
  • There is a lot of boilerplate involved in the current approach

The idea of in-line unit testing is to hit the second and third points head-on, and hit the first by virtue of the implementation we would eventually settle on. So while in-line testing is additive like you claim, without it we cannot address the core of the issue with the current testing situation in Swift. If we were, for example, to just generalize the test harness machinery we would still be living in a world with @testable breaking the access control model.

5 Likes

Thanks for clearing that up for me. I can't wait where this thread is leading. Thank you so much for starting this!

3 Likes

This might be a ridiculous pitch from a non-compiler developer but would it potentially make sense to mirror swift files for testing where those tests would receive the same visibility as in the original swift file?

MySpecialClass.swift
MySpecialClass+CoolExtension.swift
MySpecialClass+AnotherExtension.swift
//====---- Mirror for unit test ----====//
MySpecialClass.unittest
// No tests written for `MySpecialClass+CoolExtension.swift`
MySpecialClass+AnotherExtension.unittest

.unittest files would also be swift files which will view the mirrored swift file and have the same access control as the original file.

3 Likes

I'm sorry, you lost me. What did you mean there?

Isn't this basically what @Joe_Groff suggested here:

2 Likes

Tests that are testing an implementation, as opposed to testing its high-level interface, naturally want to have free access to the implementation details without regard to the access control you normally want to enforce for the component.

6 Likes

Potentially yes, reading his comment I had a different model in mind, but now the idea I presented seems to be very similar to this too. Thank you for point this out.

Thanks, I got it now :slight_smile:

And I agree. Testing only at the interface level is bound to backfire in a complex implementation.

Think crypto or hashing, for example. In a multi-step protocol you'd love to be able to test the intermediate stages of a text transformation. But exposing that in the API would make no sense and increase its surface area significantly. On the other hand, limiting testing to the higher level interface would make it really difficult to find where an error lies in the implementation.

1 Like