RFC: In-Line Tests

Of course it does. You’re creating a completely separate testing paradigm with different capabilities, bifurcating Swift’s testing story with changes to the language that are largely unused on some platforms. That’s generally a bad idea. Can’t we come up with a Swifty testing framework that doesn’t completely change everything, and then see how much further we need to go?

2 Likes

I don’t see your argument that this would go unused in some paradigms. D, Rust, Python, and Groovy are all “multi-paradigm general purpose languages” too. Going with a “doctest”-style model like Rust or Python I think would allay your fear of code getting drowned out by tests, since the tests would be delimited like doc comments are and be similarly easy to skim past. This ideally wouldn’t bifurcate the testing story, beyond existing test frameworks remaining for compatibility, since it’s a more general feature that can subsume them.

7 Likes

A way to separate tests and the code they test in different files (but in the same module) would be really nice. The most intuitive way I think this can be enabled is to allow extensions to access private members in @test methods.

Now, why would I want that: at my work we have a linter rule that limits line count of files to encourage refactor and organization of code. This helps a lot to scale a codebase with large number of developers. With tests thrown in the mix, this benefit goes out of the window until we update the linter to have more intelligence for this type of things (which may be a better solution than changing the access control rules?).

I apologize for bringing up a taboo topic.

Allowing this freely has many of the same drawbacks as @testable does now, since it would subvert the normal visibility guarantees on private members and thereby require a special compilation mode, which in turn means that the thing you’re testing is somewhat less likely to be similar the thing you release, especially regarding inlining and specialization performance impact. As an alternative, there could be a way to pair a test file with a specific implementation file, which would let the tests be built together as if they were part of that file in the same translation unit.

4 Likes

My main point is that some users of Swift, perhaps even entire platforms, would find doctests completely unsuitable, for a variety of reasons. In that case, you’ve bifurcated the language in a fundamental way, which, at the very least, is undesirable. I also disagree with your assertion that doctests would be easy to ignore. This list has had requests for the ability to separate documentation from code in a past, and such requests would only increase when even more vertical space is take with a few tests, especially of the complexity outlined in the OP.

1 Like

The pitch is very interesting but I already do not like the limitations like a required init(). Have you also considered first class unittest blocks like Joe Groff described in the other thread I created a couple month ago?

What? How is adding doc-tests bifurcating a language? Doc-tests are not, and should not, be used as a substitute for unit testing. Like I said upthread, doc-tests are useful for showing how to call/use something, and as a quick sanity check that happens while testing.

1 Like

We’ve had 5+ years to come up with one and we still have XCTest (or StdlibUnitTest for those familiar with stdlib). I say this as the author of one such framework and a former user of Lagrangian, Quick, Specta, and Expecta under the old regime. I posit we have seen the best Cocoa-style frameworks can do for the situation by now.

4 Likes

I’m sorry, but you really shouldn’t use Apple’s investment into the language as an example of everything that can be done. As you just said, there are a variety of other testing frameworks out there which have been adopted to some extent or another, in addition to the official version. Obviously there are other options, it’s just a matter of getting them adopted officially by Swift. And if you don’t think it’s possible, what is this thread for? Is it just an end run around creating such a framework?

To be honest IMO this issue only exisits because of what access control we have today. Without rehashing anything in this thread I think it wouldn’t be an issue if private was type bound for the entire module and not the current file. In that sense current fileprivate really is (file)internal.

⚠️ Please do not unfold and read this spoiler if you don't want to see anything about access control that has been discussed 1000 times.

Here is how imagine ideal access control for Swift which would have solved every possible access control issues:

  • open (can subclass, can conform to outside the module)
  • public (cannot subclass or conform to outside the module)
  • internal
  • (file)internal (like current fileprivate)
  • private (type bound access from any file of the module)
  • (file)private (same as current private)

:shushing_face::zipper_mouth_face::ship:

1 Like

In this new testing word, what would XCTest look like? If we go with the Rust-like model where unit tests live in the same file as what’s being tested, is there really a point to having XCTest? I would think we would just have a stdlib/test where most of the testing primitives would be defined.

I completely agree that something needs to happen with testing and Swift, but it seems there are lots of different opinions of what that should look like. I foresee an epic thread coming up here.

A few things that come to mind:

  • If we have the @test methods side by side with our code, the line-count could potentially explode for lots of files.
  • If we need separate frameworks for testing, we still have to have a separate test target? (I assume importing them with @test import would only import them when running the tests, but we still have to specify them for copying them into some bundle or for the linker, which is most likely oblivious of such conventions?)
  • If I need mock classes and maybe JSON file fixtures, those should of course also just be compiled/copied when I am testing, but how would I do that?

If the answer to some of those (maybe stupid or naïve) questions is that I can still use separate test-targets, then this would let me to believe that we are creating yet another parallel way of testing (the others being custom testing frameworks for example that exist now in parallel to XCTest) which wouldn’t help the whole testing mess we have at the moment.

2 Likes

I’m happy my reply in that older thread sparked this discussion.

I find the following structure appealing:

// MyClass.swift
class MyClass {
    private func add(_ a:Int, to b: Int) -> Int {
        return a+b
    }
}

// MyClassUnitTests.swift
@test extension MyClass {
    func testAdd() {
         let four = self.add(2, to: 2)
         XCTAssertEquals(4, four)
    }
}

However, as has been stated above, separating the tests into an extension in another file wouldn’t solve the problem of testing private methods. Again we fall into a case where this functionality requires rules to be broken only for tests.

2 Likes

Just like under Rust, there is still room for a testing framework to augment the usual pile of assertions. One of my goals in having this discussion is that we can come to some kind of consensus to expose the hooks to make that happen. In a world of literate testing, XCTest does not have to disappear. After all, there will still be Cocoa code, older test suites to contend with, and people that agree with @Jon_Shier in that your unit tests should be separate from your code base. In more practical terms, XCTest also exposes timing primitives, assertions for asynchronous code, and exception testing that are quite useful by themselves.

The underlying idea in the language changes is that the test discovery part of XCTest should be generalized out of XCTest itself, but that’s really it.

5 Likes

This is certainly the primary criticism of literate testing paradigms. As a counter, I would invite you to explore what that looks like in practice, particularly for larger/more popular Rust projects. The consensus seems to be to use a mix of inline unit testing for smaller invariants/private members, and broader integration test suites to cover the boundaries between interacting modules. The idea here as there isn’t all the tests are now inline the idea is relevant tests appear near relevant code. When you start to speak about testing systems instead of components, then you’re talking about integration testing, and integration testing fundamentally cannot occur at the level of the individual component.

6 Likes

I understand and I think it would be pretty cool to do this sometimes. But this is exactly the point I think – it adds on top of the other cruft and doesn’t solve that.

I really, really want to see a better solution for exposing code to tests. Thank you for starting this thread! It’s especially exciting to see this topic brought up by someone with experience working on the compiler! :slight_smile:

I have worked with same-file tests in Ruby in the past. In some cases included tests in the same file (or even local scope!) as the code under test is really convenient, especially as the code is first being developed. However, as the code grows there are often better ways to organize the code. This is of course a value judgement, but it is one that is shared by a large number of programmers.

At the same time, I think there are advantages in placing test code in much closer proximity to production code than is usually the case today in Swift and Objective-C projects. Compiling tests along with production code in some build configuration(s) thus allowing tests to participate in existing scoping mechanisms is a good way to take advantage of closer proximity.

I do have some concerns about the details of this pitch. The first and most significant is that this pitch only makes progress on the visibility issue (relative to @testable) is if tests are placed in the same file as production code. As noted above, this is convenient when first writing code but it is an approach that doesn’t scale well (at least in the opinion of many of us).

This would help a lot, but I’m not convinced it is ideal either. It couples the physical structure of our test code with the physical structure of our production code. It is not uncommon to have a test code living in one file that tests production code that residing in more than one file (and vice versa). The ideal solution to the testability problem would not force a tradeoff between suboptimal physical structure and suboptimal access control.

A submodule system would help alleviate some of the pressure of scope-based testability if it allowed us to form scopes that are larger than a file but smaller than the whole module. It would allow us define submodules which contain some production code and some test code within a “submodule internal” scope. private and fileprivate symbols still wouldn’t be visible to test code, but at least the impact of bumping their visibility is limited to a submodule rather than the entire module.

Fortunately, the above approaches are not mutually exclusive. Some tests could begin life with the @test attribute in the same file as the production code they test. As code grows most of them could move to a test file that is “paired” with the file containing production code. Additional test code including test helpers can live in separate files within the same submodule and would have access to “submodule internal” symbols in the production code.

There is a lot to like about that direction, however, I don’t think its potential is fulfilled without a suitable submodule system (which I believe is out of scope for Swift 5). It is hard to evaluate a design for scope-based testability when we don’t yet know if Swift will ever have submodules and if so, what the design would look like.

In addition to the concerns listed above I would also want:

  1. The ability to have entire files considered test code without needing to annotate every declaration.
  2. The ability to exclude test code from debug builds when not running tests.
  3. The ability to have test-only external dependencies (perhaps the @test import XCTest syntax in the pitch is relevant here).
  4. The ability to have test-only resources (such as mock json files).
  5. There are probably other things we do today that take advantage of tests being in a separate target. I haven’t had time to think through all of the implications of losing the ability to configure the test target.

This all seems reasonable, but on its own is not sufficient. We absolutely must have the ability to define test-only code that does not meet these requirements, but is used by test code that does meet them. For example, we must be able to write domain-specific test helpers that streamline our test code.

Having said all of the above, I think the discussion may be a bit premature. As mentioned above, I have some reservations about committing to a design prior to having the submodule discussion.

One alternative that would solve the conflict between access control and testability and might be viable in the Swift 5 timeframe would be to add @testinternal which would promote the visibility of a symbol to internal when building for tests. This may not be the ideal long-term solution but it would be a tangible improvement and would allow more time to think about the longer-term design, perhaps allowing us to tackle submodules first.

I am excited to see where this thread goes and whether we can converge on a solution that makes a genuine improvement in the Swift 5 timeframe.

11 Likes

Can you be specific about how this is a non-solution?

I would say that more fundamentally, from an access control standpoint, white-box implementation tests need to be treated as part of the same access scope as the thing being tested. Adding more access control levels is not going to sidestep that fundamental property, and adding backdoors to access control so tests can bypass it is treating the symptom rather than the problem.

7 Likes

Yes, sorry for being non-specific. I imagine when not doing the in-line tests, we still need separate test targets for the reasons I mentioned above. That’s why I thought the in-line tests – while being a really good thing – are additive to what we have already – hence not a solution to the original problem.

Does that make sense? (I could be completely wrong of course and misunderstand)

Terms of Service

Privacy Policy

Cookie Policy