RFC: In-Line Tests

Evolution Pitches
81

I do think that unit testing needs a look at, but I think we do need to be careful that by revamping one aspect we don't limit ourselves in other ways, or start to encourage bad practices.

I do quite like the idea behind the doc testing. I haven't personally come across it before having not used such languages as D, Rust and Groovy etc. But after having a very quick look through their documentation I especially like D's examples.

I will say I'm going to have to firmly join the side of the not exposing private methods camp though. Generally, if your private methods can't be accessed by calling your public API, then there is a high chance there is something wrong with the architecture (this isn't always the case, but in a lot of cases it highlights issues, and rightly so). Also, in the context of the doc testing, you should only be generating documentation for API's that someone can use, it's pointless generating documentation for private methods.

As for the tests in the same file, I think thats an interesting idea for just getting something going, but could get unwieldy very quickly. As someone mentioned before, sometimes your tests can outnumber your code in terms of the number of lines and you don't want to pollute your implementation file with a ton of cruft that gets in the way of seeing what the file actually does. Personally the paired file approach seems like a nice idea and has the potential to clean things up a lot, especially with the docs too. Moving tests and docs into a paired file would clean up the actual implementation quite significantly. If we look at a lot of the standard library for example, there is a lot of documentation and rightly so. But sometimes when you just want to see the interface for something you end up having to scroll down through a ton of comments when all you want to see is a small concise list of methods or properties.

There are definitely a few avenues we can take with this, so it will be interesting to see where this discussion goes and what the majority agrees on.

82

Yup (doctest). The approach is great for any language having a REPL. I don't want to have to write all tests that way, but it is indeed very useful. That's how gyb is tested, for example.

5 Likes
83

The way this is being pitched is a lot different to a doctest. I am very much in favor of a doctest inline feature over the current suggested implementation. I guess we could have both. Could the author/s clarify this for us?

2 Likes
84

Just to add: The proposal doesn't mention startUp()/tearDown(). It also doesn't mention when the tests are actually executed. Would this be manually or from an IDE menu item that runs all tests? If running all tests what is the environment? Is there an instance of the type so that self has a meaning? Do the tests have to create an object to test?

Code like the following is a kind of poor man's inline tests that would seem to give most of what the proposal offers (although not much of the extras mentioned upthread). I would use a TESTING macro instead of DEBUG so the two could be set independently.

protocol Testable {
    func startTesting()
}

extension Testable {
    func startTesting() {
    }
}

class MyClass: Testable {
    init() {
        startTesting()
    }

    func lifeIsGood() -> Bool {
        return true
    }
}

#if DEBUG
extension MyClass {
    func startTesting() {
        testLifeIsGood()
        // testTwo() etc.
    }

    func testLifeIsGood() {
        assert(lifeIsGood())
    }
}
#endif
85

I really like the idea of colocating source and test code and also doctests. I'm in favour!

1 Like
86

I've used DocTests extensively in Python, and they were wonderful - with the exception that they effectively needed to stay VERY simple. Any sort of complex setup quickly exceeded what you could easily do, and there was a natural back-pressure as they were included in the generated documentation, so you didn't want to swamp the reference details with testing code.

DocTests was also briefly discussed/mentioned during the review of DocC's Snippets feature.

DocTests solve a specific problem space that nicely combined testing and documentation, but (to me) it didn't resolve the broader testing structure and usage issues.

1 Like
87

I'm a big fan of Python's doc-test style for unit tests, for the simple reason that I can write the tests at the same time. I write code very incrementally, so having a ready to run try-out useful during the coding is crucial. Having to go to another file to set up tests at that level is a bit of a pain.

I think I conceptually like the idea that both styles would work, then you could have white-box, expected-use tests with the code itself, and black-box, throw the world-at-it tests in a separate file.

1 Like