Unit Tests in Source Code

Evolution Discussion
,
1

Hello everyone! I know this topic has been discussed before, but most arguments ended with the claim that unit tests in source code would slow down compilation speed. Please feel free to link any relevant discussion if you feel more nuanced arguments were made, but here are some of the benefits of unit tests in source.

Testing Becomes a Primary Concern

Testing is usually treated as an afterthought due to the boilerplate required and the context switching between the test and source-code files. By having tests and the tested functionality in the same file, users will have an easier time figuring out what they need to test and doing the actual debugging.

Preserved Access Control

With the current approach of @testable imports, users often mark otherwise fileprivate declarations as internal. This not only pollutes the namespace but also reduces local reasoning over code, since it's easy to forget that a property shouldn't be touched outside of its file, accidentally introducing a bug in your code. In other words, current testing practices undermine access control.

Easy Documentation

Easily accessible tests act like a kind of documentation, that can help maintainers understand what a piece of code is supposed to do and how it should be used. Even if other documentation perfectly describes a given declaration, seeing how it should be used in one place is always helpful.

Limitations

Of course, the reason why unit tests are separated from source code isn't just because some engineer didn't of the benefits; tests within source code present significant challenges.

Compilation Times

Swift is already criticized for its slow compilation times, but adding tests in the source could slow down the compiler even more. At the end of the day, unit tests are more code that needs to be parsed and type-checked. While there may be some improvements from stricter access control as discussed above, many large projects (such as the compiler itself) separate building the source and tests to accelerate compilation times.

Burdensome Prototyping

When prototyping, most users quickly implement a feature, debug it, make adjustments, and repeat until this feature meets some criteria. Testing usually comes after that to properly enforce said criteria and ensure that there are no regressions. If the program takes longer to compile, the feature will take longer to implement. Pototyping is further inhibited in the case where a tested feature is extended (e.g. changing from a struct from a CustomDebugStringConvertible -> CustomStringConvertible conformance). In this case, the user would have to implement the new conformance, but also change the tests that reference debugDescription to description. These changes happen before actually running and debugging the new code, thus adding an extra, arguably unproductive step between the implementation and debugging steps.

Potential Solutions

While the limitations are significant, they can be circumvented. The benefits mostly arise from unit tests being in close vicinity to their corresponding source code, while limitations are a result of being forced to build source and test code together.

Rust

After some quick reading, it seems that Rust uses #[cfg(test)] in unit tests, which appear near production code, to tell the compiler that the unit tests should only be compiled only when cargo test is run. In Swift, that would be similar to enclosing unit tests in #if(Test) when they are in the Sources directory. This way, all the benefits are preserved but the limitations are mostly alleviated since code in #if directives will only parse. Arguably, a compiler directive to also bypass parsing of the enclosed code would ensure minimal effect on compilation times.

SwiftUI Previews

SwiftUI previews are not exactly tests, which have their own directory within a SwiftUI app, but are a way of verifying code within the source file. They only featured #if directives in the preliminary betas, but that was dropped because they usually delegate to a view and are a direct way of debugging the given view. Nevertheless, they prove the need of verifying code within the source file.

All in all, XCTests have carried a lot of technical debt from their Objective C days and have started to show their age. Tests are still defined in classes and the assertion methods are prefixed with XCT. I think we have an opportunity to design a testing library that is consistent across platforms and better adheres to Swift's design principles. Whether unit tests should be written alongside source code is just one design consideration, of what should IMO be an overhaul of testing in Swift.

23 Likes
2

My organizational approach evolves over time and I can’t say whether I would end up permanently going with the approach of having my unit tests in the same source file as the tested code, but I think it’s a cool idea and I think it’s entirely possible that I would settle on that as the permanently best approach, so I would love to have the option.

+1

3

That's great to hear! Even if not in the same file, having tests in a subdirectory within Sources will still be less ceremonious, since the package manifest won't have to change (if the package is the only dependency). I'm interested to hear about your current test-organization methods.

4

At least in golang I prefer to write test as I am writing the feature. In fact for the vast majority of the time, the tests are the only consumers of the code I am writing. In swift perhaps I would move to using snippets for the same case ( exercising the code ).

The doctest feature is cool but I would prefer having test files along the source code.

// somePackage
feature.swift
feature_test.swift

2 Likes
5

You’re right, I was a bit too rigid on this point. What I meant was that not all feature implementations benefit from test-driven development. Thus, it’s key to have that flexibility of either running or omitting tests when prototyping.

Another case I alluded to in the original post is when tests are tangled with the interface of a feature (e.g. a change in protocol conformance). So while the new protocol’s requirements may be prototyped, other tests that do not directly correspond to that protocol may need updating to run. This is obviously not ideal for a non-finalized, upgraded protocol, since time would be spent updating unrelated tests to an interface that could possibly change.

You should be able to have this structure under the proposed model but feature_test.swift would only have internal access to feature.swift. At least the way I think about it there isn’t an elegant way to allow for private/file private access without excessively complicating the package manifest.

6

I can list some other downsides, but before I do I want to say I do like the idea of tests embedded in code, I just don't think it is appropriate to all environments or scales of work. I think it would be fantastic for starting out on smaller systems and/or prototype work.

However in a large system tests may end up needing fakes of other parts of the system adding dependencies on code (other external modules) that you don't even want in a production system. It is also sometimes useful to make some additional classes/structs to manage "fake test data" in a more natural way to set up interesting testable states.

So I think it would be great to optionally have tests intermixed with the "real" code, I also think it is very important not to lose the ability to have the tests in separate locations.

1 Like
7

Agreed; at least at the beginning only the dependencies available to the actual source code should be available to in-source unit tests. In the long term, there should probably be availability annotations so that libraries can define their test API without any additional imports for the unit tests. Another useful feature would be to have explicit test-only dependencies defined in the package manifest. This would be similar to how dependencies for tests would be declared today but probably in the dependencies of what is currently the production/source target.

Regarding organization, tests would not need to be in the exact source file of the API/functionality they target. Instead, it should be possible to just nest them in e.g. FeatureA/Tests and FeatureB/Tests. At this point, we might as well use the existing structure of Sources and Tests. I personally prefer having Sources/MyLibrary and Sources/MyLibrary/Tests because it’s less ceremonious and would probably require less configuration. Nevertheless, test organization should remain flexible and retain support for both styles.

1 Like