The Future of @testable

Evolution Discussion
1

Hello Swift community, I did a quick research and found almost zero input for this topic:

I myself always wondered why @testable, described as a hack in the first link above, can only allow testing internal members. Since we already moved to a soft private access modifier it won't make much sense to extend @testable to fileprivate only.

How hard would it be for someone to implement this feature and to allow testing all members of a module (open, public, internal, and the new ones fileprivate and private)?

I think the proposal for that feature isn't much of a problem, but the implementation for the review process is. If we can find someone who is willing to tackle the implementation and maybe is capable to provide the first version fairly fast, I think we could push this feature forward very fast and benefit from it in Swift 5.

3 Likes
2

Mangling doesn't include the file name, so where an access level allows multiple members to have the exact same name (two distinct fileprivate let foo, in different files), there is no way to refer to one but not the other.

3

@xwu would there be a way to detect this and disallow use of that member? That would be acceptable to me. It is very rare that this would be a problem in practice and there is an easy workaround with a scope of impact limited to a single file if there is a conflict: rename the member you need to access from a test.

4

This is just my bikeshedding thinking but what about something like this:

@testable(fooInFileA)
fileprivate let foo

Or something along these lines. This is similar to the following (source):

@objc(Color)
enum Цвет: Int {
    @objc(Red)
    case Красный
    
    @objc(Black)
    case Черный
}
 
@objc(Squirrel)
class Белка: NSObject {
    @objc(color)
    var цвет: Цвет = .Красный
    
    @objc(initWithName:)
    init (имя: String) {
        // ...
    }
    @objc(hideNuts:inTree:)
    func прячьОрехи(количество: Int, вДереве дерево: Дерево) {
        // ...
    }
}
5

Isn't that just defining another internal variable named fooInFileA?

6

@DevAndArtist do you think we really need that complexity? Why not just disallow the use of conflicting members in tests? They are private so by definition they are not part of an API, they are part of an implementation. Having to rename one of the two conflicting private members for the sake of testing seems acceptable to me.

If we really wanted to be able to expose conflicting private members to tests your bikeshed seems reasonable. I'm just not sure the extra implementation complexity is worth it. I suggest making this a future direction rather than something in the initial proposal. Reducing the complexity will increase the chances of this actually getting implemented.

7

Good question. I interpreted it as giving foo an alternative name that is only visible outside the module when it is imported with @testable.

8

It is totally fine by me if this issue can be diagnosed so that we can take action and fix it. I'm not opposed of a simple solution. In Cocoa-/Touch applications internal is the default and is used like public/open, therefore to hide the implementation I heavily use private/fileprivate in my projects, but this makes it impossible to test these members.

9

That was my idea too.

10

this has even bigger implications for cross-module inlining and encapsulation. if someone can think of a good way to prevent private name conflicts, we can allow inlineable functions to call private symbols.

11

Just wondering, setting aside the implementation details, have we decided that @testable exposing private members is even a Good Thing?

I thought the whole point was that private members are implementation details and that testing them is considered too fragile and therefore not desirable. That the whole point is to test public and at most internal API rather than specific implementations.

Maybe I swallowed the Kool Aid on that one too quickly but it seemed totally reasonable to me at the time and I have not missed “testing” private members since.

6 Likes
12

Shouldn't that decision be one of the developer that want to test his project? I mean, implementation detail or not, I want to test my private code for correctness and robustness without the need to expose it across the project. Look at iOS development, if you want to encapsulate the API the right way, you're simply forced to use private and fileprivate to hide it, but then you lose testability. That is a huge issue when someone else is working on the same project and misuse some exposed private API that is only made internal because it requires some testing. Prefixing the API with an _ isn't really a good solution nor does it provide any guarantees that the API won't be misused.

1 Like
13

I understand your point. You sound like me about 12 months ago :)

I have since found other ways to test the code while keeping a reasonable structure. Maybe I’m just getting older and more conservative but I think the way it is now makes a lot of sense :)

1 Like
14

@testable is best understood as a hack because, IMO, the idea of tests being a separate build product fundamentally runs against the nature of a language with access control. Tests ought to be treated as a fundamental part of the code they validate, just like types are. I would like to see the Swift ecosystem eventually evolve support for tests that sit alongside implementation, much like how they do in the D programming language:

In D, a unittest block can appear almost anywhere, and naturally has access to everything in that scope, and the compiler supports conditionally compiling all the unittest blocks so that they can be run in debug builds. This would be a large paradigm shift from what XCTest and Xcode support today in the Swift ecosystem, which is why @testable exists as a stopgap, but I don't think we'll really reach a satisfying testing story by making incremental changes to how @testable works. I believe the D model would fit much more naturally into how Swift as a language is designed.

19 Likes
15

I agree that this is a much better direction with the caveat that it will increase pressure to introduce a submodule system of some kind. Without submodules the only way this approach increases visibility of code to tests is if the tests are in the same file as the code under test. In many cases that would lead to files that are much larger than desirable. It would be an improvement on current state but would also be frustrating.

I don't have a sense of how much effort it would take to increase the hack of @testable or how long it would be until a better solution is available. It it's going to be several years before a more permanent solution is in scope and extending the hack doesn't take too long it would be worth doing, but I could easily see the tradeoff falling the other way as well.

16

I like that direction but I also see that as Matthew does. A unittest block is all neat and great as long as we can move it in a separate file, but it will probably take us years until we can introduce such feature. So I would rather prefer an extended @testable hack for now before we dreprecate it later in favor of unittest blocks.