[Pre-pitch] 'testable' visibility specifier instead of 'internal' for @testable?

uliwitness · 2021-07-28T15:43:05.506Z

@testable is a very useful feature, however I've recently had several bugs caused by accidentally accessing internal symbols, or rather, by forgetting to declare symbols in a framework public.

So I'm wondering if it was a mistake to make @testable allow access to all symbols with internal visibility, and whether it should be changed to just provide access to symbols that are explicitly marked with a special testable visibility.

For all other intents and purposes, testable would be identical to internal. The only difference would be that @testable import only makes testable symbols accessible, all symbols marked as internal stay in-accessible.

This would have several benefits:

It means unit tests can not accidentally access internal symbols that should have been made public. Unit tests can verify correct visibility of symbols too.
The few spots where internals are exposed to tests are explicit in the code, not just in the tests.

The downsides would be:

This would be a source-breaking change. All tested code would need to be updated to mark internal symbols used in tests with the new testable visibility. (Since accesses to internal symbols are a hard error though, I think it'd be possible to add a Fix-it for that)
If you are writing additional tests for someone else's library, you would be unable to write tests that use internal symbols without submitting an upstream patch to make their visibility testable.

What do you people think? Strong reasons against this? Would you want this too?

jawbroken · 2021-07-28T16:31:39.099Z

I personally don't want yet another level of access control. I think a better long-term solution would be to allow test code to be written in the same module/file as the code it is testing, so it could use the same access control rules as the rest of the language. This avoids having to design various ways to poke holes in the access control model for testing purposes and could allow @testable to be eventually deprecated. This has been discussed a few times here, but I don't have any links immediately to hand.

Edit: Here's one and another.

typesanitizer · 2021-07-28T17:06:50.083Z

jawbroken:

This avoids having to design various ways to poke holes in the access control model for testing purposes and could allow @testable to be eventually deprecated.

This isn't as simple as you make it out to be. Putting tests in the same file as the implementation requires other kinds of functionality. For example, import statements such as import XCTest would need to be marked with something that indicates "this import is for tests only." So you've mostly shifted the problem from marking imports in one place to marking imports in another place.

jawbroken · 2021-07-28T17:19:56.773Z

Sure, there are design issues to solve but none of them involve breaking access control like the current design does and any extensions would. You can then decide if you want to mark imports with an attribute, or just place them within the scope of the test (which would also be an independently-useful "scoped imports" feature), etc. And I was presenting it as an alternative to this pitch, which would greatly expand the places where things would need to be explicitly marked for testing, rather than just an alternative to @testable as it currently exists, so it's not merely a matter of shifting around import markers as you assert.

uliwitness · 2021-07-28T17:41:52.297Z

Sadly, putting tests in the same module as tested code would be the opposite of what I need for my above example issue.

At least at the moment I can forego the use of @testable to get the same behavior as any other user of my framework, including not being able to see internal methods so I notice when something important isn't public.

But if the code was in the same module, then this mistake would actually be even easier.

I actually see it as a feature that tests run like a framework client, exactly because that means internals aren't readily accessible, encapsulation is preserved.

jawbroken · 2021-07-28T18:23:33.100Z

When you want to test public interfaces you don't need any special technology or compiler support at all. Just put the testing code in another module and import as normal (or perhaps use some future submodule system). Your pitch is more about testing non-public code, which does need language support in some way. I just don't personally think fiddling with @testable and requiring marking of all @testable code is the right way to do it. In-line testing would let you test internal, fileprivate and private code at the exact point where you should have access to call or modify that code. And if desired you could come up with some spelling that would only allow access to the public interface from a particular in-line test block.

uliwitness · 2023-01-26T20:16:44.501Z

Good news! Looks like there's an _spi feature in the works that would let you mark any internal symbols that you want accessible by tests with @_spi(MyTestable) or so, and then do an @_spi(MyTestable) import MyFramework to make them available to a test (without the chance of accidentally using other methods marked as internal, especially ones you forgot to mark as public).

DevAndArtist · 2023-01-26T23:01:15.021Z

Checkout this thread: New Access Modifier: package