RFC: In-Line Tests

Having an associated test file seems feasible to me, since it aligns well with the access control model (the test file can be treated as "the same file" as the implementation) and therefore gets most of the benefits of inline tests without requiring that tests and implementation necessarily live side by side.

7 Likes

@RenanGreca's approach of declaring a @test extension in another file would seem like a more natural way of going about it, without having to rely on special file names or types in order to be picked up.

4 Likes

I'm thinking on what would be easier to integrate in an IDE or a linter. Two files seems easier to integrate at a first glance. The filename imposes a restriction on how many files could be candidates for pairing (one) and seems trivial to integrate in an IDE (where you'd maybe like to see the testing counterpart for a file as soon as you open that file, both side by side).

That said, using @test extension should work fine as well.

// Edit

Maybe I spoke too quickly. The @test extension model would need to issue a compilation error whenever the files break the rules:

  • Two different files extend the same swift file.
  • One file extends two different files
  • One file has both program implementation code and testing code.

That's a lot of work for the compiler to do. I think in this case Joe's idea seems like a better fit overall:

  • There can't be two different files that extend the same file. They would have the same name, which is impossible.
  • The compiler knows what scope a testing file has access to by default. Therefore, they won't compile right away if they try to access another file's fileprivate scope.
  • And most importantly, program implementation inside a testing file is mostly irrelevant, because they aren't compiled outside of testing situations and because their scope is unreachable from other files.
1 Like

The problem with allowing a test extension in an arbitrary file is that it breaks the normal compilation model for private things, which are expected to only be visible in the current file.

3 Likes

I feel that tests in another file should be black box tests and have the regular, non @testable access levels. It would make sense that if you wanted to write tests that used private access levels you would write them in the same file, beside the implementation. Maybe @test shouldn’t escalate access control at all.

I agree! However, for the people in this thread who raised concerns about code organization or file size with tests and implementation side-by-side, this might be a reasonable thing to support.

6 Likes

I would welcome tests in the same file and would like doc tests. I think the success or otherwise of tests in the same file will depend on how well the IDE presents them.

However I would also like to be able to do black-body testing in a seperate file. In particular I sometimes add tests for libraries that I don’t own to ensure they have the behaviour I expect.

4 Likes

I don't usually chime in, so please feel free to provide feedback at will, especially if there is more detail needed. :slightly_smiling_face:

I am very pro this proposal in spirit. I'll avoid much commentary on implementation details because I don't feel I'm qualified for that task, but would still like to chime in with my views on the subject at hand.


Bringing the ability to do white-box testing without having to mark something internal would be a huge upgrade to the state of testing in Swift today. I very often find myself deciding between two options:

  • Testing code and exposing it to consumers within the framework.
  • Leaving it private as I would like to, and leaving the code untested.

In an ideal world I would not be forced to choose between the two. I am really happy to see the possibility of that being raised.

Since this isn't a replacement for XCTest and only an addition to it, the ability to do black-box testing will still be there, so I can't foresee any conflict between the two.

Overall I'm very excited about this idea. :+1:

8 Likes

I like this idea more as part of documentation than as a testing solution, if only because the compiler would need to gain the ability to actively parse header docs, which would hopefully lead to things like automatically checking that the documentation matches the declaration. So, a limited version of the original proposal could be quite useful without the other issues that concern me.

I like this idea as well. I really like how this is done in D (e.g. unittest {} blocks) and I also like the conception of being able to test private implementation, as sometimes those pieces are harder to test individually in all of their edge cases.

I think some of the comments mentioned above would be resolved by the notion of leaving both options open - I don't think they conflict in any way. Having in-line tests would be an awesome addition, and not a burden forced on a developer by any way.

I think this would be a great language addition in general.

2 Likes

Yes, this is mostly the point of doc-tests. Show how to use the code, and have the compiler verify that the example does what it says.

Doc-strings aren't actually mentioned in the proposal. They were brought up as a general nicety.

1 Like

I essentially mocked up this feature in Swift and it looked like this: Tooling Around – Testing in Swift – owensd.io.

It was fine, but honestly, I didn't feel like I actually gained any significant amount of clarity or understanding of my code base by having tests inline vs. in some separate file. The thing I did find annoying was having different test runners for different types of tests. That required infrastructure to unify, and I found that it was the infrastructure and reporting that added a lot more value.

I feel like these types of additions are in response to the access control problem (which is the topic that shall not be named - I know), not necessarily that this approach has a significant practical advantage over other methodologies of testing that warrants a language-level change.

For instance, feeding in expected input/output files instead of writing code for tests is another extremely valuable approach to even testing at the unit level. I don't feel like this approach really enables that, but focuses specifically on one type of test authoring.

I would much rather see a more primitive building block added here that allows us to provide hooks into the compiler to re-write or to contribute additional code-gen into the pre-compiled state of the swift file. This feature is easily a great candidate for that, while also giving us hooks to build the right type of testing tools (and other code-gen tools) that our projects could leverage.

So while I think the proposal is interesting in its own right, I think it ends up providing a very narrow solution to a much more general and broad problem.

2 Likes

I like the idea very much. I also recognize that this is not a doc test feature per se. I see this as complementary to @testable

  1. I think @test extension Blah should be allowed in separate files that follow the same basename of the original. Something like mySource.swift mySource.Tests.swift
  2. Lets double down on accessibility by allowing @test extension be able to see private members as long as they are in source.test.swift files.
  3. We should be able to designate @test inits which would have initial code to be test.
  4. Perhaps overusing extension may not be a good idea but something along the lines of testBlock {} would be nice as an alternative.
  5. I do not think @testable should go away.

I'm a big fan of this idea, and would love the ability to co-locate a test function right next to some private function. I've recently fallen in love with Python's doc testing, but I think fuller testing as proposed in the original post would be a better and more flexible system. Those who wished to do so could and those who preferred the current of way of organizing their tests how we do currently. Seems like a win-win?

3 Likes

This is a great idea - I love it in Rust and I would love it in Swift :+1:

1 Like

A more general feature that might support the test use case in addition to other use cases would be to have a general mechanism for registration-style attributes, and the ability to ask for the list of all functions or types decorated with a certain attribute. Is that the sort of thing you had in mind?

1 Like

A more general solution might be design by contract were the tests (contracts) are inherited and are part of the documentation.

A more general feature that might support the test use case in addition to other use cases would be to have a general mechanism for registration-style attributes, and the ability to ask for the list of all functions or types decorated with a certain attribute. Is that the sort of thing you had in mind?

That's one approach, but that doesn't solve the problem. This is getting a bit off-topic here, but Swift being a static language comes with much of the baggage of static languages with no ability to really generate code.

I don't merely want to query for code decorated with an attribute, I want a hook to run a set of code before the compiler compiles the code in question so that I can do any number of interesting things, such as generate the body of a test method, create a whole set of test methods based on input/output files, etc...

This is possible today to do with outside tooling and hiding the interesting bits in comments and then having your custom tools run before builds. However, this approach is pretty ad-hoc and requires investment in tools that break outside of people being able to just run build.

In regards to the proposal, I could do something like this:

@tests(
    category: "math.addition.basics",
    valid: [
        (2, 2, 4),
        (-1, 1, 0),
        (124, 2, 126)
    ],
    overflows: [
        (Int.min, -1),
        (Int.max, 1),

        // I should also be able to generate the permutations instead of also declaring them here...
        (-1, Int.min),
        (1, Int.min),
    ])

Writing out the actual code that runs those tests is tedious, error-prone, and inflexible. It's also overly verbose and takes up a lot of screen space for what amounts to just busy-work. This is one of the reasons for the popularity of BDD-style tests.

Especially as the goal with more functional-style code authoring, unit tests basically boil down to "with these inputs, I expect these outputs."

However, I personally want this feature for things much more than tests, such as generating properties on enumeration types, etc...

3 Likes

I am in favor of both inline tests and doctests. These can and should stand alongside the current testing methods.

5 Likes

Fair enough. I think it's a good idea to keep this in the back of our minds so that a solution to the testing problem interacts well with improvements in modularity.

What I meant here is that I, like many people, prefer to keep files to a few hundred lines when possible. I find it easier to navigate code when I do this. This will be the case for as long as files are a fundamental unit of storage exposed by development tools which is something I think we should take as a given for the sake of this discussion.

Keeping file sizes manageable is hard enough without having to fit our test code into the same file as the implementation. This is especially true in Swift which uses files as a boundary for access control (at least for those of us who don't like to expose symbols that are only intended to be part of the implementation).

I looked at some code from a recent project to see how Joe's suggestion of paring a test file with a production file would play out. At a glance, I think it would allow me to uncomment all of the /* private */ annotations I have if I restructured the test code a little bit. This looks like a very promising direction.

It would be even more promising if instead of pairing a production file with a test file we allow a one-to-many relationship. There will be cases where there is enough test code for one production file that it would be very desirable to break up the test code into multiple files.

There are some other advantages to distinguishing a file as a test file. It solves the "test-only" code problem I discuss below. If I my test code lives in a .swifttest file I will be able to use arbitrary features of the language while still omitting the code from non-test builds. @test can be used within that file to indicate types and methods that should be executed as tests.

I write test helpers that make use of arbitrary features of Swift. Most of this code lives in test-only types. Some of these types require initializer arguments. I suppose it would be possible to declare these types nested inside an @test type with a default initializer, but requiring that would be unfortunate. It would result in nesting that is sometimes less than ideal.

Some of the test helper code resides in extensions on production types. Perhaps all we need to do to support this would be allowing @test on extensions, therefore omitting the extension when building without test code.

I'm going to speculate that the reason for the restrictions you proposed are that a test driver needs to be able to instantiate @test types and call @test methods. The restrictions you propose make sense for this purpose but there is still a need to support arbitrary test-only code, ideally without requiring nesting in an @test type.

Can you elaborate on this? This appears to suggest that all member functions of an @test type would be tests. This is not desirable IMO. I often write helper methods on test classes that are not themselves tests. These helper methods often require arguments and should not be run independently when the test suite is run.

This makes sense for actual test cases. But I would still like to be able to write test helpers that participate in access control. I write test helpers that are used by a bunch of tests and live in separate files so they must be internal.

Test-only code should only be visible to other test-only code and it is fine to require actual @test cases to be private, but other test-only code should interact with access control in the normal way.

I think a bigger open question in my mind than any of the above concerns is what other kind of test-only build configuration options we need. I mentioned test-only resources (mock json files, etc) in my previous reply. What else are people commonly doing with test targets and how would we address use cases in this new system of testing where tests live within a module?

Bummer, I may have misinterpreted. I thought maybe you were considering working on this for Swift 5. It's good to see this conversation moving forward anyway. :slight_smile:

2 Likes