GSoC and Swift Testing

[maartene (Maarten Engels)]

OK, so it took longer than expected to create this topic. But I guess it is still OK as we already applied as an organization?

Anyway, as mentioned in this topic, what I would like to pitch is that we use GSoC to accelerate progress on swift-testing: in my opinion of the most important projects currently underway in the Swift ecosystem as improvements to testability benefit everyone.

So, these are the things I was thinking about as possible improvements:
1 Use a closure for the test code (instead of having to make - and name - a function)
2 Nicer test output
3 Easier integration for test doubles
4 Testing fatalError, assert, precondition (has its own thread )

Ad 1. Swift Testing uses the following syntax:

@Test("Some description of the test, i.e. some Gherkin text or anything else") func actualFunctionName() { // your test code here }
Using the description to name the test is very convenient. It is close to BDD style test frameworks and is versatile in that it supports anything a String supports. However, naming the function is somewhat inconvenient. If you are like me, you more or less copy_the_description_in_snake_case. It would be an improvement if you could define a test as a closure:
@Test("Your description here") { // your test code }

Ad 2 Nicer test output

Swift Testing uses XCTest under the hood, which means that tests report a big wall of text. Finding the failed test can take some searching, especially as there is no coloring or anything. And there isn't even a list of tests that failed.
Ideally, we'd have something like Jest that provides a very nice view of test results. And when tests fail, which test failed and where it failed. Add in features to 'watch' for code changes and automatically rerun failed tests. Very nice. Makes TDD in JavaScript/TypeScript very fast and with minimal friction.
Could we create something like this? Perhaps we can't rebuild XCTest, but do something to make parsing the results easier?

Ad 3. Easier integration of test doubles

This one came up in the forum post a few times. This is not my field of expertise, so I hope others can chime in. What I do know is that I too have a lot of problems effectively testing UI, testing notifications and testing frameworks such as CloudKit.

Ad 4. Testing fatalError and friends

There is a whole forum topic associated with this issue. It describes both the issue (i.e. can't test code paths that trigger these functions) as well as potential solutions. Or workarounds typically. Personally, I am of the opinion that a convenient way of testing these functions would help in safely using these functions where they are applicable.

Anyway, these are the projects I collected earlier. I can imagine there are way more opportunities where we can benefit from accelerating swift-testing. So please chime in, using this topic.

Cheers, Maarten

[grynspan (Jonathan Grynspan)]

Hi again Maarten! We're mulling over your post here and will reply shortly. There are four distinct topics here and we want to make sure we give each one a thoughtful answer.

[allevato (Tony Allevato)]

@maartene

Ad 1. Swift Testing uses the following syntax:

@Test("Some description of the test, i.e. some Gherkin text or anything else") func actualFunctionName() { // your test code here }
Using the description to name the test is very convenient. It is close to BDD style test frameworks and is versatile in that it supports anything a String supports. However, naming the function is somewhat inconvenient. If you are like me, you more or less copy_the_description_in_snake_case. It would be an improvement if you could define a test as a closure:
@Test("Your description here") { // your test code }

FWIW, functions and closures have subtly different semantics and I'd be concerned about an approach that blurs the line between the two. Especially when it comes to debugging; the framework needs to invent some name for that test for stack traces, so its options are to either construct something completely obtuse, or to try to transform the description into a name, in which case you're back_to_something_like_this except now you don't control it.

(I have other reasons for wanting this but) I would personally love to revisit a subset of SE-0275 so that we could just name the test function based on the description. This solves all the problems nicely: the user doesn't have to contrive a separate name from the description, and stack traces and other tools would use the same name the user gave the function.

[maartene (Maarten Engels)]

@allevato

(I have other reasons for wanting this but) I would personally love to revisit a subset of SE-0275 so that we could just name the test function based on the description. This solves all the problems nicely: the user doesn't have to contrive a separate name from the description, and stack traces and other tools would use the same name the user gave the function.

That would actually be a very nice way of solving this (and has applications elsewhere as well). I didn’t know this SE. I’m going to read up on this. Thanks!

[smontgomery (Stuart Montgomery)]

Hi @maartene! Thank you for your interest in proposing a GSoC project for swift-testing, we welcome involvement from you or anyone else who is motivated and interested in participating. Those of us who primarily maintain swift-testing have not acted as mentors for a GSoC before, so please bear with us as we navigate this process for the first time and explore what kind of project might make sense.

You put together a nice list of topics, so I'd like to consider each of those individually and offer some thoughts below.

@maartene

1 Use a closure for the test code (instead of having to make - and name - a function)

This is a topic a few others have commented on and offered arguments in either direction. I think there are two questions to figure out in this space:

1. Is it technically feasible; and

2. Is it a generally worthwhile and good feature for swift-testing to include.

Re: 1, @grynspan has experimented with implementing this in the past and encountered some technical limitations both in Swift Macros and in the underlying language and compiler. He may be able to reply with more details if it would be helpful, but solving those technical challenges would likely be a prerequisite to the introduction of such a feature in swift-testing.

That doesn't mean it isn't worthwhile to consider this idea for GSoC, though! If these technical blockers could be lifted through improvements to the language or tools (likely through contributions to Swift and related packages like swift-syntax), that might allow swift-testing to eventually adopt this as a feature. As a community, we would still want to discuss question 2 above though (whether it's worthwhile) before committing to this.

@maartene

2 Nicer test output

To me, this topic seems the most immediately actionable for a GSoC project. @Sajjon put together a compelling prototype of "live progress" console output a while back which is worth looking at for inspiration.

I commented on that draft PR at the time that, in order to realize that idea, we would probably need to implement a bit more infrastructure to have output generated from a supervisory or "harness" process. Some of that work our team has considered beginning soon, in support of our goals around tools integration. So a GSoC project along these lines could be viable, and may involve some combination of:

• Targeted improvements to the existing output, such as a summary or richer colors;

• Introducing a supervisory "harness" process, if such a mechanism hasn't been built yet; or

• Leveraging the harness mechanism to implement live reporting or "watch for failure and rerun" functionality.

Re: item 1, the console output has already been getting regular, incremental improvements so it might not represent a full GSoC project, while the latter two items would be more involved.

Note: swift-testing originally used XCTest to drive execution and reporting, but it no longer requires this when using SwiftPM from recent Swift main snapshot toolchains.

@maartene

3 Easier integration for test doubles

This topic is very interesting, and frequently requested by the community. I would love to see work in this area, but I believe the fundamental limitations which make it difficult are in the language, so it's not clear to me that a GSoC project for swift-testing specifically would be actionable. If anyone thinks there are opportunities for swift-testing improvements in this area though, lets discuss more.

@maartene

4 Testing fatalError, assert, precondition

This topic overlaps some with "Nicer test output" above, in that we believe fully realizing this goal may require a supervisory or "harness" process architecture. A project in this space may also involve adding new traits or other APIs to allow users to control this behavior. This is definitely an area where we'd like to improve, and AFAIK it's not blocked by any fundamental langauge or compiler limitations, so I'd consider this pretty actionable, too.

Hope that's helpful, and I look forward to hearing whether any of these responses resonate.

[maartene (Maarten Engels)]

Hi @smontgomery , thank you for the extensive reply!

From what I read it seems that the "supervisory harness" might hit two marks in one blow: allow for nicer test output (incl live reporting and rerun on failure) as well as allow for testing fatalErrors and friends. So in my book that sounds like a good start for a project.

If "closures as test definition" and "test doubles" really need changes to the Swift language, these might fit SE better.

So I would be interested to hear from you and others: should we zoom in on the supervisory harness or explore more ideas?

Note: swift-testing originally used XCTest to drive execution and reporting, but it no longer requires this when using SwiftPM from recent Swift main snapshot toolchains.

Didn't know! I am definitely going to try this out.