A New Approach to Testing in Swift

Here's an example of some test code for a crossword project I worked on. (Blah blah views blah blah employer, this was a side project.)

This code tests arrow-key movement in a crossword grid. Inside each Pair is an input (a board selection state plus a movement direction) and an output (a resulting board selection state). The assertCustom function comes from TestCleaner, and handles some boilerplate around unwrapping the test values and keeping track of what lines they came from so we can forward those to XCTAssert*.

// n.b. 'S' and 'C' are local test utility functions to make Selection and Coordinate values.

let testCases: [TestPair<(Selection?, MovementDirection), Selection?>] = [
    // Invalid state. Jump to top left corner.
    Pair((S(blackSquare, .horizontal), .up), S(upperLeftCorner, .horizontal)),
    Pair((S(blackSquare, .horizontal), .down), S(upperLeftCorner, .horizontal)),
    Pair((S(blackSquare, .horizontal), .left), S(upperLeftCorner, .horizontal)),
    Pair((S(blackSquare, .horizontal), .right), S(upperLeftCorner, .horizontal)),

    /* ✂️ snip about 80 similar lines ✂️ */

    // End of 29 Down

    Pair((S(endOfTwentyNineDown, .vertical), .up), S(C(col: 7, row: 8), .vertical)),
    Pair((S(endOfTwentyNineDown, .vertical), .down), S(C(col: 8, row: 5), .vertical)),
    Pair((S(endOfTwentyNineDown, .vertical), .left), S(endOfTwentyNineDown, .horizontal)),
    Pair((S(endOfTwentyNineDown, .vertical), .right), S(endOfTwentyNineDown, .horizontal)),
]
assertCustom(testCases: testCases) { (pair, file, line) in
    let ((selection, direction), expectedResult) = try (pair.left, pair.right)
    // 👀 This is the actual function being tested
    let next = nextSelection(from: selection, direction: direction, puzzle: initialState.puzzle)
    // ✅ And here's the assertion
    XCTAssertEqual(next, expectedResult, file: file, line: line)
}
1 Like

This all looks a hopeful step forward, thank you and others for your effort so far! Very excited for the renewed focus on testing.

Off the top of my head, the biggest and most obvious difficulty and challenge is designing and controlling dependencies so code can be testable. For example, let's say the FoodTruck has to make a network request or read files from disk to determine quantities of food. How should I design that dependency? How do I control that dependency for this test case? For another example, lets say the FoodTruck tracks a "quantity check" event so my employer can measure what are the most popular foods and understand our users better. How should I design the event tracker? How I control that dependency to verify side effects and general correctness of the code?

Perhaps another way of saying this is: the difficulty testing in Swift today is a lack of discipline with program architecture and dependency management. These problems are an obstacle to testing because they tend to be discovered after the fact when it is very difficult to change and increase the testability of code.

I would say another challenge is Apple. Their APIs are not particularly conducive to stubbing. Most of us here are probably developing for Apple platforms. I cannot easily stub a URLSession and I cannot stub the Core Location API at all. So an explosion of boiler platey wrapper types ensures. Also, related to the architecture problem, the way Apple encourages people to design apps does not help (e.g.: how do I verify the correctness of my SwiftUI view when it appears?).

I know some of what I've said is a bit more higher level where swift-testing seems to be laying foundations. However, I would wager that most of us are building features or SDKs for users, not writing algorithms. We want the confidence to make changes to our codebases while at the same time be sure our features will continue to work. I hope these foundations have that end in sight and can serve it.

If you and the team working on this have not already, I highly recommend you check out the work of PointFree: swift-dependencies and their Dependencies collection.

8 Likes

At the risk of going off topic, I wrap APIs like these in protocols and then provide various implementations of those protocols, including stubs, spies and fakes (and of course one with the real API implementation underneath). Are there situations where this isn't feasible?

2 Likes

The biggest pain point, for me, is around the build system and tooling. When developing in Java or Kotlin, I can write implementation code in the tests that doesn't compile (for example, a new function that I want to create), and the IDE will immediately recognize that this function doesn't exist in the implementation and provide shortcuts for its creation. The same thing goes for things like new parameters in functions, new properties in data classes/enums, differing parameter types, etc.

When testing in Xcode, live errors and warnings in the editor for test code sometimes don't match the reality of the main target (presumably because caches aren't up to date). I have to trigger a full build. When "fix this" helpers in Xcode appear (not as often as I'd like) they usually don't have the desired solution. When it is what I want, for example implementing protocol stubs, there doesn't seem to be a keyboard shortcut to do it quickly.

2 Likes

Thanks for sharing. This looks promising.

Here are a couple areas of opportunity that would be great to see:

  1. I wish the assertion functions were strongly typed to clearly disambiguate between what was expected vs what was the actual observed value. Just having an expectation that x == y gives no indication whether x or y was expected.

  2. (Mentioned earlier in this thread) Allow a mechanism to allow sample code in docc code snippets to be tested. It’s too easy to document an example usage where it either doesn’t compile or run, or over time gets stale when compared to the real API being documented.

Btw, both of these capabilities exist in C# tooling.

2 Likes

This looks really cool.

Have you considered supporting nested tests like Quick (BDD style)? XCTest supporting only a flat test structure was the number 1 reason we decided to migrate to using Quick/Nimble.

This is really exciting!

One of the primary motivations behind the proposal SE-0385 (custom reflection metadata) was to establish a discovery pattern for XCTest, and you have effectively addressed this by utilizing macros and compiling test cases into a static array.

Could we consider extending this discovery mechanism to a more general context? The motivation section of SE-0385 outlines several use cases that could greatly benefit from similar approaches.


Update: My mistake, it appears that the library utilizes a function called swift_enumerateAllMetadataSection that is exported by the Swift runtime that enumerates all metadata sections loaded into the current process.

While this approach successfully addresses a specific use-case mentioned in the original proposal, it does so at the library level, bypassing the formal Swift evolution process for wider language-level adoption.

The one thing that’s incredibly painful in swift testing is:

MOCKS

Find a way to do those at test runtime without generating dumb class files and I will hug you

2 Likes

This is really cool! Thanks to everyone who worked on it—I'm really looking forward to using it to replace XCTest, which is definitely showing its age when using it in Swift.

One thing that I wasn't able to glean from the documentation so far is what kind of capabilities will there be for customizing expectations, in the sense that other testing frameworks can have custom matchers? For example,

  • If I write an #expect statement that compares two multi-line strings, I'd like the failure output to be a diff of the two strings.
  • If I write an #expect statement that compares two protobufs, I'd like the failure output to be a bit more semantic (this field is missing, this field has the wrong value), and perhaps even be able to say "ignore this field but compare everything else".

Anyone can write a boolean function to do those comparisons but the desirable feature is being able to customize that to provide structured failure feedback. I'm really curious what your thoughts are about how that kind of functionality could fit into the #expect(...) syntax!

11 Likes

snippets?

1 Like

I've played around with using Result Builders to define a new DSL for Quick, but didn't have time to polish it off.

When macros came out, it got me wondering: What's the correct mechanism for meta-programming? Are there rules of thumb for choosing betwen result builders and macros?

How is the performance of having so many macros in a system?

I would naively assume that Result Builders can be faster because they run in-process in the compiler (and because there's incentive to optimize them heavily given how many SwiftUI views would be in a typical project), and don't need to start up a bunch of subprocesses to run the macros, is that the case?

1 Like

I think the issue isn't feasibility, it's that each of us is duplicating that effort. Perhaps it would help if Apple provided those protocols alongside their APIs.

4 Likes

Hi Laszlo! Our colleagues are actively investigating a general-purpose solution for runtime symbol discovery and we're eager to adopt it once it becomes available. For more information on that effort, check out Kuba's pitch here.

4 Likes

Hi Tony! We know there's interest in having custom matchers that use the #expect() macro, and it's definitely something we'd like to support. While we have some ideas here already, we want to hear more from the community about what would be useful.

With regard to string comparisons, we have built in the ability to compare any two collections (using the Swift standard library's CollectionDifference API.) Strings are special-cased and opt out of this extra handling right now, but we hope to change that in the near term to something similar to diff.

I'd love to discuss this further with you—mind starting up a separate thread under the swift-testing subcategory and we can brainstorm?

5 Likes

While developing swift-testing we also experimented with using Result Builders for test definition, but encountered several significant challenges which I described under Alternatives Considered in our vision document draft. Indeed, one of those challenges was the burden we knew this approach would place on the compiler's type-checker, especially since test code can often be quite lengthy compared to other APIs which use result builders. There were other notable difficulties beyond that, too.

By contrast, we believe our current macros-based approach is much simpler from a type-checking perspective since the @Test and @Suite macros expand to more ordinary Swift code consisting of named functions, properties, and types. Although it's true that macros can involve inter-process communication between the compiler and a macro plugin, in practice we've found that type-checking is the more relevant factor when examining our tests' build times.

8 Likes

I briefly reviewed the documentation and sources, and I didn't find any information regarding support for expected fatals. Did I miss something?

1 Like

Looks interesting, look forward to tinker with this. Now, if you could somehow integrate a gherkin parser to enable bdd testing! Current frameworks out there like cucumberish or xctest-gherkin are quite outdated (still very useful though)

This is very exciting – a huge thank you for rethinking XCTest!

Over the years I've gone back and forth between Quick/Nimble and XCTest. Same with Rspec and Minitest in Ruby/Rails world. But I keep coming back to the simplicity of XCTest and Minitest.

Nested contexts, before/after blocks, subjects, and shared examples always seem to confuse me more than help. Sure, it looks great and I have DRY test code. But I come back to the suite a week or month later and lose myself in my own code.

I would hate for XCTest to leave this and be more a "BDD-like" test framework. I see some inklings but also some stuff that I love in the proposal so far.

My two biggest gripes for XCTest are:

  1. Naming tests via function. It looks like @Test() solves this right away. I am most excited for this! I would be just as happy if Xcode 16 launched with this single improvement to XCTest. I wrote about a workaround but it has the problem @ratkins mentioned: Xcode can't discover each test.
  2. The limited usability of matchers and output. Without a ton of custom helper methods, test failures usually read "expected this to happen and it didn't". Which isn't super helpful compared to other test frameworks.

That said, it looks like I'll be very happy with this when it launches! Looking forward to following along.

5 Likes

This is very exciting! I especially like the possibility to do parameter-based testing. Is there any chance/interest of extending this to also accommodate property-based testing a la SwiftCheck? Or at least allow third-parties to integrate and provide this functionality? Maybe in addition to providing the parameters directly, the parameters could be given by a type conforming to ParameterProvider or similar. Then third-party frameworks could provide composable value generators that test authors could use to generate parameters. The other half would be the property definition which would stop and reduce the failing input to the simplest failing case and then report the simplified failure case.
This would make property testing a lot more accessible to people than what currently exists in the ecosystem.

5 Likes

I created a swift.org account just so I could come here and say… YAY! YAY! :star_struck:

Items on my wishlist you've already tackled:

  • @Test annotation for discovery and control :white_check_mark:
  • natural language description of test :white_check_mark:
  • parameterized tests! :white_check_mark:

A fulfilled wish I didn't know I had:

  • Avoid early instantiation of all test cases at once, which creates so much confusion for Swift devs around test property lifetime. Instantiate for single test case execution instead :white_check_mark:

Here are my further wishes…

Naming

  • I'd like to avoid creating a function name. Having to name foodAvailable is cognitive overhead when we've just provided the name "The Food Truck has enough burritos".

Assertions

  • A way to provide custom #expect statements. That is, I want to write helper assertions.
  • In addition to x == y where the order doesn't matter, some way to write assertions that clearly identifies expected value vs. actual value.
  • I love composable matchers (Hamcrest) for their power. But I think most folks prefer the AssertJ-style "fluent" matchers for left-to-right reading.

Test Runner

  • Have the test runner remember the last failing test suites (and the failing tests within those suites) and run them first. This is really helpful for faster feedback when running all tests.
  • Similarly, keep track of test times. After running any previously failing tests, run the fastest test cases (and suites).
  • Randomized XCTest order never got the ability to re-run with a specified seed. Would love to see, "Gosh, this randomized test run failed. It says it used seed BLAH. Here, let's re-run with seed BLAH to reproduce."
29 Likes