Best practice to test generating values then test each value

Development Swift Testing
1

I have a number of XCTests that have two phases:

  1. The first phase tests code that generates an array of values
  2. The second phase ensures each value equals an expected value

Something like this:

func testSomething() throws {

    // 1. Test code that generates an array of values
    let values = try generateValues()
    XCTAssertEqual(values.count, expectedCount)

   // 2. Test that each value matches expected values
   for (value, expectedValue) in zip(values, expectedValues) {
       XCTAssertEqual(value, expectedValue)
   } 
}

With Swift Testing, I would prefer to have Phase 2 be a parameterized test (which look very nice!) which would mean breaking each phase out into a separate test.

Is there a best practice for having the results of one test (Phase 1 above) be used as the arguments for a second test (Phase 2 above)?

For example, would a serialized sub suite struct with two tests and a static variable to hold the generated arguments be a valid or supported approach?

2

No, and in fact I call this out specifically as dangerous in Swift 6 in Go Further.

What you can do however is use a static let to construct your inputs. Something like:

static let oddNumbers = [1, 3, 5, 7, /*...*/]

@Test(arguments: oddNumbers) func f(i: Int) {
  // ...
}

Your real-world computation is of course going to be more complex.

3

Yes, the difficulty is that I need to test the real-world computation as well.

I think what I need is something conceptually like a @Subtest, which would be a function able to be called from within a test and able to take arguments, but wouldn't be called as a test on its own.

For now, I'll just forgo using arguments and keep the loop within my test.

Thank you for your response and for the work on Swift Testing and the session.

I've already migrated a number of XCTests and it has been a very good experience so far.

4

Can you elaborate on why, in the original XCTest example, you first used XCTAssertEqual(values.count, expectedCount) before the for...in loop? Was it to protect against the possibility of values and expectedValues being of different lengths, and then zip(...) only using the shorter of the two lengths?

If that's the reason, then I'm not sure that concern would still be relevant if you adapted this pattern to a parameterized Swift Testing test. Using a parameterized test, I would recommend that you pass a Collection of 2-tuples, each containing a actual value and an expected value. For example:

func generateValues() throws -> [(value: Value, expectedValue: Value)] { ... }

@Test(arguments: try generateValues())
func example(value: Value, expectedValue: Value) {
  #expect(value == expectedValue)
}

With this structure it's not possible for there to be a differing number of actual values and expected values, so it's not something you need to explicitly guard against, and you only need a single @Test function.

If there was some other reason for the original pattern, or if there's more nuance in your real code that the minimal example doesn't capture, I'd be interested to understand the rationale behind pattern you were using better.

5

The reason is because the value generation is functionality of the code I am testing, it isn't a test fixture.

So, if the code I am testing generates a different number of values than I am expecting, the test needs to fail.

Yes, I may have made my initial example too minimal.

I am testing code that, given a set of data, generates a report.

The code that generates the report is part of the module I am testing, not a test fixture.

So, first, if generating the report throws an error, I want the test to fail and stop.

Next, the report has various properties, some properties are single values, some are arrays of values, and there may be one or more computed properties.

Currently I check that the non-array properties (stored and computed) equal their expected results.

This is also where I ensure the count of each array of values that was generated matches the expected count.

Then I iterate through each of the arrays of generated values in a for loop checking that each value matches its expected value. Each array is handled in its own loop.

So, if the report has two properties that are arrays of values, there are two loops, one per property.

Handling these separately, as opposed to comparing the entire report to an expected report value, has given me fairly good granularity in easily seeing which values in the report failed.

In Swift Testing, I can continue to do what I have been doing in XCTest, it works fine.

But, with the emphasis put on using arguments and parameterized testing instead of looping through values in the test itself, it made me wonder if there is any way to take advantage of its benefits for this use case. (Especially interesting was being able to run the test on only a single iteration of the loop)

And of course, I am asking these questions literally on day one of me using Swift Testing after watching two WWDC videos about it (which I thought were excellent, btw). So my understanding of Swift Testing is not very deep.

@smontgomery I really appreciate you gathering feedback and all the work done on Swift Testing. So far it has been frictionless to begin migrating XCTests and at first look it seems like it manages to be both very approachable and straightforward, with a small number of building blocks, but still very flexible and powerful.

6

On additional question.

Does this need to be top-level code?

When I try to do something similar with both functions being members of a struct test suite I get the error:

Instance member 'generateValues' cannot be used on type 'ExampleTestSuite'; did you mean to use a value of this type instead?

7

Supposedly, example was for top-level code. In case of suits, you can make static func generateValues().

1 Like
8

Thank you @vns that works exactly as expected.