1

Sometimes tests might get into a state (either due the test code itself or due to the code they're testing) where they don't make forward progress and hang.
Swift Testing provides a way to handle these issues using the TimeLimit trait:

@Test(.timeLimit(.minutes(60))
func testFunction() { ... }

Currently there exist multiple overloads for the .timeLimit trait: one that
takes a Swift.Duration which allows for arbitrary Duration values to be
passed, and one that takes a TimeLimitTrait.Duration which constrains the
minimum time limit as well as the increment to 1 minute.

Read the full proposal here.

5 Likes
2

I very much appreciate the flexibility this provides - while I expect that I won't use it very often on unit tests, I often abuse the existing/built-in testing frameworks to do functional/integration testing, where this level of control would be extremely welcome for the outlier scenario or situation.

2 Likes
3

Thank you, @Dennis!

As mentioned in the thread, the usage of coarse-grained timeout values is not frequently needed for fast-running unit tests, but can be a very useful backstop for when things go badly wrong especially during integration testing, and being able to declare the expected maximum duration on a per-test/suite basis via traits provides useful flexibility.

Feedback in this thread has been light but positive, and the proposal is accepted.

3 Likes
4

Oops, I missed this thread.

There is a regression between this and functionality offered by XCTest, namely, that each awaited expectation can have different timeouts.

I don't know if anyone uses different timeout values in a test. While I do not and I would not miss it, I figured it was worth calling it out just in case some people do expect that functionality. Also, having the timeout closer to the particular lines of long running code might make it easier for unacquainted developers to identify the long running code.

5

Thanks for the feedback! This is something we've discussed previously (although I think only in the context of the confirmation() function?)

It is intentional that we don't provide timeouts on individual expectations or statements inside a test function. Our experience maintaining XCTest has shown us that these sorts of timeouts are pretty universally brittle. For example:

  • If you run a test in CI, it tends to run significantly slower than at your desk; or
  • If you run a test on your coworker's computer while they're also mining cryptocurrency and hosting six VMs in the background, it tends to run even worse.

As well, when tests run in parallel, they tend to individually run more slowly (although the overall execution time is typically lower than if they ran serially.) This can befoul any fine-grained timeouts in your tests.

Rather than using timeouts within tests, setting a timeout that applies to the test function as a whole lets you "smooth out" the performance bumps that may occur during a test. And performance tests (a future direction for Swift Testing) are generally a better abstraction than timeouts for ensuring performance-sensitive code stays fast because they can measure regressions in percentages rather than in clock time.

2 Likes
6

Thanks for taking the time to craft a generous reply! I cannot fault the intention and my experience about brittleness matches yours.