Sometimes tests might get into a state (either due the test code itself or due to the code they're testing) where they don't make forward progress and hang.
Swift Testing provides a way to handle these issues using the TimeLimit trait:
Currently there exist multiple overloads for the .timeLimit trait: one that
takes a Swift.Duration which allows for arbitrary Duration values to be
passed, and one that takes a TimeLimitTrait.Duration which constrains the
minimum time limit as well as the increment to 1 minute.
I very much appreciate the flexibility this provides - while I expect that I won't use it very often on unit tests, I often abuse the existing/built-in testing frameworks to do functional/integration testing, where this level of control would be extremely welcome for the outlier scenario or situation.
As mentioned in the thread, the usage of coarse-grained timeout values is not frequently needed for fast-running unit tests, but can be a very useful backstop for when things go badly wrong especially during integration testing, and being able to declare the expected maximum duration on a per-test/suite basis via traits provides useful flexibility.
Feedback in this thread has been light but positive, and the proposal is accepted.
I don't know if anyone uses different timeout values in a test. While I do not and I would not miss it, I figured it was worth calling it out just in case some people do expect that functionality. Also, having the timeout closer to the particular lines of long running code might make it easier for unacquainted developers to identify the long running code.
Thanks for the feedback! This is something we've discussed previously (although I think only in the context of the confirmation() function?)
It is intentional that we don't provide timeouts on individual expectations or statements inside a test function. Our experience maintaining XCTest has shown us that these sorts of timeouts are pretty universally brittle. For example:
If you run a test in CI, it tends to run significantly slower than at your desk; or
If you run a test on your coworker's computer while they're also mining cryptocurrency and hosting six VMs in the background, it tends to run even worse.
As well, when tests run in parallel, they tend to individually run more slowly (although the overall execution time is typically lower than if they ran serially.) This can befoul any fine-grained timeouts in your tests.
Rather than using timeouts within tests, setting a timeout that applies to the test function as a whole lets you "smooth out" the performance bumps that may occur during a test. And performance tests (a future direction for Swift Testing) are generally a better abstraction than timeouts for ensuring performance-sensitive code stays fast because they can measure regressions in percentages rather than in clock time.
While I understand that small time limits can lead to flaky tests in different environments, having a granularity of 1 minute is very limiting and can be tedious specially when these kind of tests start failing due to a recent change.
A very common example would be a test asserting that a view model's @Published property is updated as expected after a specific function is called. The easiest way to do this is to await on the emitted values of the publisher for the expected one but since those are non-finishing publishers the test will hang when a bug is introduced unless we specify a timeLimit.
Having a minimum of 1 minute in those cases is a bit excessive specially if you like some TDD.
I'm wondering whether it would be better if an arbitrary limit can be passed and leave the responsibility to fix flaky tests to the users. This would allow users to create their own constants and those could have different values for each environment (local/CI) based on their capabilities.
I also have the same thought that something is missing for confirmation() functions not having a timeout parameter similar to XCTest expectations. In these scenarios XCTest can still give a faster failure.
Unlike XCTestExpectation and wait(for:)/fulfillment(of:), confirmation() does not block or suspend while waiting for its condition to be met. If the condition is not met before confirmation() returns, the test fails. The general expectation is that code being tested will use Swift concurrency mechanisms like task groups or continuations to control its own execution in a concurrency-safe way.
Let me try to explain a simple scenario and the problem I see with the current 1 minute (as minimum) timeout. Consider the following code:
class ViewModel: ObservableObject {
@Published var value: Int = 0
func doSomething() {
value = 2
}
}
struct TimeoutTests {
@Test(.timeLimit(.minutes(1)))
func test1() async {
let viewModel = ViewModel()
async let expectedValue = await viewModel.$value.values.first(where: { $0 == 2})
viewModel.doSomething()
#expect(await expectedValue == 2)
}
@Test(.timeLimit(.minutes(1)))
func test2() async {
let viewModel = ViewModel()
await confirmation() { confirmation in
viewModel.doSomething()
for await value in viewModel.$value.values {
if value == 2 {
confirmation()
break
}
}
}
}
}
These tests work fine until a bug is introduced by mistake. Imagine someone changes the 'doSomething' to return another value. When running the existing test suite it will take 1 additional minute for each similar test until the developer sees the test results. Being able to specify lower timeouts is highly needed I think. Otherwise I'd appreciate to see if there's another way to achieve this.
