Controllable clock support in `swift-testing`

Development Swift Testing
1

Hello,

I’m curious if there’s interest in adding a controllable or “test” clock to swift-testing. I’ve found in my own projects that being able to manipulate and advance time deterministically during unit tests can be incredibly helpful, especially for testing features like retries, delays, and timeouts.

Here’s a rough sketch of the kind of interface I have in mind:

struct TestClock<Duration>: Clock {
  func sleep(until deadline: Instant, tolerance: Instant.Duration?) async throws
  func advance(to instant: Instant)
  func advance(by duration Duration)
}

The idea would be to provide a clock implementation where time only moves forward when you explicitly tell it to, which enables precise and reliable testing of time-based behavior.

I’ve seen similar utilities (or actually a clock with almost the same interface and functionality) in Pointfree‘s swift-clocks GitHub - pointfreeco/swift-clocks: ⏰ A few clocks that make working with Swift concurrency more testable and more versatile., but introducing a full dependency solely for test purposes can feel a bit heavy, especially when the functionality needed is quite (test) focused.

Would others find this useful in swift-testing? I’d love to hear if anyone else has run into the same need or has thoughts on design or scope.

Thanks.

9 Likes
Mocked-Clock implementation for public use
2

This has come up a few times in the past. We're open to it, but it hasn't been a priority for us to implement. If you'd like to design an API and draft a Swift Evolution proposal, the testing workgroup would be happy to review it.

3 Likes
3

On this note, I think it would be very useful if Swift Package Manager would permit a package to declare a dependency which is used only by its test targets, and for such a dependency to be ignored by other downstream packages that depend on it. In other words, "test-only" dependencies.

Luckily, test targets are already ignored by dependent packages today. Adding support for test-only dependencies would allow you to use existing libraries like swift-clocks in your tests without impacting your clients.

3 Likes
4

This already works, your clients will only see dependencies of any products you vend. Dependencies of test targets or any other targets that aren't exposed to clients are automatically excluded.

4 Likes
5

That's great to hear! Thanks for confirming.

I may be conflating that with a similar but different feature which IIRC is missing: the ability to express that a particular compiler setting should only be applied "for development", i.e. when a package is at the root, and not being used as a dependency. If that's still not supported, I think it'd be great to add that. But sorry if I mixed these up.

But @ph1ps, it sounds like you could use swift-clocks without impacting clients? In case that was your primary concern. (Not to dispute that this could potentially be a useful addition to Swift Testing, too.)

6

Right, maybe my motivation is weaker than I thought, after all.

Just to clarify, I am not entirely against using dependencies in this way. I just had the urge to ask if something like this could be streamlined because I was in need of such a clock quite often when writing tests, so it felt natural to have it already bundled with swift-testing.

Maybe Pointfree also has opinions about this, I certainly don‘t want to end up stealing their ideas.

1 Like
7

We've expressed that we'd love built-in support for test clocks, and would happily retire our package when the time comes!

4 Likes
8

And we want to encourage and grow the Swift ecosystem! Whatever shall we do!?

2 Likes
9

I didn't realize this post existed so I created my own post basically asking for a testing clock:

So yeah. Basically a +1 to this post.

1 Like
10

Since @MahdiBM posted about something similar recently, I felt the urge to explain why this hasn't made any progress.

It was pointed out to me, and I later came to the same conclusion, that a controllable clock would not be very helpful with the current state of Concurrency. I don't want to go into much detail, because there is already quite a lengthy discussion about something similar here: Reliably testing code that adopts Swift Concurrency?.

Essentially, even seemingly very simple code can't reliably be tested:

@MainActor class Model {
  
  var counter = 0
  let clock: any Clock<Duration>
  
  init(clock: any Clock<Duration>) {
    self.clock = clock
  }
  
  func doSomething() async throws {
    for _ in 0..<3 {
      try await clock.sleep(for: .seconds(1))
      counter += 1
    }
  }
}

Assuming this is the test for this function:

@Test @MainActor func test() async {
  let clock = TestClock<Duration>()
  let model = Model(clock: clock)
  let task = Task {
    await model.doSomething()
  }
  assert(model.counter == 0)
  clock.advance(by: .seconds(1))
  assert(model.counter == 1)
  clock.advance(by: .seconds(1))
  assert(model.counter == 2)
  clock.advance(by: .seconds(1))
  assert(model.counter == 3)
  await task.value
}

(advance(by:) is a function that will advance the test clock by the given duration, emulating time ticking)

There are multiple problems about this:
You don't know when the code inside the Task will be scheduled. By the time you call .advance the first time, you need the sleep inside the model to already have calculated its deadline. If advance is called before sleep, the clock ticks and sleep will calculate its deadline with the already ticked clock.

You can somewhat mitigate this with the new Task.immediate.

However, this brings me to another problem. Clocks functions sleep(for:) and sleep(until:) are not nonisolated(nonsending), which means there is a suspension point at sleep(for:), which means Task.immediate will end its immediacy right before the deadline would be calculated, coming to the same result as before.

Pointfree's version of this clock works around these problems, by using their megaYield function. But even if you could work around this, it still feels like there is a missing piece in this puzzle. Something that would make it easier to control how things are scheduled. Until then, I can't seem to figure out how to make use of a TestClock reliably. Maybe I am also missing something, but I could not find a way to make this work, yet.

3 Likes
11

All of that is true, yet PointFree's version is usually reliable right away, with some tests needing some megaYield tuning in CI. So I'd rather have API that is 99% testable rather than 0% testable. But you're right, we fundamentally need core concurrency testing features, like the ability to wait for the runtime to finish all work, or to add probes or barriers to know when certain sets of work have completed. Really, any testing support would be good, as right now there's nothing at all.

(I believe the sleep nonisolated(nonsending) issue was recently discovered and discussed here, so perhaps we'll see some changes.)

3 Likes
12

I do believe the tools have come a long way since we posted that thread, as you point out with Task.immediate, and when it comes to executors, task executors, and nonisolated(nonsending), but it is worth maybe a separate discussion/pitch to see if some of these APIs should change to be nonisolated(nonsending) and avoid those suspension points.

Even so, by embracing more recent concurrency tooling, our Task.megaYield() can often be swapped out for a plain ole Task.yield() these days.

1 Like