Recommended way to measure time in Swift?

[Jens]

I've never seen a clear answer, despite numerous related blog posts and conversations.

Here's a recent one (that can continue in this thread).

@Nevin

I’m going off-topic here, and I’m not an expert on benchmarking, but I have to wonder if QuartzCore.CABase.CACurrentMediaTime is really the recommended way to measure time. I don’t see any alternative in the standard library, but Foundation does have a clock() function.

Is there some reason not to use Foundation.clock() ?

@scanon

FWIW, my default choice for benchmarking on Darwin is mach_absolute_time() . It's slightly quirky in that you have to query mach_timebase_info(...) to convert the result to ns, but a lovely lightweight thing otherwise.

@Jon_Shier

Benchmarks I’ve seen usually show CACurrentMediaTime as the fastest option that doesn’t require conversion like mach_absolute_time . I usually use CFAbsoluteTimeGetCurrent() since it doesn’t require a separate import.

@Jens

I've "investigated" this a number of times and landed in CACurrentMediaTime which if I remember correctly is essentially a wrapper around mach_absolute_time() , doing the conversion from CPU ticks to seconds for you, and thereby being some fraction of a microsecond slower than mach_absolute_time() . But I'm happy to change my habit if there's a problem with it. I doubt that the efficiency and/or accuracy of the timing function would make much difference in this particular case though. According to my experience, optimization glitches depending on seemingly unrelated context in various crazy ways is still much more of an issue in this type of benchmarking.

@Nevin

Oh I wasn’t doubting the accuracy, just questioning the verbosity and discoverability. If Foundation.clock() isn’t good enough, then Swift should either fix it or add something that is good enough with an equally appealing spelling.

@benrimmington

Possible alternatives to mach_absolute_time() :

• clock_gettime_nsec_np(CLOCK_UPTIME_RAW) — a non-portable Darwin extension in <time.h>
• DispatchTime.now().uptimeNanoseconds
• ProcessInfo.processInfo.systemUptime — except the implementation for Windows uses a lower-resolution clock?

@benrimmington

I think Foundation.clock() is re-exported from the standard C library.

It measures processor time, used by the calling process, in microseconds.

import Darwin

clock()

clock_gettime_nsec_np(CLOCK_PROCESS_CPUTIME_ID) / 1000

@Jean-Daniel

IHMO, you should never use system clock for benchmark, and always use a monotonic clock.

System time has no guarantee to be monotonic, and is usually less accurate and far slower to query.

Which means that anything that rely on system clock should be ban from benchmarking (ie: CFAbsoluteTimeGetCurrent ).

@Jon_Shier

Few benchmarks need to worry about that. Any jumps in the system clock should be both obvious and smoothed by repeated runs. In any case, I wasn’t recommending a time source for highly sensitive benchmarks, just a convenient one that’s performant.

It mentions the following alternatives:

• CACurrentMediaTime()

• Foundation.clock()

• mach_absolute_time() together with mach_timebase_info(…)

• CFAbsoluteTimeGetCurrent()

• clock_gettime_nsec_np(CLOCK_UPTIME_RAW)

• DispatchTime.now().uptimeNanoseconds

• ProcessInfo.processInfo.systemUptime

---

Is there a way that is accurate, discoverable, non-verbose and works reliably across most common platforms without specific imports etc?

If not, should/could one be added to the Standard Library? If so, how?

---

---

Edit: The accepted answer (as of 2020-02-05) is:

• ProcessInfo.processInfo.systemUptime

It is used in XCtest and it …

… ultimately calls down to mach_absolute_time on Darwin, and clock_gettime with   CLOCK_MONOTONIC on Linux.

1. Imprecision w.r.t. conversion to Double is a non-issue for this level of API.
2. Using mach_absolute_time is the right answer on OS X, via a wrapper in Foundation seems reasonable to me.

(source)

And also:

@benrimmington

I've opened SR-12124 to improve ProcessInfo.processInfo.systemUptime for BSD and Windows.

But I think DispatchTime.now().uptimeNanoseconds is the only cross-platform API (from the original list) which seemingly guarantees "nanosecond precision".

[benrimmington]

macOS 10.15 has new XCTest APIs for measuring time and other performance metrics.

It makes the test method from your Improving Float.random(in:using:) post a bit simpler.

import RandomFloat
import XCTest

final class RandomFloatTests: XCTestCase {

  override class var defaultMetrics: [XCTMetric] {
    [
      XCTClockMetric(),
      XCTCPUMetric(limitingToCurrentThread: true),
      XCTMemoryMetric(),
      XCTStorageMetric(),
    ]
  }

  func test() {
    let floatsPerRun = 10_000_000
    var meanSum = 0.0
    var numRuns = 0
    var prng = WyRand()

    measure(metrics: Self.defaultMetrics) {
      var sum = 0.0
      for _ in 0 ..< floatsPerRun {
        let v = pseudorandomFloatInClosedUnitRange(using: &prng)
        sum += Double(v)
      }
      meanSum += sum
      numRuns += 1
    }

    print("Total mean float value:",
          meanSum / Double(floatsPerRun * numRuns))
  }
}

[Jens]

It depends on what you mean by simple. I'd say using XCTest APIs is not as simple as:

1. Write/download/copy-paste a single source file (command line) program
2. Compile (from command line according to instructions within the file)
3. Run

And XCTest is macOS (and Xcode?) only.

---

Also, according to my experience, testing for performance will almost always mean bumping into various context dependent optimizer glitches etc, and therefore it's important to be able to measure and profile some piece of code in many different contexts. XCTest forces your code into a specific context and is a lot of boilerplate.

@Jens

Note also that these issues are impossible to represent/test/benchmark in eg XCTest and The Swift Benchmarking Suite. Why? Because the issues are all about context , and they are interacting in various funny ways as shown by eg the string-interpolation-trick working in reverse or not depending on whether base was declared in top-level or not. XCTest and The Swift Benchmarking Suite forces some layer(s) of context on the tested code, and thus their harness becomes part of the tested code.

[xwu (Xiaodi Wu)]

@Jens

And XCTest is macOS (and Xcode?) only.

GitHub - apple/swift-corelibs-xctest: The XCTest Project, A Swift core...

[Jens]

XCTest seems to use
Foundation.ProcessInfo.processInfo.systemUptime
to measure time.

[benrimmington]

The choice of ProcessInfo.processInfo.systemUptime is discussed in apple/swift-corelibs-xctest#109 (search for mach_absolute_time in the conversation).

For a single source benchmark, you could still have a separate measure function, to contain the boilerplate code.

You could also put assert(false, "Compile with optimizations") at the top of the file, if you wanted to stop accidental -Onone results.

[Jens]

I quote the relevant part here:

I am no expert in benchmarking, so I am not especially confident in the optimal mechanism for measuring the passage of time for this purpose, however after some cursory research, I settled on using NSProcessInfo 's systemUptime property, which ultimately calls down to mach_absolute_time on Darwin, and clock_gettime with CLOCK_MONOTONIC on Linux. As far as I can tell, these are the appropriate primitives to use for this purpose in these environments. One potential source of error is introduced in that the time values are being converted to Double , but I'm unsure of the practical impact of that.

[benrimmington]

And the reply from Daniel Dunbar:

The time measurement parts sound reasonable to me for an initial implementation:

1. Imprecision w.r.t. conversion to Double is a non-issue for this level of API.
2. Using mach_absolute_time is the right answer on OS X, via a wrapper in Foundation seems reasonable to me.

[Jens]

So it looks like
ProcessInfo.processInfo.systemUptime
is the answer to the question of this thread then, thanks!

[mattrips]

I'm assuming that this is the answer, because (1) ProcessInfo is part of the Foundation framework, and thus is the solution closest to being fully available on all platforms, and (2) its use by the XCTest framework seems to validate its reliability for the benchmarking use case.

[AlexanderM (Alexander Momchilov)]

Aren't tests run in debug mode, with minimal optimizations? How would you ensure you're measuring "real" (prod-build) performance using these APIs?

[scanon (Steve Canon)]

Tests are run in whatever mode you tell the build system to run the tests. It’s certainly possible to write tests that are intended to be run in release mode.

[benrimmington]

@AlexanderM You can use swift test --configuration release on the command line; or edit the scheme in Xcode, so the Test action uses the Release build configuration.

And as I mentioned earlier, you can add an assert(false, "Compile with optimizations") statement to enforce this.

[AlexanderM (Alexander Momchilov)]

@benrimmington

And as I mentioned earlier, you can add an assert(false, "Compile with optimizations") statement to enforce this.

Ooooo that's clever!

[Jean-Daniel]

CFAbsoluteTimeGetCurrent() don't give any guarantee about its resolution (and don't use an high resolution clock under the hood AFAIK).

[Jens]

See edit of OP, I added the (currently) accepted answer, which is not CFAbsoluteTimeGetCurrent().

[benrimmington]

@Jean-Daniel

CFAbsoluteTimeGetCurrent() don't give any guarantee about its resolution (and don't use an high resolution clock under the hood AFAIK).

The documentation doesn't mention resolution. The implementation uses:

• the GetSystemTime function on Windows, where the SYSTEMTIME structure has a resolution of 1 millisecond.

• the "obsolescent" gettimeofday function on all other platforms, where the timeval structure has a resolution of 1 microsecond.

---

On macOS:

import Darwin

// Resolution of `gettimeofday`:
var rt = timespec()
clock_getres(CLOCK_REALTIME, &rt)
print(rt) //> timespec(tv_sec: 0, tv_nsec: 1000)

// Resolution of `mach_absolute_time`:
var ut = timespec()
clock_getres(CLOCK_UPTIME_RAW, &ut)
print(ut) //> timespec(tv_sec: 0, tv_nsec: 1)

---

@Jens

See edit of OP, I added the (currently) accepted answer, which is not CFAbsoluteTimeGetCurrent().

I've opened SR-12124 to improve ProcessInfo.processInfo.systemUptime for BSD and Windows.

But I think DispatchTime.now().uptimeNanoseconds is the only cross-platform API (from the original list) which seemingly guarantees "nanosecond precision".