Benchmark stable release

Hi,

We're happy to announce that Benchmark now has shipped with what we believe is an API stable interface (which follows semantic versioning) with significantly improved capabilities compared to the initial release last fall (several thanks to the new underpinning Histogram foundation) .

Shiny new documentation is available outlining both setup and how to write benchmarks, largely thanks to @Joseph_Heck and the DocC team (and SPI for hosting!).

There are also some sample code on GitHub.

For those who have existing benchmarks there are a few (mostly search-and-replace) changes to the API between 0.9.0 and 1.2.0 which is the latest release - please see the release notes for migration notes (or check the changes to the sample code project above).

Thanks to the use of a built tool plugin, the boilerplate needed is now trivial:

import Benchmark

let benchmarks = {
    Benchmark("MyBenchmark") { benchmark in
      // Something to measure here
    }

    Benchmark("MyOtherBenchmark") { benchmark in
      // Something to measure here
    }
}

New is also the ability to run benchmark on a platform that does not have jemalloc available by setting up an environment variable that disables the jemalloc requirement:

> BENCHMARK_DISABLE_JEMALLOC=true swift package benchmark

Each benchmark is now run completely isolated so resident memory statistics etc should be consistent across runs now.

It's also possible to use --filter, --skip, --target and --skip-target with Regex to choose what benchmarks should be run.

Multiple output formats is now supported, including raw percentile data as well as data suitable for visualisation with online tools such as JMH and HDR Histogram plots or even piping output to e.g. Youplot.



There's also the support for grouping per metric or delta comparisons:



Also, there is extensive support for automation of benchmarks through CI, where either two branches (e.g. main and PR) can be compared with custom deviation thresholds configured (absolute and/or relative) per metric supported by Benchmark.


Additionally, it's now also possible to check benchmarks results against an absolute set of values (useful if the number of permutations of toolchain/OS/repo is big, to reduce the build matrix) - although for most projects automated checks of PR vs main would be recommended.

Many thanks to the people who've provided feedback and please file issues / PR:s (or DM me here if you want offline feedback).

Cheers,

Joakim

13 Likes

We just added a significant new feature with 1.3.0 - the ability to track ARC traffic (thanks to those who helped make it happen!).

These metrics are very useful for a few interesting use cases:

  • Zero ARC traffic regressions benchmark checks (code that never should emit any ARC traffic can easily be tested by CI now, either for zero ARC or for expected X...)
  • Optimizations of ARC traffic (quick iterations without having to run instruments, also on Linux)
  • Check that retain/release delta is zero (for most benchmarks that would be the expected result, if you have a retain cycle it will quickly run up, so good to check with CI too)

We find this a great complement to the memory allocation stats also in place, hope it'll be useful!

3 Likes

Thanks a lot for adding the ARC tracking @hassila, this is looking very good :partying_face:

I'm actually wondering if you'd like a category under Related Projects - Swift Forums for the project, rather than re-using the same "initial announcement" thread? I think benchmarking is an important topic and the project is shaping up very well, so I'd be happy to assist in getting that set up :slight_smile:

4 Likes

Would be super, thanks!

3 Likes

Already helped find one funny thing:

:slight_smile:

1 Like

Added a convenience benchmark creation command in the spirit of swift package init which will create the initial boilerplate and add the target for you (dependency still must be added manually)

So getting started is faster than ever:

  1. Add the dependency to package-benchmark
  2. Run swift package --allow-writing-to-package-directory benchmark init MyNewBenchmark
  3. Make any changes to the boilerplate so you measure the right stuff
  4. Run it with swift package benchmark
2 Likes