Benchmark package initial release

hassila · March 15, 2023, 1:32pm

Should be there in 0.9.0, thanks.

hassila · March 15, 2023, 4:29pm

And a sample how a larger benchmark suite could look like:

github.com/apple/swift-nio

Moving NIOPerformanceTester to use package-benchmark for additional metrics

apple:main ← ordo-one:03-15-Benchmark_support

opened 07:46AM - 16 Mar 23 UTC

hassila

+4241 -0

Adding support for package-benchmark ### Motivation: It'd be good to get a… wider range of metrics for some performance tests (e.g. UDP/TCP could provide context switches, syscalls etc) and also possibly clean up the integration testing of malloc stuff to be part of this as some of those tests seems duplicated. This is a first stab at using [package-benchmark](https://swiftpackageindex.com/ordo-one/package-benchmark/main/documentation/benchmark) to get feedback from the NIO team. To get fully comparable numbers, one might want to tweak the iterations (package-benchmark uses 1K or 1M as typical iteration counts, so 100K was rounded up to 1M for these tests). ### Modifications: Added a subdirectory with a separate project (as package-benchmark depends on jemalloc) to not add any dependencies to the main swift-nio repo Basically the code from NIOPerformanceTester is copied over and made into a library target that is used by the benchmarks, major changes is making things public so the benchmarks can see them. `main.swift` has been renamed `Utilities.swift` and just contains the supporting setup that was top level before, all the benchmarks have been migrated to the actual benchmark executable targets `Benchmark.swift` has been reduced to just define the protocols implemented by the NIO benchmarks (which are still in use to minimise the changes to the benchmarks themselves) The benchmarks classes have been made public to be accessible by the benchmark executable targets. The actual benchmarks are largely untouched, do a diff vs the NIOPerformanceTester target directory and the original sources for overview (I didn't want to move them, as this PR could be landed first and then the legacy NIOPerformanceTester tests could be removed in a separate PR). ### Result: Wider range of benchmark metrics, richer set of PR threshold checks (usable for e.g. liburing work to see syscalls/context switches etc), better benchmark result analytics with comparisons possible between different NIO releases, less code. Possibility to kill the integration tests with malloc checks to something that will continuously will be updated (and possibly remove duplicated code there, as at a quick glance some of those tests are dups it seems?). ### Tasks - [ ] Get feedback from NIO team on quality of measurements / setup - [ ] Define acceptable thresholds for various metrics that can be compared with `check` (these can basically replace the old malloc counters in the integration test suite, not sure those tests are needed anymore, they at a quick glance looks like duplicates of the NIOPerformanceTester ones?) - [ ] Integration with NIO build infrastructure (would need help with that) - [sample GitHub workflow action](https://raw.githubusercontent.com/ordo-one/package-benchmark/main/.github/workflows/benchmark-pr.yml) - [ ] Try running with different NIO versions (would easiest be done by changing this package manifest to whatever version you want to test) - [ ] Showcase output with JMH, Delta comparisons and checks versus different versions / PR:s - [ ] Use wider range of metrics as required (started with e.g. UDPandTCP) - [ ] Review / discussion of any minor benchmark changes done (fundamentally iteration count setup)

hassila · March 23, 2023, 8:30am