What's Changed
Adding support for new thresholds subcommands - used for storing/reading/checking against static saved thresholds - by default output into Thresholds in the current directory, but location can be specified with --path too.
This fundamentally replaces the old options check-absolute and check-absolute-path as well as the explicit export using metricP90AbsoluteThresholds format with new swift package benchmark thresholds commands.
To clarify, the existing thresholds in the benchmark code configuration is really threshold tolerances - unfortunately difficult to rename now, but basically the new thresholds commands operate on static thresholds, while the code defined tolerances continue to be applied to both dynamic and static benchmark checking as outlined in the documentation.
New thresholds support
Update static thresholds (in the Thresholds/ directory)
swift package --allow-writing-to-package-directory benchmark thresholds update
Build of product 'BenchmarkTool' complete! (2.44s)
Build complete!
Building BenchmarkTool in release mode...
Building benchmark targets in release mode for benchmark run...
Building URLBenchmarks
Building StringBenchmarks
Building PredicateBenchmarks
Building JSONBenchmarks
Building InternationalizationBenchmarks
Building FormattingBenchmarks
Building EssentialsBenchmarks
Building DataIOBenchmarks
Building Base64Benchmarks
Building AttributedStringBenchmarks
==================
Running Benchmarks
==================
100% [------------------------------------------------------------] ETA: 00:00:00 | AttributedStringBenchmarks:createLongString-swift
100% [------------------------------------------------------------] ETA: 00:00:00 | AttributedStringBenchmarks:createManyAttributesString-swift
100% [------------------------------------------------------------] ETA: 00:00:00 | AttributedStringBenchmarks:enumerateAttributes-swift
100% [------------------------------------------------------------] ETA: 00:00:00 | AttributedStringBenchmarks:enumerateAttributesSlice-swift
....
100% [------------------------------------------------------------] ETA: 00:00:00 | URLBenchmarks:URLComponents-SetEncodableComponents
100% [------------------------------------------------------------] ETA: 00:00:00 | URLBenchmarks:URLComponents-SetEncodableQueryItems
100% [------------------------------------------------------------] ETA: 00:00:00 | URLBenchmarks:URLComponents-SetQueryItems
==================
Writing Thresholds
==================
Writing to ./Thresholds/PredicateBenchmarks.predicateKeypathComputedPropertyCondition.p90.json
Writing to ./Thresholds/PredicateBenchmarks.predicateTrivialCondition.p90.json
Writing to ./Thresholds/StringBenchmarks.read-utf8.p90.json
...
Reading thresholds
package benchmark thresholds read
...
Build complete!
Building BenchmarkTool in release mode...
Building benchmark targets in release mode for benchmark run...
Building URLBenchmarks
Building StringBenchmarks
Building PredicateBenchmarks
Building JSONBenchmarks
Building InternationalizationBenchmarks
Building FormattingBenchmarks
Building EssentialsBenchmarks
Building DataIOBenchmarks
Building Base64Benchmarks
Building AttributedStringBenchmarks
Reading thresholds from "Thresholds"
utf16-decode
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโ
โ Metric โ Threshold .p90 โ Allowed % โ Allowed ฮ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโก
โ Time (total CPU) โ 3208473041 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโค
โ Time (wall clock) โ 3211281125 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโ
Canada-encodeToJSON
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโ
โ Metric โ Threshold .p90 โ Allowed % โ Allowed ฮ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโก
โ Time (total CPU) โ 62029823 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโค
โ Throughput (# / s) โ 17 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโ
utf32-decode
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโ
โ Metric โ Threshold .p90 โ Allowed % โ Allowed ฮ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโก
โ Time (wall clock) โ 3341120667 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโค
โ Time (total CPU) โ 3341006500 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโ
allocationsForAutoupdatingCurrentCalendar
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโ
โ Metric โ Threshold .p90 โ Allowed % โ Allowed ฮ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโก
โ Malloc (total) โ 4000000 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโค
โ Time (total CPU) โ 1360681292 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโค
โ Memory (resident peak) โ 12009472 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโค
โ Throughput (# / s) โ 1 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโ
getAttributeSubrange-swift
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโ
โ Metric โ Threshold .p90 โ Allowed % โ Allowed ฮ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโก
โ Throughput (# / s) โ 1479 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโค
โ Time (wall clock) โ 730111 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโค
โ Time (total CPU) โ 728575 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโ
write-regularFile
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโโโโโโ
โ Metric โ Threshold .p90 โ Allowed % โ Allowed ฮ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโโโโโโก
โ Time (total CPU) โ 2656255 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโค
โ Syscalls (total) โ 5 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโค
โ Malloc (total) โ 10 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโค
โ Throughput (# / s) โ 291 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโค
โ Time (wall clock) โ 3737599 โ 0.0 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโโโโโโ
Check static thresholds
swift package benchmark thresholds check --filter base64-decode-1MB-fromString-noOptions
...
Build of product 'BenchmarkTool' complete! (2.45s)
Build complete!
Building BenchmarkTool in release mode...
Building benchmark targets in release mode for benchmark run...
Building URLBenchmarks
Building StringBenchmarks
Building PredicateBenchmarks
Building JSONBenchmarks
Building InternationalizationBenchmarks
Building FormattingBenchmarks
Building EssentialsBenchmarks
Building DataIOBenchmarks
Building Base64Benchmarks
Building AttributedStringBenchmarks
==================
Running Benchmarks
==================
100% [------------------------------------------------------------] ETA: 00:00:00 | Base64Benchmarks:base64-decode-1MB-fromString-noOptions
Reading thresholds from "Thresholds"
Checking ["Base64Benchmarks:base64-decode-1MB-fromString-noOptions"]
===========================================================================================
Deviations worse than threshold for Base64Benchmarks:base64-decode-1MB-fromString-noOptions
===========================================================================================
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโ
โ Throughput (# / s) (#, ฮ) โ p90 threshold โ Current_run โ Difference ฮ โ Threshold ฮ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโก
โ p90 โ 332 โ 322 โ 10 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโ
โ Throughput (# / s) (#, %) โ p90 threshold โ Current_run โ Difference % โ Threshold % โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโก
โ p90 โ 332 โ 322 โ 3 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโ
============================================================================================
Deviations better than threshold for Base64Benchmarks:base64-decode-1MB-fromString-noOptions
============================================================================================
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโ
โ Time (total CPU) (ฮผs, ฮ) โ p90 threshold โ Current_run โ Difference ฮ โ Threshold ฮ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโก
โ p90 โ 3217 โ 3111 โ -106 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโคโโโโโโโโโโโโโโโโโโ
โ Time (total CPU) (ฮผs, %) โ p90 threshold โ Current_run โ Difference % โ Threshold % โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโชโโโโโโโโโโโโโโโโโโก
โ p90 โ 3217 โ 3111 โ -3 โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโงโโโโโโโโโโโโโโโโโโ
The baseline 'Current_run' is WORSE than the defined thresholds.
error: benchmarkThresholdRegression