In the last two weeks I have noticed weird change in how build-script runs the benchmarks. When executed with following command line options:
swift-source $ ./swift/utils/build-script -R -B --no-assertions
…all parts of the build proceeds as usual, up until the point when its time to run the benchmarks. Then it stops showing incremental progress as the individual benchmarks get executed. Instead it appears to be stuck.
But the Activity Monitor on my 2 core CPU is always 50% busy, even though no process is listed as using that much CPU. The responsible process is somehow hidden from Activity Monitor. If I'm patient, eventually the benchmark run will finish and the logs are printed.
[1537/1537] Running utility command for swift-benchmark-macosx-x86_64
--- Running tests for swift ---
--- check-swift-benchmark-macosx-x86_64 ---
+ /usr/local/bin/cmake --build /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64 -- -j2 check-swift-benchmark-macosx-x86_64
[1/1] cd /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/...ipt /Users/mondo/Developer/swift-source/swift/benchmark/scripts/compare_perf_tests.p
# TEST SAMPLES MIN(μs) Q1(μs) MEDIAN(μs) Q3(μs) MAX(μs) MAX_RSS(B)
1 Ackermann 57 774 774 774 774 780 45056
2 AngryPhonebook 56 6608 6615 6615 6643 7077 53248
3 AnyHashableWithAClass 57 403500 408000 409500 423000 429500 73728
4 Array2D 58 15536 15568 15584 15616 15776 552960
5 ArrayAppend 56 2000 2000 2000 2050 2220 831488
This looks like quite a significant regression in usability of the build-script! I have killed the process several times before I decided to let run overnight to see if it will finish eventually. For comparison, here is the relevant part from an older run, where the second line starting with cd /Users… shows the full command that gets executed. This info is now missing in the current version of the build-script.
[1613/1613] Running utility command for swift-benchmark-macosx-x86_64
--- Running tests for swift ---
--- check-swift-benchmark-macosx-x86_64 ---
+ /usr/local/bin/cmake --build /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64 -- -j2 check-swift-benchmark-macosx-x86_64
[1/1] cd /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-...ers/mondo/Developer/swift-source/swift/benchmark/scripts/compare_perf_tests.p
FAILED: benchmark/CMakeFiles/check-swift-benchmark-macosx-x86_64
cd /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver run -o O --output-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --independent-samples 3 && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver run -o Onone --output-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --independent-samples 3 && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver compare --log-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --compare-script /Users/mondo/Developer/swift-source/swift/benchmark/scripts/compare_perf_tests.py
# TEST SAMPLES MIN(μs) Q1(μs) MEDIAN(μs) Q3(μs) MAX(μs) MAX_RSS(B)
2 AngryPhonebook 59 6592 6643 6727 6958 7829 36864
3 AnyHashableWithAClass 55 393500 396500 397500 399500 407500 53248
4 Array2D 57 17106 17679 17930 18350 23416 8871936
5 ArrayAppend 56 2000 2000 2000 2010 2170 839680
I am not familiar with the implementation of this and during a cursory look at the recent edit history of build-script and build-script-impl I wasn't able to identify the source and author of the new behavior.
Could somebody please restore the original behavior of continuous benchmark report generation and non-hidden benchmarking process?
