Changes to how build-script runs benchmarks?

benchmarks
build-script

(Pavol Vaskovic) #1

In the last two weeks I have noticed weird change in how build-script runs the benchmarks. When executed with following command line options:

swift-source $ ./swift/utils/build-script -R -B --no-assertions

…all parts of the build proceeds as usual, up until the point when its time to run the benchmarks. Then it stops showing incremental progress as the individual benchmarks get executed. Instead it appears to be stuck.

But the Activity Monitor on my 2 core CPU is always 50% busy, even though no process is listed as using that much CPU. The responsible process is somehow hidden from Activity Monitor. If I'm patient, eventually the benchmark run will finish and the logs are printed.

[1537/1537] Running utility command for swift-benchmark-macosx-x86_64
--- Running tests for swift ---
--- check-swift-benchmark-macosx-x86_64 ---
+ /usr/local/bin/cmake --build /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64 -- -j2 check-swift-benchmark-macosx-x86_64
[1/1] cd /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/...ipt /Users/mondo/Developer/swift-source/swift/benchmark/scripts/compare_perf_tests.p
  # TEST                      SAMPLES MIN(μs) Q1(μs) MEDIAN(μs) Q3(μs) MAX(μs) MAX_RSS(B)
  1 Ackermann                      57     774    774        774    774     780      45056
  2 AngryPhonebook                 56    6608   6615       6615   6643    7077      53248
  3 AnyHashableWithAClass          57  403500 408000     409500 423000  429500      73728
  4 Array2D                        58   15536  15568      15584  15616   15776     552960
  5 ArrayAppend                    56    2000   2000       2000   2050    2220     831488

This looks like quite a significant regression in usability of the build-script! I have killed the process several times before I decided to let run overnight to see if it will finish eventually. For comparison, here is the relevant part from an older run, where the second line starting with cd /Users… shows the full command that gets executed. This info is now missing in the current version of the build-script.

[1613/1613] Running utility command for swift-benchmark-macosx-x86_64
--- Running tests for swift ---
--- check-swift-benchmark-macosx-x86_64 ---
+ /usr/local/bin/cmake --build /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64 -- -j2 check-swift-benchmark-macosx-x86_64
[1/1] cd /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-...ers/mondo/Developer/swift-source/swift/benchmark/scripts/compare_perf_tests.p
FAILED: benchmark/CMakeFiles/check-swift-benchmark-macosx-x86_64 
cd /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver run -o O --output-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --independent-samples 3 && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver run -o Onone --output-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --independent-samples 3 && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver compare --log-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --compare-script /Users/mondo/Developer/swift-source/swift/benchmark/scripts/compare_perf_tests.py
  # TEST                      SAMPLES MIN(μs) Q1(μs) MEDIAN(μs) Q3(μs) MAX(μs) MAX_RSS(B)
  2 AngryPhonebook                 59    6592   6643       6727   6958    7829      36864
  3 AnyHashableWithAClass          55  393500 396500     397500 399500  407500      53248
  4 Array2D                        57   17106  17679      17930  18350   23416    8871936
  5 ArrayAppend                    56    2000   2000       2000   2010    2170     839680

I am not familiar with the implementation of this and during a cursory look at the recent edit history of build-script and build-script-impl I wasn't able to identify the source and author of the new behavior.

Could somebody please restore the original behavior of continuous benchmark report generation and non-hidden benchmarking process?


(Pavol Vaskovic) #2

The benchmarked process doesn't show up in Activity monitor, because it is generally short lived, under 1 second, so that part is OK.

But it is still a problem that Benchmark_Driver's output is being buffered and not continuously printed as before. Anybody knows what has changed in the build-script's handling of this in past 2–3 weeks?


(Pavol Vaskovic) #3

Building from yesterday's master, the benchmark invocation fails because of missing values for the number of --independent-samples:

--- Running tests for swift ---
--- check-swift-benchmark-macosx-x86_64 ---
+ /usr/local/bin/cmake --build /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64 -- -j2 check-swift-benchmark-macosx-x86_64
[1/1] cd /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/...ipt /Users/mondo/Developer/swift-source/swift/benchmark/scripts/compare_perf_tests.p
FAILED: benchmark/CMakeFiles/check-swift-benchmark-macosx-x86_64 
cd /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver run -o O --output-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --independent-samples  && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver run -o Onone --output-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --independent-samples  && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver compare --log-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --compare-script /Users/mondo/Developer/swift-source/swift/benchmark/scripts/compare_perf_tests.py
usage: Benchmark_Driver run [-h] [-f PATTERN] [-t TESTS] [-o OPT]
                            [-i INDEPENDENT_SAMPLES] [--output-dir OUTPUT_DIR]
                            [--swift-repo SWIFT_REPO]
                            [BENCHMARK [BENCHMARK ...]]
Benchmark_Driver run: error: argument -i/--independent-samples: expected one argument
ninja: build stopped: subcommand failed.
./swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

But it looks like the benchmark invocation command is getting printed again… :+1:


(Pavol Vaskovic) #4

I've filed https://bugs.swift.org/browse/SR-9551 in hope this get's to a responsible person that way...