Hi Mishal,
Douglas Gregor suggested in SR-4814 (that was merged as dupe into SR-4669) that I get in contact with you because I’m working on issue SR-4669 Add a Benchmark_Driver --rerun N option <Issues · apple/swift-issues · GitHub; and I should make sure that I don’t break the Swift-CI.
The point of that is to deal with false changes (regressions and improvements) that are often seen in benchmark reports from CI.
I have run Benchmark_O with --num-iters=100 on my machine for the the whole performance test suite, to get a feeling for the distribution of benchmark samples, because I also want to move the Benchmark_Driver to use MEAN instead of MIN in the analysis.
There was a tremendous difference on the quality of measured samples depending on how busy my machine was. When I had idle browser windows open in the background, or Xcode open, the results had very high variance. When I purged all I could the results were very stable.
First I want to check with you that the Macs that run the CI and benchmarks do have turned off the Spotlight indexing on the folder that they check out the project into (/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift). Just making sure that `mds`, `mds_stores` and `mds_worker` processes don’t unnecessarily impact the benchmark runs. How about Time Machine's `backupd`?
···
---
Also I have recently filed SR-4659 Benchmark logs should be tied to tested tree version <Issues · apple/swift-issues · GitHub; and implemented a fix that got merged in PR #8978 <https://github.com/apple/swift/pull/8978>, adding short hash of the commit to the Benchmark log file name. But that works only while you don’t make local changes. In the end I would like to have fully identified versions of benchmarks in the comparison report. In order to make that work, I’ve filed SR-4675 Report detailed build version from Benchmark drivers <Issues · apple/swift-issues · GitHub, that calls for having something akin to `swift —version` be baked into the Benchmark_O{Onone, Ounchecked}.
Given we are on git now, I think the solution is to modify cmake to run `git describe` and generate some kind of version file that will be included during build and reported by the driver when running benchmarks.
I thought it would be nice touch to report also the branch names like one that is cited in the PRs on github. For example:
apple:master vs palimondo:SR-4659
I found that getSVN.cmake <https://github.com/apple/swift-llvm/blob/stable/cmake/modules/GetSVN.cmake#L73> from llvm has all the commands to extract the needed information… But, whenI looked at full console logs from swift-ci to see if that would work with the way how the branches are checked out on the build machine, I’m a bit confused:
Cloning the remote Git repository
Cloning repository git@github.com:apple/swift.git
> git init /Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift # timeout=10
Fetching upstream changes from git@github.com:apple/swift.git
> git --version # timeout=10
> git fetch --tags --progress git@github.com:apple/swift.git +refs/heads/*:refs/remotes/origin/*
> git config remote.origin.url git@github.com:apple/swift.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url git@github.com:apple/swift.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
No valid HEAD. Skipping the resetting
> git clean -fdx # timeout=10
Fetching upstream changes from git@github.com:apple/swift.git
> git fetch --tags --progress git@github.com:apple/swift.git +refs/pull/*:refs/remotes/origin/pr/*
> git rev-parse refs/remotes/origin/pr/9330/merge^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/pr/9330/merge^{commit} # timeout=10
Checking out Revision 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5 (refs/remotes/origin/pr/9330/merge)
> git config core.sparsecheckout # timeout=10
> git checkout -f 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5
Using 'Changelog to branch' strategy.
[swift-PR-osx-perf] $ /bin/bash -e /var/folders/_8/79jmzf2142z2xydc_01btlx00000gn/T/hudson7057602820198606732.sh
Cloning into 'ninja'...
Branch: master
Found related pull requests:
I don’t quite understand how is the PR branch cloned from github. It definitely looks different from the situation wehn I build on my local machine.
What would be the way to extract remote repository and branch name for the PR being merged?
When I’ll be working on these changes, It would be able to build just the benchmark suite, but SR-4593 Following instructions to independently build benchmarks fails to do so <https://bugs.swift.org/browse/SR-4593>\. Who can help me here? (So far I have always been using ./swift/utils/build-script -R, but that always takes ages, and often rebuilds whole compiler infrastructure.)
Best regards
Pavol Vaskovic