Questions about Swift-CI


(Pavol Vaskovic) #1

Hi Mishal,

Douglas Gregor suggested in SR-4814 (that was merged as dupe into SR-4669) that I get in contact with you because I’m working on issue SR-4669 Add a Benchmark_Driver --rerun N option <https://bugs.swift.org/browse/SR-4669> and I should make sure that I don’t break the Swift-CI.

The point of that is to deal with false changes (regressions and improvements) that are often seen in benchmark reports from CI.

I have run Benchmark_O with --num-iters=100 on my machine for the the whole performance test suite, to get a feeling for the distribution of benchmark samples, because I also want to move the Benchmark_Driver to use MEAN instead of MIN in the analysis.

There was a tremendous difference on the quality of measured samples depending on how busy my machine was. When I had idle browser windows open in the background, or Xcode open, the results had very high variance. When I purged all I could the results were very stable.

First I want to check with you that the Macs that run the CI and benchmarks do have turned off the Spotlight indexing on the folder that they check out the project into (/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift). Just making sure that `mds`, `mds_stores` and `mds_worker` processes don’t unnecessarily impact the benchmark runs. How about Time Machine's `backupd`?

···

---

Also I have recently filed SR-4659 Benchmark logs should be tied to tested tree version <https://bugs.swift.org/browse/SR-4659> and implemented a fix that got merged in PR #8978 <https://github.com/apple/swift/pull/8978>, adding short hash of the commit to the Benchmark log file name. But that works only while you don’t make local changes. In the end I would like to have fully identified versions of benchmarks in the comparison report. In order to make that work, I’ve filed SR-4675 Report detailed build version from Benchmark drivers <https://bugs.swift.org/browse/SR-4675>, that calls for having something akin to `swift —version` be baked into the Benchmark_O{Onone, Ounchecked}.

Given we are on git now, I think the solution is to modify cmake to run `git describe` and generate some kind of version file that will be included during build and reported by the driver when running benchmarks.

I thought it would be nice touch to report also the branch names like one that is cited in the PRs on github. For example:

apple:master vs palimondo:SR-4659

I found that getSVN.cmake <https://github.com/apple/swift-llvm/blob/stable/cmake/modules/GetSVN.cmake#L73> from llvm has all the commands to extract the needed information… But, whenI looked at full console logs from swift-ci to see if that would work with the way how the branches are checked out on the build machine, I’m a bit confused:

Cloning the remote Git repository
Cloning repository git@github.com:apple/swift.git
> git init /Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift # timeout=10
Fetching upstream changes from git@github.com:apple/swift.git
> git --version # timeout=10
> git fetch --tags --progress git@github.com:apple/swift.git +refs/heads/*:refs/remotes/origin/*
> git config remote.origin.url git@github.com:apple/swift.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url git@github.com:apple/swift.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
No valid HEAD. Skipping the resetting
> git clean -fdx # timeout=10
Fetching upstream changes from git@github.com:apple/swift.git
> git fetch --tags --progress git@github.com:apple/swift.git +refs/pull/*:refs/remotes/origin/pr/*
> git rev-parse refs/remotes/origin/pr/9330/merge^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/pr/9330/merge^{commit} # timeout=10
Checking out Revision 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5 (refs/remotes/origin/pr/9330/merge)
> git config core.sparsecheckout # timeout=10
> git checkout -f 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5
Using 'Changelog to branch' strategy.
[swift-PR-osx-perf] $ /bin/bash -e /var/folders/_8/79jmzf2142z2xydc_01btlx00000gn/T/hudson7057602820198606732.sh
Cloning into 'ninja'...
Branch: master
Found related pull requests: []

I don’t quite understand how is the PR branch cloned from github. It definitely looks different from the situation wehn I build on my local machine.

What would be the way to extract remote repository and branch name for the PR being merged?

When I’ll be working on these changes, It would be able to build just the benchmark suite, but SR-4593 Following instructions to independently build benchmarks fails to do so <https://bugs.swift.org/browse/SR-4593>. Who can help me here? (So far I have always been using ./swift/utils/build-script -R, but that always takes ages, and often rebuilds whole compiler infrastructure.)

Best regards
Pavol Vaskovic


(Dave Abrahams) #2

Hi Mishal,

Douglas Gregor suggested in SR-4814 (that was merged as dupe into
SR-4669) that I get in contact with you because I’m working on issue
SR-4669 Add a Benchmark_Driver --rerun N option
<https://bugs.swift.org/browse/SR-4669> and I should make sure that I
don’t break the Swift-CI.

The point of that is to deal with false changes (regressions and improvements) that are often seen
in benchmark reports from CI.

I have run Benchmark_O with --num-iters=100 on my machine for the the
whole performance test suite, to get a feeling for the distribution of
benchmark samples, because I also want to move the Benchmark_Driver to
use MEAN instead of MIN in the analysis.

I'm concerned about that, especially for microbenchmarks; it seems to me
as though MIN is the right measurement. Can you explain why MEAN is
better?

There was a tremendous difference on the quality of measured samples
depending on how busy my machine was. When I had idle browser windows
open in the background, or Xcode open, the results had very high
variance. When I purged all I could the results were very stable.

First I want to check with you that the Macs that run the CI and
benchmarks do have turned off the Spotlight indexing on the folder
that they check out the project into
(/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift). Just
making sure that `mds`, `mds_stores` and `mds_worker` processes don’t
unnecessarily impact the benchmark runs. How about Time Machine's
`backupd`?

---

Also I have recently filed SR-4659 Benchmark logs should be tied to
tested tree version <https://bugs.swift.org/browse/SR-4659> and
implemented a fix that got merged in PR #8978
<https://github.com/apple/swift/pull/8978>, adding short hash of the
commit to the Benchmark log file name. But that works only while you
don’t make local changes. In the end I would like to have fully
identified versions of benchmarks in the comparison report.

Have you thought about using git notes to attach results to commits?
That has always seemed to me like a good way to manage test results.

···

on Thu May 11 2017, Pavol Vaskovic <swift-dev-AT-swift.org> wrote:

In order to make that work, I’ve filed SR-4675 Report detailed build
version from Benchmark drivers
<https://bugs.swift.org/browse/SR-4675>, that calls for having
something akin to `swift —version` be baked into the
Benchmark_O{Onone, Ounchecked}.

Given we are on git now, I think the solution is to modify cmake to
run `git describe` and generate some kind of version file that will be
included during build and reported by the driver when running
benchmarks.

I thought it would be nice touch to report also the branch names like one that is cited in the PRs
on github. For example:

apple:master vs palimondo:SR-4659

I found that getSVN.cmake
<https://github.com/apple/swift-llvm/blob/stable/cmake/modules/GetSVN.cmake#L73>
from llvm has all the commands to extract the needed information… But,
whenI looked at full console logs from swift-ci to see if that would
work with the way how the branches are checked out on the build
machine, I’m a bit confused:

Cloning the remote Git repository
Cloning repository git@github.com:apple/swift.git
> git init /Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift # timeout=10
Fetching upstream changes from git@github.com:apple/swift.git
> git --version # timeout=10
> git fetch --tags --progress git@github.com:apple/swift.git +refs/heads/*:refs/remotes/origin/*
> git config remote.origin.url git@github.com:apple/swift.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url git@github.com:apple/swift.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
No valid HEAD. Skipping the resetting
> git clean -fdx # timeout=10
Fetching upstream changes from git@github.com:apple/swift.git
> git fetch --tags --progress git@github.com:apple/swift.git +refs/pull/*:refs/remotes/origin/pr/*
> git rev-parse refs/remotes/origin/pr/9330/merge^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/pr/9330/merge^{commit} # timeout=10
Checking out Revision 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5 (refs/remotes/origin/pr/9330/merge)
> git config core.sparsecheckout # timeout=10
> git checkout -f 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5
Using 'Changelog to branch' strategy.
[swift-PR-osx-perf] $ /bin/bash -e

/var/folders/_8/79jmzf2142z2xydc_01btlx00000gn/T/hudson7057602820198606732.sh

Cloning into 'ninja'...
Branch: master
Found related pull requests: []

I don’t quite understand how is the PR branch cloned from github. It definitely looks different from
the situation wehn I build on my local machine.

What would be the way to extract remote repository and branch name for the PR being merged?

When I’ll be working on these changes, It would be able to build just
the benchmark suite, but SR-4593 Following instructions to
independently build benchmarks fails to do so
<https://bugs.swift.org/browse/SR-4593>. Who can help me here? (So far
I have always been using ./swift/utils/build-script -R, but that
always takes ages, and often rebuilds whole compiler infrastructure.)

Best regards
Pavol Vaskovic

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
-Dave


(Andrew Trick) #3

Using MEAN wasn’t part of the aforementioned SR-4669. The purpose of that task is to reduce the time CI takes to get useful results (e.g. by using 3 runs as a baseline). MEAN isn’t useful if you’re only gathering 3 data points.

-Andy

···

On May 11, 2017, at 1:33 PM, Pavol Vaskovic via swift-dev <swift-dev@swift.org> wrote:

Douglas Gregor suggested in SR-4814 (that was merged as dupe into SR-4669) that I get in contact with you because I’m working on issue SR-4669 Add a Benchmark_Driver --rerun N option <https://bugs.swift.org/browse/SR-4669> and I should make sure that I don’t break the Swift-CI.

The point of that is to deal with false changes (regressions and improvements) that are often seen in benchmark reports from CI.

I have run Benchmark_O with --num-iters=100 on my machine for the the whole performance test suite, to get a feeling for the distribution of benchmark samples, because I also want to move the Benchmark_Driver to use MEAN instead of MIN in the analysis.


(Mishal Shah) #4

Hi Pavol,

Hi Mishal,

Douglas Gregor suggested in SR-4814 (that was merged as dupe into SR-4669) that I get in contact with you because I’m working on issue SR-4669 Add a Benchmark_Driver --rerun N option <https://bugs.swift.org/browse/SR-4669> and I should make sure that I don’t break the Swift-CI.

The point of that is to deal with false changes (regressions and improvements) that are often seen in benchmark reports from CI.

I have run Benchmark_O with --num-iters=100 on my machine for the the whole performance test suite, to get a feeling for the distribution of benchmark samples, because I also want to move the Benchmark_Driver to use MEAN instead of MIN in the analysis.

There was a tremendous difference on the quality of measured samples depending on how busy my machine was. When I had idle browser windows open in the background, or Xcode open, the results had very high variance. When I purged all I could the results were very stable.

First I want to check with you that the Macs that run the CI and benchmarks do have turned off the Spotlight indexing on the folder that they check out the project into (/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift). Just making sure that `mds`, `mds_stores` and `mds_worker` processes don’t unnecessarily impact the benchmark runs. How about Time Machine's `backupd`?

Yes, we have these services disabled.

---

Also I have recently filed SR-4659 Benchmark logs should be tied to tested tree version <https://bugs.swift.org/browse/SR-4659> and implemented a fix that got merged in PR #8978 <https://github.com/apple/swift/pull/8978>, adding short hash of the commit to the Benchmark log file name. But that works only while you don’t make local changes. In the end I would like to have fully identified versions of benchmarks in the comparison report. In order to make that work, I’ve filed SR-4675 Report detailed build version from Benchmark drivers <https://bugs.swift.org/browse/SR-4675>, that calls for having something akin to `swift —version` be baked into the Benchmark_O{Onone, Ounchecked}.

Given we are on git now, I think the solution is to modify cmake to run `git describe` and generate some kind of version file that will be included during build and reported by the driver when running benchmarks.

I thought it would be nice touch to report also the branch names like one that is cited in the PRs on github. For example:

apple:master vs palimondo:SR-4659

For pull request jobs following env will be available.

${ghprbTargetBranch} (master) vs ${ghprbSourceBranch} (SR-4659)

I found that getSVN.cmake <https://github.com/apple/swift-llvm/blob/stable/cmake/modules/GetSVN.cmake#L73> from llvm has all the commands to extract the needed information… But, whenI looked at full console logs from swift-ci to see if that would work with the way how the branches are checked out on the build machine, I’m a bit confused:

Cloning the remote Git repository
Cloning repository git@github.com <mailto:git@github.com>:apple/swift.git
> git init /Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift # timeout=10
Fetching upstream changes from git@github.com <mailto:git@github.com>:apple/swift.git
> git --version # timeout=10
> git fetch --tags --progress git@github.com <mailto:git@github.com>:apple/swift.git +refs/heads/*:refs/remotes/origin/*
> git config remote.origin.url git@github.com <mailto:git@github.com>:apple/swift.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url git@github.com <mailto:git@github.com>:apple/swift.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
No valid HEAD. Skipping the resetting
> git clean -fdx # timeout=10
Fetching upstream changes from git@github.com <mailto:git@github.com>:apple/swift.git
> git fetch --tags --progress git@github.com <mailto:git@github.com>:apple/swift.git +refs/pull/*:refs/remotes/origin/pr/*
> git rev-parse refs/remotes/origin/pr/9330/merge^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/pr/9330/merge^{commit} # timeout=10
Checking out Revision 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5 (refs/remotes/origin/pr/9330/merge)
> git config core.sparsecheckout # timeout=10
> git checkout -f 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5
Using 'Changelog to branch' strategy.
[swift-PR-osx-perf] $ /bin/bash -e /var/folders/_8/79jmzf2142z2xydc_01btlx00000gn/T/hudson7057602820198606732.sh
Cloning into 'ninja'...
Branch: master
Found related pull requests: []

I don’t quite understand how is the PR branch cloned from github. It definitely looks different from the situation wehn I build on my local machine.

Github provides merge ref for PR, for example "origin/pr/9330/merge”. This allows us to test PR with latest changes from targeted branch, avoiding out of date PR testing.

What would be the way to extract remote repository and branch name for the PR being merged?

You can use ${ghprbTargetBranch} for targeted branch, ${ghprbSourceBranch} for PR branch, ${ghprbPullAuthorLogin} for PR author.

Full list of env: https://ci.swift.org/view/Pull%20Request/job/swift-PR-osx-perf/558/parameters/

···

On May 11, 2017, at 1:33 PM, Pavol Vaskovic <pali@pali.sk> wrote:

When I’ll be working on these changes, It would be able to build just the benchmark suite, but SR-4593 Following instructions to independently build benchmarks fails to do so <https://bugs.swift.org/browse/SR-4593>. Who can help me here? (So far I have always been using ./swift/utils/build-script -R, but that always takes ages, and often rebuilds whole compiler infrastructure.)

Best regards
Pavol Vaskovic