[Swift CI] Build Still Failing: 0. OSS - Swift Incremental RA - Ubuntu 16.10 (master) #3698


(Michael Ilseman) #1

(sending to everyone and swift-corelibs-dev for real this time)

Failure in dispatch-starfish. This test looks timing-sensitive. Boris, is this unlikely to be related to your changes? Does anyone familiar with dispatch have suggestions, such as increasing the latency threshold?

[BEGIN] Latency
  Actual: 1553
  Expected: <1000
[FAIL] Latency (dispatch_starfish.c:82)
  dispatch_starfish.c:82
lap: 9
count: 1000
delta: 6756400000 ns
math: 3374.825175 ns / lap

[BEGIN] Latency
  Actual: 3374
  Expected: <1000
[FAIL] Latency (dispatch_starfish.c:82)
  dispatch_starfish.c:82

...

···

On May 17, 2017, at 10:52 AM, no-reply@swift.org wrote:

New issue found!

[FAILURE] oss-swift-incremental-RA-linux-ubuntu-16_10 [#3698]

Build URL: https://ci.swift.org/job/oss-swift-incremental-RA-linux-ubuntu-16_10/3698/
Project: oss-swift-incremental-RA-linux-ubuntu-16_10
Date of build: Wed, 17 May 2017 10:30:19 -0700
Build duration: 21 min
Tests:

Name: Swift(linux-x86_64)
Failed: 0 test(s), Passed: 9400 test(s), Total: 9400 test(s)
Name: Swift-Unit
Failed: 0 test(s), Passed: 432 test(s), Total: 432 test(s)

Changes

Commit 369d92878bc5203439b09a6680424acf6b2d22ac by Boris Buegling:
Add roots to verbose description of `StaleFileRemovalCommand`

edit: lib/BuildSystem/BuildSystem.cpp


(Dave Grove) #2

The test is subject to failing spuriously if the machine is badly
overloaded. One could increase the threshold by changing the #define on
line 43 of dispatch_starfish.c

There's a trade-off between spurious failures due to overloads on build
machines vs. missing performance regressions by making the latency
thresholds much too lenient.

We've been running with the current threshold in dispatch_starfish since we
began running the dispatch tests in the CI system last summer. So if this
isn't a one-off failure caused by CI load, it might be pointing to an
underlying performance regression.

--dave

···

swift-dev-bounces@swift.org wrote on 05/17/2017 02:04:23 PM:

Failure in dispatch-starfish. This test looks timing-sensitive.
Boris, is this unlikely to be related to your changes? Does anyone
familiar with dispatch have suggestions, such as increasing the
latency threshold?

[BEGIN] Latency
Actual: 1553
Expected: <1000
[FAIL] Latency (dispatch_starfish.c:82)
dispatch_starfish.c:82
lap: 9
count: 1000
delta: 6756400000 ns
math: 3374.825175 ns / lap

[BEGIN] Latency
Actual: 3374
Expected: <1000
[FAIL] Latency (dispatch_starfish.c:82)
dispatch_starfish.c:82


(Michael Ilseman) #3

Ok. If this occurs again, please update the threshold.

···

On May 17, 2017, at 12:03 PM, David P Grove <groved@us.ibm.com> wrote:

swift-dev-bounces@swift.org wrote on 05/17/2017 02:04:23 PM:
>
> Failure in dispatch-starfish. This test looks timing-sensitive.
> Boris, is this unlikely to be related to your changes? Does anyone
> familiar with dispatch have suggestions, such as increasing the
> latency threshold?
>
> [BEGIN] Latency
> Actual: 1553
> Expected: <1000
> [FAIL] Latency (dispatch_starfish.c:82)
> dispatch_starfish.c:82
> lap: 9
> count: 1000
> delta: 6756400000 ns
> math: 3374.825175 ns / lap
>
> [BEGIN] Latency
> Actual: 3374
> Expected: <1000
> [FAIL] Latency (dispatch_starfish.c:82)
> dispatch_starfish.c:82
>

The test is subject to failing spuriously if the machine is badly overloaded. One could increase the threshold by changing the #define on line 43 of dispatch_starfish.c

There's a trade-off between spurious failures due to overloads on build machines vs. missing performance regressions by making the latency thresholds much too lenient.

We've been running with the current threshold in dispatch_starfish since we began running the dispatch tests in the CI system last summer. So if this isn't a one-off failure caused by CI load, it might be pointing to an underlying performance regression.

--dave