commit 24f685c1d29cd926b1674e099bfcca6ed8673491
Author: Ben Langmuir <blangmuir@apple.com>
Decrease the probability that stdlib/Hashing.swift will fail
Bump the number of trials from 10 to 14, which makes the test orders of
magnitude less likely to fail. For a range of size 10, doing 10 trials
meant that a single call to checkRange would fail ~0.03% of the time;
with 14 trials the spurious failure rate is ~0.0003%. We have 10 calls
to checkRange in this test with ranges of size <= 10.
While this test didn't fail that often before, we have a very large
number of automated builds and it has been a constant low-level source
of friction that this test fails and we have to look at and then ignore
the results.
···
Date: Wed Mar 16 10:36:02 2016
On Mar 16, 2016, at 10:01 AM, Jordan Rose via swift-dev <swift-dev@swift.org> wrote:
Unrelated and harmless: this test is probabilistic.
On Mar 16, 2016, at 10:00 , no-reply@swift.org <mailto:no-reply@swift.org> wrote:
IMO, we should remove/disable this test unless it can be “really” fixed. It is unacceptable to have a test that we know injects noise into our CI systems. Having to second guess whether failures are “real” or not undermines their value, and we should continue to stomp out any nondeterminism from the build and test cycle.
-Chris
···
On Mar 16, 2016, at 11:34 AM, Ben Langmuir via swift-dev <swift-dev@swift.org> wrote:
I was tired of seeing and writing these emails:
commit 24f685c1d29cd926b1674e099bfcca6ed8673491
Author: Ben Langmuir <blangmuir@apple.com <mailto:blangmuir@apple.com>>
Date: Wed Mar 16 10:36:02 2016
Decrease the probability that stdlib/Hashing.swift will fail
Bump the number of trials from 10 to 14, which makes the test orders of
magnitude less likely to fail. For a range of size 10, doing 10 trials
meant that a single call to checkRange would fail ~0.03% of the time;
with 14 trials the spurious failure rate is ~0.0003%. We have 10 calls
to checkRange in this test with ranges of size <= 10.
While this test didn't fail that often before, we have a very large
number of automated builds and it has been a constant low-level source
of friction that this test fails and we have to look at and then ignore
the results.
On Mar 16, 2016, at 10:01 AM, Jordan Rose via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
Unrelated and harmless: this test is probabilistic.
On Mar 16, 2016, at 10:00 , no-reply@swift.org <mailto:no-reply@swift.org> wrote:
On Mar 16, 2016, at 10:33 PM, Chris Lattner via swift-dev <swift-dev@swift.org> wrote:
IMO, we should remove/disable this test unless it can be “really” fixed. It is unacceptable to have a test that we know injects noise into our CI systems. Having to second guess whether failures are “real” or not undermines their value, and we should continue to stomp out any nondeterminism from the build and test cycle.
-Chris
On Mar 16, 2016, at 11:34 AM, Ben Langmuir via swift-dev <swift-dev@swift.org> wrote:
I was tired of seeing and writing these emails:
commit 24f685c1d29cd926b1674e099bfcca6ed8673491
Author: Ben Langmuir <blangmuir@apple.com>
Date: Wed Mar 16 10:36:02 2016
Decrease the probability that stdlib/Hashing.swift will fail
Bump the number of trials from 10 to 14, which makes the test orders of
magnitude less likely to fail. For a range of size 10, doing 10 trials
meant that a single call to checkRange would fail ~0.03% of the time;
with 14 trials the spurious failure rate is ~0.0003%. We have 10 calls
to checkRange in this test with ranges of size <= 10.
While this test didn't fail that often before, we have a very large
number of automated builds and it has been a constant low-level source
of friction that this test fails and we have to look at and then ignore
the results.
On Mar 16, 2016, at 10:01 AM, Jordan Rose via swift-dev <swift-dev@swift.org> wrote:
Unrelated and harmless: this test is probabilistic.
On Mar 16, 2016, at 10:00 , no-reply@swift.org wrote: