First off, I think the whole LLDB team can agree on the simple principle
that tests that sometimes fail is not something we’re shooting for, so
we’re on the same page with the general sentiment.
But, rather than talking in sweeping generalizations, let’s take this exact
failure as an example. It was a socket attempting to connect. It uses
general socket timeouts. If it’s using a default, it will typically be
something like 30 seconds. Per my previous comment, this is not something
we see fail with any kind of frequency. We just had a failure on it now.
Is the solution to say we want 35 seconds? 1 minute? 2 minutes? We’re
using end-to-end testing here, and therefore we are subject to how a real
system behaves. If we crank that up, there could be a time where we
overload the server further, and the socket facility just fails. What do
you do then, claim the test as unworthy?
The challenge we run into on LLDB is that, while we do have a growing
number of unit tests that are strict input/output verification, far too
much of a functioning debugger’s operations are not covered adequately by
that. So we have a large body of end to end tests. Sometimes those fail
under heavy load. Sometimes those issues are ours - we have hidden races
that are only exposed when the machine is grinding to a halt. Sometimes it
is that the OS fails to handle requests for resources that the debugger
needs.
So it isn’t as simple as saying “file a bug to make the LLDB test suite
more reliable.” (I wish it were!)
This is specifically why I proposed a concrete strategy like running a
stress test of the test suite. That is actionable, and should shake some
bugs I would hope?
The stress test idea seems like a good one to me. I think that would be a
great opportunity for our quality engineer, who is also looking at other
aspects like code coverage.
-Todd
Another actionable strategies (although one I'm not a big fan of) is
enhance the test suite to automatically re-attempt flaky tests.
On the positive side, we have more resources going into improving our
quality in the upcoming months, and we now have a dedicated quality
engineer working with us. I expect this will help things as we move
forward. And we’re looking to sink some of the system testing that
currently exists end-to-end in LLDB to a more input-output style testing
that we can do without needing the whole of LLDB (e.g. DWARF verification,
DWARF type round-tripping through the compiler, etc. - Greg and Sean can
speak more to what we’re doing there).
Great!
- Daniel
The net effect is that we should be hitting less of this noise.
-Todd
···
On August 19, 2016 at 2:26:18 PM, Daniel Dunbar (daniel_dunbar@apple.com) wrote:
On Aug 19, 2016, at 2:22 PM, Todd Fiala <todd.fiala@gmail.com> wrote:
On August 19, 2016 at 2:10:14 PM, Daniel Dunbar via swift-lldb-dev ( swift-lldb-dev@swift.org) wrote:
The amount of flakiness in the LLDB tests is disturbing... is there a
high-level bug tracking improving this? It seems like it might be worth
running a stress test of the tests on a loaded machine to try and shake out
such problems.
- Daniel
On Aug 19, 2016, at 1:58 PM, Todd Fiala via swift-lldb-dev < swift-lldb-dev@swift.org> wrote:
On Aug 19, 2016, at 1:42 PM, Enrico Granata <egranata@apple.com> wrote:
On Aug 19, 2016, at 11:20 AM, Todd Fiala via swift-lldb-dev < swift-lldb-dev@swift.org> wrote:
It looks like this may be the packaging build:
oss-swift-package-linux-ubuntu-15_10
It was this build:
https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7
It has since gone blue.
I don’t think I recall ever seeing that test fail before in an intermittent
way, but that might be what happened here.
Given the failure mode, it does sound likely:
File "/usr/lib/python2.7/socket.py", line 206, in accept sock, addr =
self._sock.accept() timeout: timed out
If we see that again, we may be able to tweak the socket timeout to allow
for heavily loaded test CI.
-Todd
On Aug 19, 2016, at 11:15 AM, Kate Stone via swift-lldb-dev < swift-lldb-dev@swift.org> wrote:
Where is this failure being reported? I’m not seeing anything on
swift-3.0-branch or master on ci.swift.org.
Kate Stone k8stone@apple.com
Xcode Low Level Tools
On Aug 19, 2016, at 12:15 AM, mishal_shah via swift-lldb-dev < swift-lldb-dev@swift.org> wrote:
+ swift-lldb-dev group.
On Aug 18, 2016, at 9:47 PM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:
Failing in lldb:
Command invoked:
/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py
--executable
/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb
--rerun-all-issues -C
/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang
-s 2016-08-18-15_15_07 --results-port 41187 --inferior -p
TestStubReverseConnect.py
/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test
--event-add-entries worker_index=14:int Configuration: arch=x86_64
compiler=/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang
----------------------------------------------------------------------
Collected 2 tests
ERROR: test_reverse_connect_works_llgs
(TestStubReverseConnect.TestStubReverseConnect)
----------------------------------------------------------------------
Traceback (most recent call last): File
"/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py",
line 121, in wrapper func(*args, **kwargs) File
"/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/decorators.py",
line 121, in wrapper func(*args, **kwargs) File
"/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py",
line 89, in test_reverse_connect_works_llgs self.reverse_connect_works()
File
"/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test/tools/lldb-server/commandline/TestStubReverseConnect.py",
line 67, in reverse_connect_works (stub_socket, address) =
self.listener_socket.accept() File "/usr/lib/python2.7/socket.py", line
206, in accept sock, addr = self._sock.accept() timeout: timed out
Config=x86_64-/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang-3.9
---------------------------------------------------------------------- Ran
2 tests in 22.006s RESULT: FAILED (0 passes, 0 failures, 1 errors, 1
skipped, 0 expected failures, 0 unexpected successes) Session logs for test
failures/errors/unexpected successes can be found in directory
'2016-08-18-15_15_07'[TestStubReverseConnect.py FAILED] Command invoked:
/usr/bin/python
/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/test/dotest.py
--executable
/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/lldb-linux-x86_64/bin/lldb
--rerun-all-issues -C
/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/build/buildbot_linux/llvm-linux-x86_64/bin/clang
-s 2016-08-18-15_15_07 --results-port 41187 --inferior -p
TestStubReverseConnect.py
/home/buildnode/jenkins/workspace/oss-swift-package-linux-ubuntu-15_10/lldb/packages/Python/lldbsuite/test
--event-add-entries worker_index=14:int
- Daniel
On Aug 18, 2016, at 9:18 PM, no-reply@swift.org wrote:
[FAILURE] oss-swift-package-linux-ubuntu-15_10 [#1930]
Build URL:
https://ci.swift.org/job/oss-swift-package-linux-ubuntu-15_10/1930/
Project: oss-swift-package-linux-ubuntu-15_10
Date of build: Thu, 18 Aug 2016 20:26:52 -0700
Build duration: 51 min Identified problems:
- Regression test failed: This build failed because a regression test in
the test suite FAILed. Below is a list of all errors:
- Indication 1
<https://ci.swift.org//job/oss-swift-package-linux-ubuntu-15_10/1930/consoleFull#465392916fca400bf-2f4a-462e-b517-e058d770b2d7>
Changes
- Commit *2ebab526bfd7a5ce6ef460bebee0399fe669612d* by *xi_ge:*
[FixCode] Apply coercion fixits on return statement and initialization
- *edit*: test/FixCode/fixits-apply-objc.swift
- *edit*: test/FixCode/fixits-apply-objc.swift.result
- *edit*: lib/Sema/CSDiag.cpp
- *edit*: test/FixCode/fixits-apply.swift.result
-
- Commit *6447a2d16edcfbceab940d58a321b3ef70525f1a* by *daniel_dunbar:*
[Basic] Update Thread shim to match changes on Linux side.
- *edit*: Sources/Basic/Thread.swift
_______________________________________________
swift-lldb-dev mailing list
swift-lldb-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-lldb-dev
_______________________________________________
swift-lldb-dev mailing list
swift-lldb-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-lldb-dev
_______________________________________________
swift-lldb-dev mailing list
swift-lldb-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-lldb-dev
Thanks,
*- Enrico*
egranata@ <about:blank> <about:blank>.com <about:blank> 
27683
_______________________________________________
swift-lldb-dev mailing list
swift-lldb-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-lldb-dev
_______________________________________________
swift-lldb-dev mailing list
swift-lldb-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-lldb-dev