LLDB test failures while building Swift 5.0 toolchain

swift5
#7

Checked, sources are built with -g.

#8

Current status of lldb test suite is as below:-
Expected Passes : 1402
Unsupported Tests : 49
Unexpected Failures: 127

Many of the remaining test cases throw up errors like below:-
Error Type 1(Approx 49 occurences):-

error: Couldn't lookup symbols:
swift_beginAccess
swift_endAccess
error: Couldn't lookup symbols:
swift_allocError
swift_getWitnessTable
swift_willThrow
error: Couldn't lookup symbols:
method descriptor for Swift.Hashable._rawHashValue(seed: Swift.Int) -> Swift.Int
method descriptor for Swift.Hashable.hash(into: inout Swift.Hasher) -> ()
method descriptor for Swift.Hashable.hashValue.getter : Swift.Int
protocol descriptor for Swift.Hashable
[...Many More...]

Error Type 2(Approx 51 occurences):-

AssertionError: False is not True : inline test did not hit a single breakpoint

Error Type 3(Approx 13 occurences):-

AssertionError: False is not True : Expected 1 thread to stop at breakpoint, 0 did.

Investigating on the failure test cases further.

Tried setting the appropriate environment variables like LD_LIBRARY_PATH, LIBRARY_PATH, PATH, etc, however they don't help much.

(Davide C. C. Italiano) #9

In my experience this happens because the REPL doesn't know how to locate libswiftCore.so/dylib.

Can you make sure the RPATH of the repl_swift binary is set up correctly?
(points to something that contains libSwiftcore ?)

To test my theory you can run lldb --repl and type print() and see if you see the errors.

#10

Hi @dcci ,

I looked around the internet and tried exporting the following environment variables with the paths containing 'libswiftCore.so' library:
LD_LIBRARY_PATH, LIBRARY_PATH, RPATH, RUNPATH, SWIFTPM_EMBED_RPATH, REPL_SWIFT_RPATH, DT_RUNPATH, DT_RPATH, DYLD_LIBRARY_PATH.

However even after setting the environment variables, I am still facing the error - "error: Couldn't lookup symbols:"

root@d75bebab716c:~# /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/bin/lldb --repl
Welcome to Swift version 5.0-dev (LLVM f63b283c71, Clang 41ac4c4262, Swift d4598ec15a).
Type :help for assistance.
1> print
error: Couldn't lookup symbols:
Swift.print(_: Any..., separator: Swift.String, terminator: Swift.String) -> ()
swift_beginAccess
swift_endAccess
swift_release

1> print("Hello World!")
error: Couldn't lookup symbols:
Swift.String.init(_builtinStringLiteral: Builtin.RawPointer, utf8CodeUnitCount: Builtin.Word, isASCII: Builtin.Int1) -> Swift.String
type metadata for Swift.String
Swift.allocateUninitializedArray(Builtin.Word) -> (Swift.Array, Builtin.RawPointer)
Swift.print(
: Any..., separator: Swift.String, terminator: Swift.String) -> ()
type metadata for Any
swift_bridgeObjectRelease
swift_release

1>

Am I missing anything?

(Davide C. C. Italiano) #11

Assuming you're building with Cmake, you can go in:
lldb/tools/repl/swift
and see in the CMakeList.txt file:

# Set the correct rpath to locate libswiftCore.
if (LLDB_BUILD_FRAMEWORK)
  set(CMAKE_EXE_LINKER_FLAGS "-Wl,-rpath \
      -Wl,${CMAKE_BINARY_DIR}/${LLDB_FRAMEWORK_INSTALL_DIR}/${LLDB_FRAMEWORK_RESOURCE_DIR}/Swift/macosx")
elseif( CMAKE_SYSTEM_NAME MATCHES "Linux" )
  # Set the correct rpath to locate libswiftCore
  set(CMAKE_EXE_LINKER_FLAGS "-Wl,-rpath,${CMAKE_BINARY_DIR}/../swift-linux-x86_64/lib${LLVM_LIBDIR_SUFFIX}/swift/linux -Wl,-ldl")
  set(CMAKE_INSTALL_RPATH "$ORIGIN/../lib/swift/linux:${CMAKE_INSTALL_RPATH}")
endif()

My wild guess here is that the -Wl,-rpath passed here is wrong. I would say the guess isn't that wild either, because, as you can see, the cmake bits hardcode the architecture, which doesn't match the one you're running on.

To see which RPATH is embedded in the binary generated, repl_swift, I do recommend to use readelf or objdump. BTW, if this is the issue, it would be great if we can get that fixed.

#12

Hi @dcci ,

Thanks for the inputs, I made the below changes in the CMakeList.txt:-

if (CMAKE_HOST_SYSTEM_PROCESSOR MATCHES "ppc64le")
set(CMAKE_EXE_LINKER_FLAGS "-Wl,-rpath,{CMAKE_BINARY_DIR}/../swift-linux-powerpc64le/lib{LLVM_LIBDIR_SUFFIX}/swift/linux/powerpc64le -Wl,-ldl")
else()
set(CMAKE_EXE_LINKER_FLAGS "-Wl,-rpath,{CMAKE_BINARY_DIR}/../swift-linux-x86_64/lib{LLVM_LIBDIR_SUFFIX}/swift/linux -Wl,-ldl")

With this the "error: Couldn't lookup symbols:" are no longer encountered, and the lldb test failure count has come down to 117 from earlier 127 failures.
However for tests I am still seeing errors like below:-

1.) AssertionError: False is not True : inline test did not hit a single breakpoint(51 Occurences)
2.) AssertionError: False is not True : Expected 1 thread to stop at breakpoint, 0 did.(13 Occurences)
3.) AssertionError: False is not True(39 Occurences)
4.) AssertionError: False is not True : Command 'bt
Error output:
error: invalid thread
' returns successfully
(1 Occurence)
5.) AssertionError: False is not True : Command 'frame variable -d run -- o
Error output:
error: invalid thread
' returns successfully
(1 Occurence)

Does anything look familiar or I am still missing something? :|

(Davide C. C. Italiano) #13

I see the failures went down, can you submit a patch?

For the other failures, I'm afraid I can't be of much help.
From what I see you're debugging ppcle which is a very different architecture than the one most people run on. You might consider start debugging the failures, and ask here if you get stuck.

#14

Trying a clean build on the latest swift master branch. However I am facing the following issue:-

Linking /root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le/.bootstrap/bin/swift-test
Linking /root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le/.bootstrap/bin/swift-run
Linking /root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le/.bootstrap/bin/swift-build
Linking /root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le/.bootstrap/bin/swift-package
--- bootstrap: note: building self-hosted 'swift-build': env SWIFT_EXEC=/root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le/powerpc64le-unknown-linux/release/swiftc SWIFTPM_BUILD_DIR=/root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le SWIFTPM_BOOTSTRAP=1 /root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le/powerpc64le-unknown-linux/release/swift-build-stage1 --disable-sandbox -Xlinker -rpath -Xlinker $ORIGIN/../lib/swift/linux -Xswiftc -Xcc -Xswiftc -DSPM_BUILD_IDENT=e49bb01 -Xswiftc -I/root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le/.bootstrap/lib/swift/linux/x86_64 -Xswiftc -Xcc -Xswiftc -F/root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le/.bootstrap/lib/swift/linux/x86_64 -Xswiftc -I/root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le/.bootstrap/lib/swift/linux/x86_64/dispatch -Xlinker -L/root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le/.bootstrap/lib/swift/linux -Xlinker -rpath -Xlinker /root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le/.bootstrap/lib/swift/linux -Xswiftc -I/root/swift-source/build/buildbot_linux/llbuild-linux-powerpc64le/products/llbuildSwift -Xswiftc -I/root/swift-source/llbuild/products/libllbuild/include -Xlinker -rpath -Xlinker $ORIGIN/../lib/swift/pm/llbuild -Xlinker -L/root/swift-source/build/buildbot_linux/llbuild-linux-powerpc64le/lib -Xlinker -rpath -Xlinker /root/swift-source/build/buildbot_linux/llbuild-linux-powerpc64le/lib -Xswiftc -enable-testing --configuration release --build-tests
/root/swift-source/build/buildbot_linux/swiftpm-linux-powerpc64le/powerpc64le-unknown-linux/release/swift-build-stage1: error while loading shared libraries: libicui18nswift.so.61: cannot open shared object file: No such file or directory
--- bootstrap: error: build failed with exit status 127
Building the standard library for: swift-stdlib-linux-powerpc64le
Running Swift tests for: check-swift-all-linux-powerpc64le check-swift-all-optimize-linux-powerpc64le
./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

(Simon Evans) #15

It looks like there is no libicu library, did you pass the --libicu argument to build-script to get it to build?

#16

I am using the build-toolchain script to build as below:-
./swift/utils/build-toolchain srtbundle --test

This was working fine as recently as yesterday, only failing during testing step at the lldb-suite related test suite.

(Simon Evans) #17

Maybe try deleting the build/buildbot_linux/libicu-linux-powerpc64le directory (or whatever the arch specific directory is called) and try building again, it looks to be using the buildbox_linux preset so should be included there.

#18

Thanks @spevans, I did a clean build on another system and it proceeded smoothly.

#19

Hi @dcci ,

Out of the 109 remaining lldb related test failures I am seeing the following kind of errors for 107 tests. On careful observation I found that the --arch parameter passed to the dotest.py is empty. i.e. "Command invoked: /root/swift-source/lldb/test/dotest.py -q --arch= -s /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-traces".
Is this expected or this could be causing issues?

Detailed error is as below(seen for 107/109 failures):-

******************** TEST 'lldb-Suite :: lang/swift/archetype_resolution_subclass/TestArchetypeResolutionSubclass.py' FAILED ********************
lldb version 7.0.0 (https://github.com/apple/swift-lldb.git revision a6ad475f9ee5558baf6baf49e2be7b77ff46df5f)
Swift-5.0 (revision d29b545870946e72549ce2f32dfda8116ba5829f)
clang revision 38ef19dd1f3ea3d09a8b7e77b29aa95fd6ad7d87
llvm revision 28bfcfbd091e6112f164c00f9c555cd2cadfcc64
LLDB library dir: /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/bin
LLDB import library dir: /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/bin
Skipping following debug info categories: ['dsym', 'gmodules']

Session logs for test failures/errors/unexpected successes will go into directory '/root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-traces'
Command invoked: /root/swift-source/lldb/test/dotest.py -q --arch= -s /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-traces --build-dir /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex -S nm -u CXXFLAGS -u CFLAGS --executable /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/./bin/lldb --dsymutil /root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/./bin/dsymutil --filecheck /root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/./bin/FileCheck -C /root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang --swift-compiler /root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/bin/swiftc --swift-library /root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy --build-dir /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex --test-subdir lang/swift --skip-category=watchpoint --skip-category=dwo -t -E -Xcc -F/root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/CoreFoundation-prefix/System/Library/Frameworks -I/root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/swift -I/root/swift-source/swift-corelibs-libdispatch -L/root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le -L/root/swift-source/build/buildbot_linux/libdispatch-linux-powerpc64le -L/root/swift-source/build/buildbot_linux/libdispatch-linux-powerpc64le/src -Xlinker -rpath -Xlinker /root/swift-source/build/buildbot_linux/libdispatch-linux-powerpc64le/src -Xlinker -rpath -Xlinker /root/swift-source/build/buildbot_linux/libdispatch-linux-powerpc64le -Xlinker -rpath -Xlinker /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass -p TestArchetypeResolutionSubclass.py
Change dir to: /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass
UNSUPPORTED: LLDB (/root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang-7-ppc64le) :: test_dsym (lldbsuite.test.lldbtest.TestArchetypeResolutionSubclass) (test case does not fall in any category of interest for this run)
runCmd: settings set symbols.clang-modules-cache-path "/root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/module-cache-lldb"
output: None

runCmd: settings set symbols.enable-external-lookup false
output: None

os command: make VPATH=/root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass -C /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf -I /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass -f /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass/Makefile MAKE_DSYM=NO ARCH=ppc64le CC=/root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang-7 SWIFTC=/root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/bin/swiftc
with pid: 5241
stdout: make: Entering directory '/root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf'

Compiling /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass/main.swift

/root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/bin/swift -frontend -c -primary-file /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass/main.swift

-g -Onone -serialize-debugging-options -module-cache-path /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/module-cache-clang -sdk / -enable-anonymous-context-mangled-names -module-name a -emit-module-path
/root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf/main.partial.swiftmodule
-o /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf/main.o

Merging swift modules for a

/root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/bin/swift -frontend -g -Onone -serialize-debugging-options -module-cache-path /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/module-cache-clang -sdk / -enable-anonymous-context-mangled-names -merge-modules
-emit-module main.partial.swiftmodule
-parse-as-library -sil-merge-partial-modules
-disable-diagnostic-passes -disable-sil-perf-optzns
-module-name a
-o /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf/a.swiftmodule

Wrapping swift module

/root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/bin/swift -modulewrap a.swiftmodule -o /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf/a.swiftmodule.o

Linking a.out

/root/swift-source/build/buildbot_linux/swift-linux-powerpc64le/bin/swiftc a.swiftmodule.o main.o -Onone -Xfrontend -serialize-debugging-options -module-cache-path /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/module-cache-clang -sdk "/" -o "a.out"
make: Leaving directory '/root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/archetype_resolution_subclass/lldbsuite.test.lldbtest.test_dwarf'

stderr: /root/swift-source/lldb/packages/Python/lldbsuite/test/lang/swift/archetype_resolution_subclass/main.swift:11:1: warning: result of call to 'f' is unused
f(x)
^~~~

retcode: 0

FAIL: LLDB (/root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang-7-ppc64le) :: test_dwarf (lldbsuite.test.lldbtest.TestArchetypeResolutionSubclass)
<bound method SBProcess.Kill of <lldb.SBProcess; proxy of <Swig Object of type 'lldb::SBProcess *' at 0x3fff767cee70> >>: success

UNSUPPORTED: LLDB (/root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang-7-ppc64le) :: test_dwo (lldbsuite.test.lldbtest.TestArchetypeResolutionSubclass) (test case does not fall in any category of interest for this run)
UNSUPPORTED: LLDB (/root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang-7-ppc64le) :: test_gmodules (lldbsuite.test.lldbtest.TestArchetypeResolutionSubclass) (test case does not fall in any category of interest for this run)
Restore dir to: /root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/test-results

FAIL: test_dwarf (lldbsuite.test.lldbtest.TestArchetypeResolutionSubclass)

Traceback (most recent call last):
File "/root/swift-source/lldb/packages/Python/lldbsuite/test/lldbtest.py", line 1744, in test_method
return attrvalue(self)
File "/root/swift-source/lldb/packages/Python/lldbsuite/test/decorators.py", line 144, in wrapper
func(*args, **kwargs)
File "/root/swift-source/lldb/packages/Python/lldbsuite/test/lldbinline.py", line 134, in _test
self.do_test()
File "/root/swift-source/lldb/packages/Python/lldbsuite/test/lldbinline.py", line 161, in do_test
"inline test did not hit a single breakpoint")
AssertionError: False is not True : inline test did not hit a single breakpoint
Config=ppc64le-/root/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/clang-7

Ran 4 tests in 3.177s

RESULT: FAILED (0 passes, 1 failures, 0 errors, 3 skipped, 0 expected failures, 0 unexpected successes)

#20

@spevans do you find anything familiar in the above error related to --arch parameter passed to the dotest.py being empty?

#21

The following values were traced while debugging in File "/root/swift-source/lldb/packages/Python/lldbsuite/test/lldbinline.py", line 171, in 'do_test', which is involved in the lldb test failures and reports "AssertionError: False is not True : inline test did not hit a single breakpoint" for 51 test failures:-

exe:-
/root/swift-source/build/buildbot_linux/lldb-linux-powerpc64le/lldb-test-build.noindex/lang/swift/associated_self_type/lldbsuite.test.lldbtest.test_dwarf/a.out

source_files:-
['main.swift']

target:-
a.out

process:-
SBProcess: pid = 20460, state = exited, threads = 1, executable = a.out

Since the process is in exited state on start up, the intended breakpoints do not get hit in the while loop. Hence the failures are seen.
Wondering why the process exits on start up.

#22

All failing lldb test cases have the following type of error message:-
<bound method SBProcess.Kill of <lldb.SBProcess; proxy of <Swig Object of type 'lldb::SBProcess *' at 0x3fff9c15cf90> >>: success

#23

After skipping the failing lldb tests, I am able to complete the libdispatch test suite. However, facing the following issues in TestFoundation test suite:-

[257/257][100%][91.650s] Running utility command for TestFoundation
--- Running tests for foundation ---
--- test ---

  • /usr/bin/cmake --build /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le -- -j8 test
    [0/1][ 0%][0.000s] Running tests...
    Test project /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le
    Start 1: TestFoundation
    1/1 Test #1: TestFoundation ...................***Failed 0.00 sec
    /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/TestFoundation/TestFoundation: error while loading shared libraries: libicui18nswift.so.61: cannot open shared object file: No such file or directory

0% tests passed, 1 tests failed out of 1

Total Test time (real) = 0.00 sec

The following tests FAILED:
1 - TestFoundation (Failed)
Errors while running CTest
FAILED: CMakeFiles/test.util
cd /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le && /usr/bin/ctest --force-new-ctest-process
ninja: build stopped: subcommand failed.
Building the standard library for: swift-stdlib-linux-powerpc64le
Running Swift tests for: check-swift-all-linux-powerpc64le check-swift-all-optimize-linux-powerpc64le
./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

#24

Iam invoking the toolchain and the tests using the following command:-

	./swift/utils/build-toolchain srtbundle --test

I assume this creates a release build, where source files are compiled without any debug information.
So in that case the binaries(a.out and others) might not really help in debugging with lldb, hence the tests fail. Could this be the case here?

#25

Used the 'llvm-readelf' utility to find details about the libraries linked with lldb binary and realised that the directories mentioned in RUNPATH might not have the required libraries.

Similar to reasons mentioned in my earlier issue "TestFoundation test suite failing while building Swift Toolchain", I manually copied the libraries and it worked. The lldb test suite was executed, though it fails now for only 7 test cases:-
Failing Tests (7):
LLDB :: SwiftREPL/ErrorReturn.test
LLDB :: tools/lldb-mi/symbol/symbol-list-lines.test
lldb-Suite :: lang/swift/dwarfimporter/C/TestSwiftDWARFImporterC.py
lldb-Suite :: lang/swift/reference_storage_types/TestSwiftReferenceStorageTypes.py
lldb-Suite :: lang/swift/repl_in_c/TestSwiftReplInC.py
lldb-Suite :: lang/swift/unknown_self/TestSwiftUnknownSelf.py
lldb-Suite :: lang/swift/variables/swift/TestSwiftTypes.py

#26

Attached to this issue are the details of the final 7 failing lldb tests. - https://bugs.swift.org/browse/SR-9888