I have been trying to build Swift 5.0 master on PowerPC(powerpc64le) using the build-toolchain script to run the test suite as well. I had been facing issues with tests in lldb suite(LLDB test failures while building Swift 5.0 toolchain). After skipping the failing lldb tests, I am able to complete the libdispatch test suite. However, facing the following issues in TestFoundation test suite:-
[257/257][100%][91.650s] Running utility command for TestFoundation
--- Running tests for foundation ---
--- test ---
+ /usr/bin/cmake --build /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le -- -j8 test
[0/1][ 0%][0.000s] Running tests...
Test project /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le
Start 1: TestFoundation
1/1 Test #1: TestFoundation ...................***Failed 0.00 sec
/root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/TestFoundation/TestFoundation: error while loading shared libraries: libicui18nswift.so.61: cannot open shared object file: No such file or directory
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.00 sec
The following tests FAILED:
1 - TestFoundation (Failed)
Errors while running CTest
FAILED: CMakeFiles/test.util
cd /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le && /usr/bin/ctest --force-new-ctest-process
ninja: build stopped: subcommand failed.
Building the standard library for: swift-stdlib-linux-powerpc64le
Running Swift tests for: check-swift-all-linux-powerpc64le check-swift-all-optimize-linux-powerpc64le
./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
The rpath probably doesn't include the ICU build and the ICU that you are building against is a local build (probably a build-script issue, which I don't really use).
Are you referring to any specific file which needs to be looked into in order to correct the rpath and include ICU build?
For example making changes in 'CMakeList.txt' file under 'lldb/tools/repl/swift' directory to include correct rpath for powerpc64le helped resolve quite a few lldb suite related test cases.
The rpath in this case should be setup properly through CMake. Did you specify -DICU_ROOT to the root of the ICU build? Or the appropriate variable set ICU_{I18N,UC}{,_LIB} IIRC? Without that, the link will reference the wrong libraries. I would suggest that you take a look at TestFoundation's link information. llvm-readelf -program-headers is what you want IIRC.
@compnerd thanks for your inputs.
1.) Tried setting ICU_ROOT & DICU_ROOT in "swift/utils/build-script-impl". However faced the same errors as before - "error while loading shared libraries: libicui18nswift.so.61: cannot open shared object file: No such file or directory". Correct me if this is not correct approach to set the two variables.
2.) Used the 'llvm-readelf' utility to find details about the libraries linked with TestFoundation binary:-
Figured out that the directories mentioned in RUNPATH did not have the required library - libicui18nswift.so.61
sar@e2641fd15168:~$ ll /home/sar/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux
total 16004
drwxr-xr-x 3 sar sar 4096 Feb 13 07:11 ./
drwxr-xr-x 5 sar sar 4096 Feb 13 07:00 ../
-rwxr-xr-x 1 sar sar 8805520 Feb 13 07:11 libswiftCore.so*
-rwxr-xr-x 1 sar sar 84192 Feb 13 07:11 libswiftGlibc.so*
-rw-r--r-- 1 sar sar 4674 Feb 13 06:57 libswiftImageInspectionShared.a
-rw-r--r-- 1 sar sar 1100452 Feb 13 06:57 libswiftReflection.a
-rwxr-xr-x 1 sar sar 688424 Feb 13 06:57 libswiftRemoteMirror.so*
-rwxr-xr-x 1 sar sar 72576 Feb 13 07:11 libswiftRuntimeUnittest.so*
-rwxr-xr-x 1 sar sar 71280 Feb 13 07:11 libswiftSIMDOperators.so*
-rwxr-xr-x 1 sar sar 3534296 Feb 13 07:11 libswiftStdlibCollectionUnittest.so*
-rwxr-xr-x 1 sar sar 237328 Feb 13 07:11 libswiftStdlibUnicodeUnittest.so*
-rwxr-xr-x 1 sar sar 961320 Feb 13 07:11 libswiftStdlibUnittest.so*
-rwxr-xr-x 1 sar sar 529968 Feb 13 07:11 libswiftSwiftOnoneSupport.so*
-rwxr-xr-x 1 sar sar 97056 Feb 13 07:11 libswiftSwiftPrivate.so*
-rwxr-xr-x 1 sar sar 81360 Feb 13 07:11 libswiftSwiftPrivateLibcExtras.so*
-rwxr-xr-x 1 sar sar 82736 Feb 13 07:11 libswiftSwiftPrivateThreadExtras.so*
**drwxr-xr-x 4 sar sar 4096 Feb 13 09:46 powerpc64le/**
However, the /home/sar/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le has all the required libraries. I tried setting RUNPATH through various files which referenced RUNPATH, but it did not help much.
Finally I manually copied the libraries in the directory mentioned in the RUNPATH and it worked. The TestFoundation test suite was executed, though it failed for 4 test cases since I was running as a root user. Running as normal non root user worked and all the test cases for TestFoundation passed successfully:-
[256/257][ 99%][98.957s] Generating TestFoundation.dir/TestFoundation
[257/257][100%][99.479s] Running utility command for TestFoundation
--- Running tests for foundation ---
--- test ---
/usr/bin/cmake --build /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le -- -j7 test
[0/1][ 0%][0.000s] Running tests...
Test project /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le
Start 1: TestFoundation
1/1 Test #1: TestFoundation ................... Passed 53.48 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 53.49 sec
-- test finished -- --- Finished tests for foundation ---
Question:-
How do I set the correct RUNPATH for the TestFoundation binary. Any specific file that I need to look at and make changes for this?
The libicu would actually be loaded by either libswiftCode.so or libFoundation.so, so you would need to check the RUNPATH for those libraries.
I think the libicu libraries are being installed in the powerpc64le subdirectory which would match the location of the corresponding x86_64 ones and these are installed by the build-script-impl when it builds the ICU libraries. ie on my x86_64 Ubuntu I have:
If they are then we just need to determine why x86_64 hosts are finding them in the x86_64 directory not powerpc64le hosts are not finding the corresponding file.
I investigated further and I think the reason it works on x86_64 Linux and not on powerpc64le is more by luck than design. None of the RUNPATHs for TestFoundation, libswiftCore.so or libFoundation.so actually contain the x86_64 subdirectory:
But I think the reason it still finds the ICU libraries in the x86_64 subdirectory is due to a 'Hardware capabilities' check that ld.so does when it loads dynamic libraries and looks in an extra set of per cpu-arch defined subdirectories.
An strace when running the TestFoundation binary shows the extra subdirectories it checks:
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/tls/haswell/x86_64", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/tls/haswell", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/tls/x86_64", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/tls", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/haswell/x86_64", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/haswell", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/x86_64", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
Note also the tls and haswell directories.
This feature is designed so that per sub-architecture libraries can be used to use any specific features that the CPU.
So ultimately it means it runs on x86_64 mostly by accident and power64le exposes this problem.
I know that @compnerd was working towards putting all of the shared libraries into a per-CPU directory, so that a swift distribution could be used for cross-compiling and building binaries for different architectures in one go. It may now be worth accelerating that change to solve your issue correctly. It would also affect other architectures as well.
Used the 'llvm-readelf' utility to find details about the libraries linked with lldb binary and realised that the directories mentioned in RUNPATH might not have the required libraries.
Similar to reasons mentioned in my earlier issue "TestFoundation test suite failing while building Swift Toolchain - #6 by stamba", I manually copied the libraries and it worked. The lldb test suite was executed, though it fails now for only 7 test cases:-
Failing Tests (7):
LLDB :: SwiftREPL/ErrorReturn.test
LLDB :: tools/lldb-mi/symbol/symbol-list-lines.test
lldb-Suite :: lang/swift/dwarfimporter/C/TestSwiftDWARFImporterC.py
lldb-Suite :: lang/swift/reference_storage_types/TestSwiftReferenceStorageTypes.py
lldb-Suite :: lang/swift/repl_in_c/TestSwiftReplInC.py
lldb-Suite :: lang/swift/unknown_self/TestSwiftUnknownSelf.py
lldb-Suite :: lang/swift/variables/swift/TestSwiftTypes.py
IIRC, lldb relies on LD_LIBRARY_PATH For the tests so the RPATH isn't as important (though, that is something we aught to fix). It sounds like one of the paths being computed in build-script is incorrect if the libraries are failing to be found.
Hi @compnerd, I did try to set the LD_LIBRARY_PATH with all the relevant paths containing the libraries, however it did not work out. The manual copying of the libraries was the last resort that needed to be taken.
That really does sound like LD_LIBRARY_PATH is not setup properly. Note that setting it is insufficient, and that you need to set the correct variables for that to be threaded to the actual test invocation.
