TestFoundation test suite failing while building Swift Toolchain


#1

Hi,

I have been trying to build Swift 5.0 master on PowerPC(powerpc64le) using the build-toolchain script to run the test suite as well. I had been facing issues with tests in lldb suite(LLDB test failures while building Swift 5.0 toolchain). After skipping the failing lldb tests, I am able to complete the libdispatch test suite. However, facing the following issues in TestFoundation test suite:-

[257/257][100%][91.650s] Running utility command for TestFoundation
--- Running tests for foundation ---
--- test ---
+ /usr/bin/cmake --build /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le -- -j8 test
[0/1][  0%][0.000s] Running tests...
Test project /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le
    Start 1: TestFoundation
1/1 Test #1: TestFoundation ...................***Failed    0.00 sec
/root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/TestFoundation/TestFoundation: error while loading shared libraries: libicui18nswift.so.61: cannot open shared object file: No such file or directory


0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.00 sec

The following tests FAILED:
	  1 - TestFoundation (Failed)
Errors while running CTest
FAILED: CMakeFiles/test.util 
cd /root/swift-source/build/buildbot_linux/foundation-linux-powerpc64le && /usr/bin/ctest --force-new-ctest-process
ninja: build stopped: subcommand failed.
Building the standard library for: swift-stdlib-linux-powerpc64le
Running Swift tests for: check-swift-all-linux-powerpc64le check-swift-all-optimize-linux-powerpc64le
./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
./utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

(John McCall) #2

Moved this to the right category.


(Saleem Abdulrasool) #3

The rpath probably doesn't include the ICU build and the ICU that you are building against is a local build (probably a build-script issue, which I don't really use).


#4

Hi @compnerd

Are you referring to any specific file which needs to be looked into in order to correct the rpath and include ICU build?

For example making changes in 'CMakeList.txt' file under 'lldb/tools/repl/swift' directory to include correct rpath for powerpc64le helped resolve quite a few lldb suite related test cases.

diff --git a/tools/repl/swift/CMakeLists.txt b/tools/repl/swift/CMakeLists.txt
index e7733d3..dd419d2 100644
--- a/tools/repl/swift/CMakeLists.txt
+++ b/tools/repl/swift/CMakeLists.txt
@@ -4,8 +4,13 @@ if (LLDB_BUILD_FRAMEWORK)
-Wl,{CMAKE_BINARY_DIR}/{LLDB_FRAMEWORK_INSTALL_DIR}/${LLDB_FRAMEWORK_RESOURCE_DIR}/Swift/macosx")
elseif( CMAKE_SYSTEM_NAME MATCHES "Linux" )

Set the correct rpath to locate libswiftCore

  • set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} \
  •  -Wl,-rpath,${CMAKE_BINARY_DIR}/../swift-linux-x86_64/lib${LLVM_LIBDIR_SUFFIX}/swift/linux -Wl,-ldl")
    
  • if (CMAKE_HOST_SYSTEM_PROCESSOR MATCHES "ppc64le")
  • set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} \
  •    -Wl,-rpath,${CMAKE_BINARY_DIR}/../swift-linux-powerpc64le/lib${LLVM_LIBDIR_SUFFIX}/swift/linux/powerpc64le -Wl,-ldl")
    
  • else()
  • set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} \
  •    -Wl,-rpath,${CMAKE_BINARY_DIR}/../swift-linux-x86_64/lib${LLVM_LIBDIR_SUFFIX}/swift/linux -Wl,-ldl")
    
  • endif()
    set(CMAKE_INSTALL_RPATH "ORIGIN/../lib/swift/linux:{CMAKE_INSTALL_RPATH}")
    endif()

Please suggest, your inputs might make all the difference! :slight_smile:


(Saleem Abdulrasool) #5

The rpath in this case should be setup properly through CMake. Did you specify -DICU_ROOT to the root of the ICU build? Or the appropriate variable set ICU_{I18N,UC}{,_LIB} IIRC? Without that, the link will reference the wrong libraries. I would suggest that you take a look at TestFoundation's link information. llvm-readelf -program-headers is what you want IIRC.


#6

@compnerd thanks for your inputs.
1.) Tried setting ICU_ROOT & DICU_ROOT in "swift/utils/build-script-impl". However faced the same errors as before - "error while loading shared libraries: libicui18nswift.so.61: cannot open shared object file: No such file or directory". Correct me if this is not correct approach to set the two variables.
2.) Used the 'llvm-readelf' utility to find details about the libraries linked with TestFoundation binary:-

	sar@e2641fd15168:~$ /home/sar/swift-source/build/buildbot_linux/llvm-linux-powerpc64le/bin/llvm-readelf -dynamic-table /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le/TestFoundation/TestFoundation
	DynamicSection [ (40 entries)
	  Tag                Type                 Name/Value
	  0x0000000000000003 PLTGOT               0x7d9eb8
	  0x0000000000000002 PLTRELSZ             41448 (bytes)
	  0x0000000000000017 JMPREL               0x86b80
	  0x0000000000000014 PLTREL               RELA
	  0x0000000000000007 RELA                 0x32118
	  0x0000000000000008 RELASZ               346728 (bytes)
	  0x0000000000000009 RELAENT              24 (bytes)
	  0x000000006ffffff9 RELACOUNT            5957
	  0x0000000000000015 DEBUG                0x0
	  0x0000000070000000 PPC64_GLINK          0x6457d0
	  0x0000000000000006 SYMTAB               0x290
	  0x000000000000000b SYMENT               24 (bytes)
	  0x0000000000000005 STRTAB               0x103b8
	  0x000000000000000a STRSZ                132946 (bytes)
	  0x000000006ffffef5 GNU_HASH             0x30b10
	  0x0000000000000001 NEEDED               Shared library: [libswiftCore.so]
	  0x0000000000000001 NEEDED               Shared library: [libpthread.so.0]
	  0x0000000000000001 NEEDED               Shared library: [libutil.so.1]
	  0x0000000000000001 NEEDED               Shared library: [libdl.so.2]
	  0x0000000000000001 NEEDED               Shared library: [libm.so.6]
	  0x0000000000000001 NEEDED               Shared library: [libdispatch.so]
	  0x0000000000000001 NEEDED               Shared library: [libswiftDispatch.so]
	  0x0000000000000001 NEEDED               Shared library: [libBlocksRuntime.so]
	  0x0000000000000001 NEEDED               Shared library: [libXCTest.so]
	  0x0000000000000001 NEEDED               Shared library: [libswiftGlibc.so]
	  0x0000000000000001 NEEDED               Shared library: [libFoundation.so]
	  0x0000000000000001 NEEDED               Shared library: [libstdc++.so.6]
	  0x0000000000000001 NEEDED               Shared library: [libgcc_s.so.1]
	  0x0000000000000001 NEEDED               Shared library: [libc.so.6]
	  0x000000000000000c INIT                 0x90d80
	  0x000000000000000d FINI                 0x6472f0
	  0x000000000000001a FINI_ARRAY           0x792220
	  0x000000000000001c FINI_ARRAYSZ         8 (bytes)
	  0x0000000000000019 INIT_ARRAY           0x792228
	  0x000000000000001b INIT_ARRAYSZ         16 (bytes)
	  **0x000000000000001d RUNPATH              /home/sar/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux:/home/sar/swift-source/build/buildbot_linux/libdispatch-linux-powerpc64le/src**
	  0x000000006ffffff0 VERSYM               0x30b48
	  0x000000006ffffffe VERNEED              0x320b8
	  0x000000006fffffff VERNEEDNUM           3
	  0x0000000000000000 NULL                 0x0
	]

Figured out that the directories mentioned in RUNPATH did not have the required library - libicui18nswift.so.61

sar@e2641fd15168:~$ ll /home/sar/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux
total 16004
drwxr-xr-x 3 sar sar    4096 Feb 13 07:11 ./
drwxr-xr-x 5 sar sar    4096 Feb 13 07:00 ../
-rwxr-xr-x 1 sar sar 8805520 Feb 13 07:11 libswiftCore.so*
-rwxr-xr-x 1 sar sar   84192 Feb 13 07:11 libswiftGlibc.so*
-rw-r--r-- 1 sar sar    4674 Feb 13 06:57 libswiftImageInspectionShared.a
-rw-r--r-- 1 sar sar 1100452 Feb 13 06:57 libswiftReflection.a
-rwxr-xr-x 1 sar sar  688424 Feb 13 06:57 libswiftRemoteMirror.so*
-rwxr-xr-x 1 sar sar   72576 Feb 13 07:11 libswiftRuntimeUnittest.so*
-rwxr-xr-x 1 sar sar   71280 Feb 13 07:11 libswiftSIMDOperators.so*
-rwxr-xr-x 1 sar sar 3534296 Feb 13 07:11 libswiftStdlibCollectionUnittest.so*
-rwxr-xr-x 1 sar sar  237328 Feb 13 07:11 libswiftStdlibUnicodeUnittest.so*
-rwxr-xr-x 1 sar sar  961320 Feb 13 07:11 libswiftStdlibUnittest.so*
-rwxr-xr-x 1 sar sar  529968 Feb 13 07:11 libswiftSwiftOnoneSupport.so*
-rwxr-xr-x 1 sar sar   97056 Feb 13 07:11 libswiftSwiftPrivate.so*
-rwxr-xr-x 1 sar sar   81360 Feb 13 07:11 libswiftSwiftPrivateLibcExtras.so*
-rwxr-xr-x 1 sar sar   82736 Feb 13 07:11 libswiftSwiftPrivateThreadExtras.so*
**drwxr-xr-x 4 sar sar    4096 Feb 13 09:46 powerpc64le/**

However, the /home/sar/swift-source/build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le has all the required libraries. I tried setting RUNPATH through various files which referenced RUNPATH, but it did not help much.

Finally I manually copied the libraries in the directory mentioned in the RUNPATH and it worked. The TestFoundation test suite was executed, though it failed for 4 test cases since I was running as a root user. Running as normal non root user worked and all the test cases for TestFoundation passed successfully:-

[256/257][ 99%][98.957s] Generating TestFoundation.dir/TestFoundation
[257/257][100%][99.479s] Running utility command for TestFoundation
--- Running tests for foundation ---
--- test ---

  • /usr/bin/cmake --build /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le -- -j7 test
    [0/1][ 0%][0.000s] Running tests...
    Test project /home/sar/swift-source/build/buildbot_linux/foundation-linux-powerpc64le
    Start 1: TestFoundation
    1/1 Test #1: TestFoundation ................... Passed 53.48 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 53.49 sec
-- test finished --
--- Finished tests for foundation ---

Question:-
How do I set the correct RUNPATH for the TestFoundation binary. Any specific file that I need to look at and make changes for this?


(Simon Evans) #7

The libicu would actually be loaded by either libswiftCode.so or libFoundation.so, so you would need to check the RUNPATH for those libraries.

I think the libicu libraries are being installed in the powerpc64le subdirectory which would match the location of the corresponding x86_64 ones and these are installed by the build-script-impl when it builds the ICU libraries. ie on my x86_64 Ubuntu I have:

$ find build/buildbot_linux/swift-linux-x86_64/ -name 'libicu*.so'
build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libicudataswift.so
build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libicui18nswift.so
build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libicutestswift.so
build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libicuucswift.so
build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/x86_64/libicutuswift.so

Could you check they are in that subdirectory?

If they are then we just need to determine why x86_64 hosts are finding them in the x86_64 directory not powerpc64le hosts are not finding the corresponding file.


#8

Following are the ICU libraries built across the swift build directory:-

root@37e196fab451:~/swift-source# find build/buildbot_linux/ -name 'libicu*.so'
build/buildbot_linux/libicu-linux-powerpc64le/stubdata/libicudataswift.so
build/buildbot_linux/libicu-linux-powerpc64le/tools/ctestfw/libicutestswift.so
build/buildbot_linux/libicu-linux-powerpc64le/lib/libicuucswift.so
build/buildbot_linux/libicu-linux-powerpc64le/lib/libicudataswift.so
build/buildbot_linux/libicu-linux-powerpc64le/lib/libicutuswift.so
build/buildbot_linux/libicu-linux-powerpc64le/lib/libicui18nswift.so
build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le/libicuucswift.so
build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le/libicudataswift.so
build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le/libicutestswift.so
build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le/libicutuswift.so
build/buildbot_linux/swift-linux-powerpc64le/lib/swift/linux/powerpc64le/libicui18nswift.so


(Simon Evans) #9

I investigated further and I think the reason it works on x86_64 Linux and not on powerpc64le is more by luck than design. None of the RUNPATHs for TestFoundation, libswiftCore.so or libFoundation.so actually contain the x86_64 subdirectory:

$ readelf -aW build/buildbot_linux/foundation-linux-x86_64/TestFoundation/TestFoundation |grep RUNP
 0x000000000000001d (RUNPATH)            Library runpath: [/home/spse/swift-source/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux:/home/spse/swift-source/build/buildbot_linux/libdispatch-linux-x86_64/src]

$ readelf -aW build/buildbot_linux/foundation-linux-x86_64/libFoundation.so |grep RUNPA
 0x000000000000001d (RUNPATH)            Library runpath: [/home/spse/swift-source/build/buildbot_linux/swift-linux-x86_64/lib/swift/linux:/home/spse/swift-source/build/buildbot_linux/libdispatch-linux-x86_64/src:$ORIGIN]

$ readelf -aW build/buildbot_linux/swift-linux-x86_64/lib/swift/linux/libswiftCore.so |grep RUNP
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN:/usr/lib/swift/linux]

But I think the reason it still finds the ICU libraries in the x86_64 subdirectory is due to a 'Hardware capabilities' check that ld.so does when it loads dynamic libraries and looks in an extra set of per cpu-arch defined subdirectories.

An strace when running the TestFoundation binary shows the extra subdirectories it checks:

40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/tls/haswell/x86_64", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/tls/haswell", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/tls/x86_64", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/tls", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/haswell/x86_64", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/haswell", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64/x86_64", 0x7ffd944e2910) = -1 ENOENT (No such file or directory)
40311 stat("/home/spse/swift-source/build/buildbot_linux/xctest-linux-x86_64", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

Note also the tls and haswell directories.

This feature is designed so that per sub-architecture libraries can be used to use any specific features that the CPU.

So ultimately it means it runs on x86_64 mostly by accident and power64le exposes this problem.

I know that @compnerd was working towards putting all of the shared libraries into a per-CPU directory, so that a swift distribution could be used for cross-compiling and building binaries for different architectures in one go. It may now be worth accelerating that change to solve your issue correctly. It would also affect other architectures as well.