Android CrossCompilation on macOS: swiftc fails due missed symbol swift_addNewDSOImage

android
cross-compilation

(Vlad Gorlov) #1

Hello!

Need your help in order to understand the reason of missed swift_addNewDSOImage symbol error when using Swift compiler built on macOS cross-compile host. Swift compiler build with same sources on Ubuntu 18.04.1 (via Vagrant) works without such error. Seems pure CMake configuration issue.

Here is an error I am getting:

swiftrt.o:SwiftRT-ELF.cpp:function swift_image_constructor(): error: undefined reference to 'swift_addNewDSOImage'

From what I understand by exploring CMake files that symbol swift_addNewDSOImage should be inside libswiftCore.so shared object. And it seems inside:

$ nm /.../android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so | grep swift_addNewDSOImage 
003c01a4 T swift_addNewDSOImage

Undefined reference also present in swiftrt.o file.

$ nm /.../android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o | grep swift_addNewDSOImage 
U swift_addNewDSOImage

I am trying to understand purpose of swiftrt.o file and a reason of dedicated configuration for ImageInspectionELF.cpp file in swift/stdlib/public/runtime/CMakeLists.txt CMake file.

Thank you!


Here is a detail log.

mkdir -p "/Users/vova/Repositories/GitHub/Projects/android-on-swift/Build/armv7a/Hello"
cd /Users/vova/Repositories/GitHub/Projects/android-on-swift/Build/armv7a/Hello 
PATH=/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/bin:$PATH \
swiftc \
-v -tools-directory /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin \
-target armv7-none-linux-androideabi \
-Xcc -I/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/sysroot/usr/include \
-Xcc -DDEPLOYMENT_TARGET_ANDROID \
-Xcc -DDEPLOYMENT_RUNTIME_SWIFT \
-Xlinker -v /Users/vova/Repositories/GitHub/Projects/android-on-swift/Projects/Hello/hello.swift
    
Swift version 5.0-dev (LLVM f63b283c71, Clang 41ac4c4262, Swift 8e38b67d66)
Target: armv7-none-linux-android
/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/bin/swift
-frontend -c
-primary-file /Users/vova/Repositories/GitHub/Projects/android-on-swift/Projects/Hello/hello.swift
-target armv7-none-linux-android -disable-objc-interop
-Xcc -I/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/sysroot/usr/include
-Xcc -DDEPLOYMENT_TARGET_ANDROID
-Xcc -DDEPLOYMENT_RUNTIME_SWIFT -color-diagnostics
-module-name hello
-o /var/folders/7l/skdbvw8s5jx0g9vs5_qrkync0000gt/T/hello-16dd94.o
 /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/bin/swift-autolink-extract /var/folders/7l/skdbvw8s5jx0g9vs5_qrkync0000gt/T/hello-16dd94.o
-o /var/folders/7l/skdbvw8s5jx0g9vs5_qrkync0000gt/T/hello-d5644e.autolink

/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/clang++
-fuse-ld=gold -B /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin
-pie -target armv7-none-linux-androideabi /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o /var/folders/7l/skdbvw8s5jx0g9vs5_qrkync0000gt/T/hello-16dd94.o
@/var/folders/7l/skdbvw8s5jx0g9vs5_qrkync0000gt/T/hello-d5644e.autolink
-L /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android
-lswiftCore --target=armv7-none-linux-android -v -Xlinker -v -o hello

Android (4751641 based on r328903) clang version 7.0.2 (https://android.googlesource.com/toolchain/clang 003100370607242ddd5815e4a043907ea9004281) (https://android.googlesource.com/toolchain/llvm 1d739ffb0366421d383e04ff80ec2ee591315116) (based on LLVM 7.0.2svn)
    Target: armv7-none-linux-android
    Thread model: posix
    InstalledDir: /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin
    Found candidate GCC installation: /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x
    Selected GCC installation: /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x
    Candidate multilib: thumb;@mthumb
    Candidate multilib: armv7-a;@march=armv7-a
    Candidate multilib: armv7-a/thumb;@march=armv7-a@mthumb
    Candidate multilib: .;
    Selected multilib: armv7-a;@march=armv7-a

/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld.gold
--sysroot=/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../sysroot
-pie -X --enable-new-dtags --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /system/bin/linker
-o hello /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtbegin_dynamic.o
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/lib64/clang/7.0.2/lib/linux/arm
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/armv7-a
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/../lib/armv7-a
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/armv7-a
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../sysroot/usr/lib
/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o
/var/folders/7l/skdbvw8s5jx0g9vs5_qrkync0000gt/T/hello-16dd94.o -lswiftSwiftOnoneSupport -lswiftCore -lswiftCore
-v -lstdc++ -lm -lgcc -lgcc -ldl -lc -lgcc -lgcc -ldl /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtend_android.o
/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o:SwiftRT-ELF.cpp:function swift_image_constructor(): error: undefined reference to 'swift_addNewDSOImage'

GNU gold (GNU Binutils 2.27.0.20170315) 1.12
clang70++: error: linker command failed with exit code 1 (use -v to see invocation)

btw: Not sure if Swift officially support Android compilation on macOS Cross-Compile host. I am using this Automated workflow to build Swift for Android on macOS: https://github.com/vgorloff/Android-On-Swift


(Saleem Abdulrasool) #2

The swift_addDSOImage is used for image registration with the swift runtime. swiftrt.o is the image registrar and is embedded into the final image to ensure that the image is registered with the runtime upon load. It sounds like you are not linking against the right runtime?


(Vlad Gorlov) #3

It sounds like you are not linking against the right runtime?

Is it means that I have to explicitly specify some libraries (via -l arguments) from NDK for Darwin. Or swiftrt.o, libswiftCore.so, libswiftSwiftOnoneSupport.so, ... compiled with wrong settings. Or maybe Swift compiler itself compiled with wrong settings?


Here is what I have when inspecting files:

$ file /EDITED/armv7a-macos/swift/usr/bin/swift 
/EDITED/armv7a-macos/swift/usr/bin/swift: Mach-O 64-bit executable x86_64

$ file /EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o
/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o: ELF 32-bit LSB relocatable, ARM, EABI5 version 1 (SYSV), not stripped

$ file /EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so
/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so: ELF 32-bit LSB shared object ARM, EABI5 version 1 (SYSV), dynamically linked, with debug_info, not stripped

$ file /EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftSwiftOnoneSupport.so
/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftSwiftOnoneSupport.so: ELF 32-bit LSB shared object ARM, EABI5 version 1 (SYSV), dynamically linked, with debug_info, not stripped

SO-files definitely dynamically linked and seems for correct architecture. swiftrt.o – ELF 32-bit LSB relocatable. But i don't really understand what is relocatable means :smile:


The only significant arguments I am passing to swiftc seems:

  • -tools-directory /.../armv7a-macos/ndk/bin – path to Android NDK for Darwin.
  • -target armv7-none-linux-androideabi – target triple which informs that result executable should be targeted Android (i.e. ELF 32-bit LSB pie executable ARM, EABI5 ...)

Also few flags passed to avoid clang errors "due wrong/missed headers".

  • -Xcc -DDEPLOYMENT_TARGET_ANDROID
  • -Xcc -DDEPLOYMENT_RUNTIME_SWIFT

My guess was that clang++ and ld.gold from NDK doing rest of work. At least same swiftc invocation works well under Ubuntu (Vagrant).

Here is a command line used:

PATH=/EDITED/armv7a-macos/swift/usr/bin:$PATH swiftc -v \
-tools-directory /EDITED/armv7a-macos/ndk/bin \
-target armv7-none-linux-androideabi \
-Xcc -I/EDITED/armv7a-macos/ndk/sysroot/usr/include \
-Xcc -DDEPLOYMENT_TARGET_ANDROID \
-Xcc -DDEPLOYMENT_RUNTIME_SWIFT \
-Xlinker -v /EDITED/Projects/Hello/hello.swift

(Saleem Abdulrasool) #4

Uh, I don't see a -use-ld=gold or -use-ld=lld there. That definitely seems like you're not using the right tools.


(Vlad Gorlov) #5

Thank you!

I isolated issue but can't understand why it is working in one case and not working in other case :0

So, first we can lookup symbols to ensure that swift_addNewDSOImage symbol really exists (in libswiftCore.so.):

nm -a swiftrt.o
         U 
00000000 t 
00000000 t $a.0
00000130 t $d.1
00000000 a SwiftRT-ELF.cpp
00000000 t _ZL23swift_image_constructorv
00000000 b _ZN12_GLOBAL__N_18sectionsE
         U __start_swift5_assocty
         U __start_swift5_fieldmd
         U __start_swift5_protocol_conformances
         U __start_swift5_protocols
         U __start_swift5_reflstr
         U __start_swift5_replace
         U __start_swift5_type_metadata
         U __start_swift5_typeref
         U __stop_swift5_assocty
         U __stop_swift5_fieldmd
         U __stop_swift5_protocol_conformances
         U __stop_swift5_protocols
         U __stop_swift5_reflstr
         U __stop_swift5_replace
         U __stop_swift5_type_metadata
         U __stop_swift5_typeref
         U swift_addNewDSOImage

nm -a libswiftSwiftOnoneSupport.so
         U swift_addNewDSOImage

nm -a libswiftCore.so
003c01a4 T swift_addNewDSOImage

Then we can compile Swift source code:

/EDITED/armv7a-macos/swift/usr/bin/swift -frontend -c \
-primary-file /EDITED/Temp/hello.swift \
-target armv7-none-linux-android -disable-objc-interop \
-Xcc -I/EDITED/armv7a-macos/ndk/sysroot/usr/include \
-Xcc -DDEPLOYMENT_TARGET_ANDROID -Xcc -DDEPLOYMENT_RUNTIME_SWIFT \
-color-diagnostics -module-name hello \
-o /EDITED/Temp/hello.o

file /EDITED/Temp/hello.o

# Prints
/EDITED/Temp/hello.o: ELF 32-bit LSB relocatable, ARM, EABI5 version 1 (SYSV), not stripped

Then we can run swift-autolink-extract tool.

/EDITED/armv7a-macos/swift/usr/bin/swift-autolink-extract \
/EDITED/Temp/hello.o \
-o /EDITED/Temp/hello.autolink

cat /EDITED/Temp/hello.autolink

# Prints
-lswiftSwiftOnoneSupport
-lswiftCore

Then we can launch linker.

/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld.gold \
--sysroot=/EDITED/armv7a-macos/ndk/bin/../sysroot \
-pie -X --enable-new-dtags --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /system/bin/linker \
-o /EDITED/Temp/hello \
/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtbegin_dynamic.o \
-L/EDITED/armv7a-macos/ndk/lib64/clang/7.0.2/lib/linux/arm \
-L/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/armv7-a \
-L/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/../lib/armv7-a \
-L/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib \
-L/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/armv7-a \
-L/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib \
/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o \
/EDITED/Temp/hello.o \
-L/EDITED/armv7a-macos/swift/usr/lib/swift/android \
-lswiftSwiftOnoneSupport -lswiftCore \
-v -lstdc++ -lm -lgcc -lgcc -ldl -lc -lgcc -lgcc -ldl \
/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtend_android.o

# As result error:
/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o:SwiftRT-ELF.cpp:function swift_image_constructor(): error: undefined reference to 'swift_addNewDSOImage'

But if we specifying full path to SO-files, then liker works well.

/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld.gold \
--sysroot=/EDITED/armv7a-macos/ndk/bin/../sysroot \
-pie -X --enable-new-dtags --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /system/bin/linker \
-o /EDITED/Temp/hello \
/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtbegin_dynamic.o \
-L/EDITED/armv7a-macos/ndk/lib64/clang/7.0.2/lib/linux/arm \
-L/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/armv7-a \
-L/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/../lib/armv7-a \
-L/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib \
-L/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/armv7-a \
-L/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib \
/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o \
/EDITED/Temp/hello.o \
/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so \
/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftSwiftOnoneSupport.so \
-v -lstdc++ -lm -lgcc -lgcc -ldl -lc -lgcc -lgcc -ldl \
/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtend_android.o

file /EDITED/Temp/hello

# Prints
/EDITED/Temp/hello: ELF 32-bit LSB pie executable ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /system/bin/linker, with debug_info, not stripped

Difference in two linker invocations is following:

--- untitled
+++ (clipboard)
@@ -11,7 +11,7 @@
 -L/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib \
 /EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o \
 /EDITED/Temp/hello.o \
-/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so \
-/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftSwiftOnoneSupport.so \
+-L/EDITED/armv7a-macos/swift/usr/lib/swift/android \
+-lswiftSwiftOnoneSupport -lswiftCore \
 -v -lstdc++ -lm -lgcc -lgcc -ldl -lc -lgcc -lgcc -ldl \
 /EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtend_android.o

What can be the reason why ld.gold linker from Android NDK can't find symbols if Shared Objects specified not with a full path?

UPDATE: Linker shows same error even if options -lswiftSwiftOnoneSupport -lswiftCore not passed at all. Looks like linker not really using Shared Objects specified with -l option.

UPDATE 2: Order of passing libraries matters.

# Works
cd /EDITED/armv7a/Hello && /EDITED/armv7a-macos/ndk/bin/clang++ -fuse-ld=gold \
-B /EDITED/armv7a-macos/ndk/bin -pie -target armv7-none-linux-androideabi \
/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o \
/EDITED/armv7a/Hello/hello-main.o \
-l/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so \
-l/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftSwiftOnoneSupport.so \
--target=armv7-none-linux-android \
-o /EDITED/armv7a/Hello/hello

# Not working
cd /EDITED/armv7a/Hello && /EDITED/armv7a-macos/ndk/bin/clang++ -fuse-ld=gold \
-B /EDITED/armv7a-macos/ndk/bin -pie -target armv7-none-linux-androideabi \
/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o \
/EDITED/armv7a/Hello/hello-main.o \
-l/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftSwiftOnoneSupport.so \
-l/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so \
--target=armv7-none-linux-android \
-o /EDITED/armv7a/Hello/hello

/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o:SwiftRT-ELF.cpp:function swift_image_constructor(): error: undefined reference to 'swift_addNewDSOImage'
clang70++: error: linker command failed with exit code 1 (use -v to see invocation)

Super strange why libswiftCore.so needs to be passed before libswiftSwiftOnoneSupport.so.