Android CrossCompilation on macOS: swiftc fails due missed symbol swift_addNewDSOImage

v.gorlov · 2019-01-02T21:48:06.023Z

Hello!

Need your help in order to understand the reason of missed swift_addNewDSOImage symbol error when using Swift compiler built on macOS cross-compile host. Swift compiler build with same sources on Ubuntu 18.04.1 (via Vagrant) works without such error. Seems pure CMake configuration issue.

Here is an error I am getting:

swiftrt.o:SwiftRT-ELF.cpp:function swift_image_constructor(): error: undefined reference to 'swift_addNewDSOImage'

From what I understand by exploring CMake files that symbol swift_addNewDSOImage should be inside libswiftCore.so shared object. And it seems inside:

$ nm /.../android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so | grep swift_addNewDSOImage 
003c01a4 T swift_addNewDSOImage

Undefined reference also present in swiftrt.o file.

$ nm /.../android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o | grep swift_addNewDSOImage 
U swift_addNewDSOImage

I am trying to understand purpose of swiftrt.o file and a reason of dedicated configuration for ImageInspectionELF.cpp file in swift/stdlib/public/runtime/CMakeLists.txt CMake file.

Thank you!

Here is a detail log.

mkdir -p "/Users/vova/Repositories/GitHub/Projects/android-on-swift/Build/armv7a/Hello"
cd /Users/vova/Repositories/GitHub/Projects/android-on-swift/Build/armv7a/Hello 
PATH=/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/bin:$PATH \
swiftc \
-v -tools-directory /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin \
-target armv7-none-linux-androideabi \
-Xcc -I/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/sysroot/usr/include \
-Xcc -DDEPLOYMENT_TARGET_ANDROID \
-Xcc -DDEPLOYMENT_RUNTIME_SWIFT \
-Xlinker -v /Users/vova/Repositories/GitHub/Projects/android-on-swift/Projects/Hello/hello.swift
    
Swift version 5.0-dev (LLVM f63b283c71, Clang 41ac4c4262, Swift 8e38b67d66)
Target: armv7-none-linux-android
/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/bin/swift
-frontend -c
-primary-file /Users/vova/Repositories/GitHub/Projects/android-on-swift/Projects/Hello/hello.swift
-target armv7-none-linux-android -disable-objc-interop
-Xcc -I/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/sysroot/usr/include
-Xcc -DDEPLOYMENT_TARGET_ANDROID
-Xcc -DDEPLOYMENT_RUNTIME_SWIFT -color-diagnostics
-module-name hello
-o /var/folders/7l/skdbvw8s5jx0g9vs5_qrkync0000gt/T/hello-16dd94.o
 /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/bin/swift-autolink-extract /var/folders/7l/skdbvw8s5jx0g9vs5_qrkync0000gt/T/hello-16dd94.o
-o /var/folders/7l/skdbvw8s5jx0g9vs5_qrkync0000gt/T/hello-d5644e.autolink

/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/clang++
-fuse-ld=gold -B /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin
-pie -target armv7-none-linux-androideabi /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o /var/folders/7l/skdbvw8s5jx0g9vs5_qrkync0000gt/T/hello-16dd94.o
@/var/folders/7l/skdbvw8s5jx0g9vs5_qrkync0000gt/T/hello-d5644e.autolink
-L /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android
-lswiftCore --target=armv7-none-linux-android -v -Xlinker -v -o hello

Android (4751641 based on r328903) clang version 7.0.2 (https://android.googlesource.com/toolchain/clang 003100370607242ddd5815e4a043907ea9004281) (https://android.googlesource.com/toolchain/llvm 1d739ffb0366421d383e04ff80ec2ee591315116) (based on LLVM 7.0.2svn)
    Target: armv7-none-linux-android
    Thread model: posix
    InstalledDir: /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin
    Found candidate GCC installation: /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x
    Selected GCC installation: /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x
    Candidate multilib: thumb;@mthumb
    Candidate multilib: armv7-a;@march=armv7-a
    Candidate multilib: armv7-a/thumb;@march=armv7-a@mthumb
    Candidate multilib: .;
    Selected multilib: armv7-a;@march=armv7-a

/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld.gold
--sysroot=/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../sysroot
-pie -X --enable-new-dtags --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /system/bin/linker
-o hello /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtbegin_dynamic.o
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/lib64/clang/7.0.2/lib/linux/arm
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/armv7-a
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/../lib/armv7-a
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/armv7-a
-L/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../sysroot/usr/lib
/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o
/var/folders/7l/skdbvw8s5jx0g9vs5_qrkync0000gt/T/hello-16dd94.o -lswiftSwiftOnoneSupport -lswiftCore -lswiftCore
-v -lstdc++ -lm -lgcc -lgcc -ldl -lc -lgcc -lgcc -ldl /Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtend_android.o
/Users/vova/Repositories/GitHub/Projects/android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o:SwiftRT-ELF.cpp:function swift_image_constructor(): error: undefined reference to 'swift_addNewDSOImage'

GNU gold (GNU Binutils 2.27.0.20170315) 1.12
clang70++: error: linker command failed with exit code 1 (use -v to see invocation)

btw: Not sure if Swift officially support Android compilation on macOS Cross-Compile host. I am using this Automated workflow to build Swift for Android on macOS: GitHub - vgorloff/swift-everywhere-toolchain: Automated workflow to compile Swift Toolchain, for making Android apps with Swift.

compnerd · 2019-01-02T22:20:48.096Z

The swift_addDSOImage is used for image registration with the swift runtime. swiftrt.o is the image registrar and is embedded into the final image to ensure that the image is registered with the runtime upon load. It sounds like you are not linking against the right runtime?

v.gorlov · 2019-01-02T22:53:50.907Z

It sounds like you are not linking against the right runtime?

Is it means that I have to explicitly specify some libraries (via -l arguments) from NDK for Darwin. Or swiftrt.o, libswiftCore.so, libswiftSwiftOnoneSupport.so, ... compiled with wrong settings. Or maybe Swift compiler itself compiled with wrong settings?

Here is what I have when inspecting files:

$ file /EDITED/armv7a-macos/swift/usr/bin/swift 
/EDITED/armv7a-macos/swift/usr/bin/swift: Mach-O 64-bit executable x86_64

$ file /EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o
/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o: ELF 32-bit LSB relocatable, ARM, EABI5 version 1 (SYSV), not stripped

$ file /EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so
/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so: ELF 32-bit LSB shared object ARM, EABI5 version 1 (SYSV), dynamically linked, with debug_info, not stripped

$ file /EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftSwiftOnoneSupport.so
/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftSwiftOnoneSupport.so: ELF 32-bit LSB shared object ARM, EABI5 version 1 (SYSV), dynamically linked, with debug_info, not stripped

SO-files definitely dynamically linked and seems for correct architecture. swiftrt.o – ELF 32-bit LSB relocatable. But i don't really understand what is relocatable means

The only significant arguments I am passing to swiftc seems:

-tools-directory /.../armv7a-macos/ndk/bin – path to Android NDK for Darwin.
-target armv7-none-linux-androideabi – target triple which informs that result executable should be targeted Android (i.e. ELF 32-bit LSB pie executable ARM, EABI5 ...)

Also few flags passed to avoid clang errors "due wrong/missed headers".

-Xcc -DDEPLOYMENT_TARGET_ANDROID
-Xcc -DDEPLOYMENT_RUNTIME_SWIFT

My guess was that clang++ and ld.gold from NDK doing rest of work. At least same swiftc invocation works well under Ubuntu (Vagrant).

Here is a command line used:

PATH=/EDITED/armv7a-macos/swift/usr/bin:$PATH swiftc -v \
-tools-directory /EDITED/armv7a-macos/ndk/bin \
-target armv7-none-linux-androideabi \
-Xcc -I/EDITED/armv7a-macos/ndk/sysroot/usr/include \
-Xcc -DDEPLOYMENT_TARGET_ANDROID \
-Xcc -DDEPLOYMENT_RUNTIME_SWIFT \
-Xlinker -v /EDITED/Projects/Hello/hello.swift

compnerd · 2019-01-02T22:55:32.220Z

Uh, I don't see a -use-ld=gold or -use-ld=lld there. That definitely seems like you're not using the right tools.

v.gorlov · 2019-01-03T22:04:21.183Z

Thank you!

I isolated issue but can't understand why it is working in one case and not working in other case :0

So, first we can lookup symbols to ensure that swift_addNewDSOImage symbol really exists (in libswiftCore.so.):

nm -a swiftrt.o
         U 
00000000 t 
00000000 t $a.0
00000130 t $d.1
00000000 a SwiftRT-ELF.cpp
00000000 t _ZL23swift_image_constructorv
00000000 b _ZN12_GLOBAL__N_18sectionsE
         U __start_swift5_assocty
         U __start_swift5_fieldmd
         U __start_swift5_protocol_conformances
         U __start_swift5_protocols
         U __start_swift5_reflstr
         U __start_swift5_replace
         U __start_swift5_type_metadata
         U __start_swift5_typeref
         U __stop_swift5_assocty
         U __stop_swift5_fieldmd
         U __stop_swift5_protocol_conformances
         U __stop_swift5_protocols
         U __stop_swift5_reflstr
         U __stop_swift5_replace
         U __stop_swift5_type_metadata
         U __stop_swift5_typeref
         U swift_addNewDSOImage

nm -a libswiftSwiftOnoneSupport.so
         U swift_addNewDSOImage

nm -a libswiftCore.so
003c01a4 T swift_addNewDSOImage

Then we can compile Swift source code:

/EDITED/armv7a-macos/swift/usr/bin/swift -frontend -c \
-primary-file /EDITED/Temp/hello.swift \
-target armv7-none-linux-android -disable-objc-interop \
-Xcc -I/EDITED/armv7a-macos/ndk/sysroot/usr/include \
-Xcc -DDEPLOYMENT_TARGET_ANDROID -Xcc -DDEPLOYMENT_RUNTIME_SWIFT \
-color-diagnostics -module-name hello \
-o /EDITED/Temp/hello.o

file /EDITED/Temp/hello.o

# Prints
/EDITED/Temp/hello.o: ELF 32-bit LSB relocatable, ARM, EABI5 version 1 (SYSV), not stripped

Then we can run swift-autolink-extract tool.

/EDITED/armv7a-macos/swift/usr/bin/swift-autolink-extract \
/EDITED/Temp/hello.o \
-o /EDITED/Temp/hello.autolink

cat /EDITED/Temp/hello.autolink

# Prints
-lswiftSwiftOnoneSupport
-lswiftCore

Then we can launch linker.

/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld.gold \
--sysroot=/EDITED/armv7a-macos/ndk/bin/../sysroot \
-pie -X --enable-new-dtags --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /system/bin/linker \
-o /EDITED/Temp/hello \
/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtbegin_dynamic.o \
-L/EDITED/armv7a-macos/ndk/lib64/clang/7.0.2/lib/linux/arm \
-L/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/armv7-a \
-L/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/../lib/armv7-a \
-L/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib \
-L/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/armv7-a \
-L/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib \
/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o \
/EDITED/Temp/hello.o \
-L/EDITED/armv7a-macos/swift/usr/lib/swift/android \
-lswiftSwiftOnoneSupport -lswiftCore \
-v -lstdc++ -lm -lgcc -lgcc -ldl -lc -lgcc -lgcc -ldl \
/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtend_android.o

# As result error:
/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o:SwiftRT-ELF.cpp:function swift_image_constructor(): error: undefined reference to 'swift_addNewDSOImage'

But if we specifying full path to SO-files, then liker works well.

/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld.gold \
--sysroot=/EDITED/armv7a-macos/ndk/bin/../sysroot \
-pie -X --enable-new-dtags --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /system/bin/linker \
-o /EDITED/Temp/hello \
/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtbegin_dynamic.o \
-L/EDITED/armv7a-macos/ndk/lib64/clang/7.0.2/lib/linux/arm \
-L/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/armv7-a \
-L/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/../lib/armv7-a \
-L/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib \
-L/EDITED/armv7a-macos/ndk/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/armv7-a \
-L/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib \
/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o \
/EDITED/Temp/hello.o \
/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so \
/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftSwiftOnoneSupport.so \
-v -lstdc++ -lm -lgcc -lgcc -ldl -lc -lgcc -lgcc -ldl \
/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtend_android.o

file /EDITED/Temp/hello

# Prints
/EDITED/Temp/hello: ELF 32-bit LSB pie executable ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /system/bin/linker, with debug_info, not stripped

Difference in two linker invocations is following:

--- untitled
+++ (clipboard)
@@ -11,7 +11,7 @@
 -L/EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib \
 /EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o \
 /EDITED/Temp/hello.o \
-/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so \
-/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftSwiftOnoneSupport.so \
+-L/EDITED/armv7a-macos/swift/usr/lib/swift/android \
+-lswiftSwiftOnoneSupport -lswiftCore \
 -v -lstdc++ -lm -lgcc -lgcc -ldl -lc -lgcc -lgcc -ldl \
 /EDITED/armv7a-macos/ndk/bin/../sysroot/usr/lib/../lib/crtend_android.o

What can be the reason why ld.gold linker from Android NDK can't find symbols if Shared Objects specified not with a full path?

UPDATE: Linker shows same error even if options -lswiftSwiftOnoneSupport -lswiftCore not passed at all. Looks like linker not really using Shared Objects specified with -l option.

UPDATE 2: Order of passing libraries matters.

# Works
cd /EDITED/armv7a/Hello && /EDITED/armv7a-macos/ndk/bin/clang++ -fuse-ld=gold \
-B /EDITED/armv7a-macos/ndk/bin -pie -target armv7-none-linux-androideabi \
/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o \
/EDITED/armv7a/Hello/hello-main.o \
-l/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so \
-l/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftSwiftOnoneSupport.so \
--target=armv7-none-linux-android \
-o /EDITED/armv7a/Hello/hello

# Not working
cd /EDITED/armv7a/Hello && /EDITED/armv7a-macos/ndk/bin/clang++ -fuse-ld=gold \
-B /EDITED/armv7a-macos/ndk/bin -pie -target armv7-none-linux-androideabi \
/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o \
/EDITED/armv7a/Hello/hello-main.o \
-l/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftSwiftOnoneSupport.so \
-l/EDITED/armv7a-macos/swift/usr/lib/swift/android/libswiftCore.so \
--target=armv7-none-linux-android \
-o /EDITED/armv7a/Hello/hello

/EDITED/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o:SwiftRT-ELF.cpp:function swift_image_constructor(): error: undefined reference to 'swift_addNewDSOImage'
clang70++: error: linker command failed with exit code 1 (use -v to see invocation)

Super strange why libswiftCore.so needs to be passed before libswiftSwiftOnoneSupport.so.

v.gorlov · 2019-03-11T20:47:50.682Z

UPDATE 3:

After updating to Swift 5 and NDK 19 the error is different: error: undefined reference to '_swift_FORCE_LOAD_$_swiftGlibc'

hello-main.o:hello-main.o:function main: error: undefined reference to '$ss27_allocateUninitializedArrayySayxG_BptBwlFyp_Tg5'
hello-main.o:hello-main.o:function main: error: undefined reference to '$ss27_allocateUninitializedArrayySayxG_BptBwlFyp_Tg5'
hello-main.o:hello-main.o:function main: error: undefined reference to '$ss27_allocateUninitializedArrayySayxG_BptBwlFyp_Tg5'
hello-main.o:hello-main.o:function main: error: undefined reference to '$ss27_allocateUninitializedArrayySayxG_BptBwlFyp_Tg5'
hello-main.o:hello-main.o:$s5hello6PersonL_V10CodingKeysON: error: undefined reference to '_swift_FORCE_LOAD_$_swiftGlibc'

compnerd · 2019-03-11T21:06:53.019Z

Sounds like you aren't linking against swiftGlibc. With auto-link-extract, -lswiftGlibc should be added implicitly, I would suggest that you figure out what the actual linker invocation looks like and why -lswiftGlibc is missing.

v.gorlov · 2019-03-11T21:46:21.802Z

The actual liker invocation looks like below.
Swift auto-linker not used (due another link problems). -lswiftGlibc flag is present.

cd /android-on-swift/Build/armv7a/Hello && /android-on-swift/Install/armv7a-macos/swift/usr/bin/swift -frontend -c \
-primary-file /android-on-swift/Projects/Hello/hello.swift \
-target armv7-none-linux-android -disable-objc-interop -color-diagnostics \
-module-name hello -o /android-on-swift/Build/armv7a/Hello/hello-main.o \
-Xcc -I/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include \
-I/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/arm-linux-androideabi \
-Xcc -DDEPLOYMENT_TARGET_ANDROID -Xcc -DDEPLOYMENT_TARGET_LINUX -Xcc -DDEPLOYMENT_RUNTIME_SWIFT

### Checking that Swift compiled sample source code file.
file /android-on-swift/Build/armv7a/Hello/hello-main.o
/android-on-swift/Build/armv7a/Hello/hello-main.o: ELF 32-bit LSB relocatable, ARM, EABI5 version 1 (SYSV), not stripped

cd /android-on-swift/Build/armv7a/Hello && /android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi21-clang \
-fuse-ld=gold -pie -v \
-B /android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin \
/android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o \
/android-on-swift/Build/armv7a/Hello/hello-main.o \
-L /android-on-swift/Build/armv7a/Hello -lswiftCore -lswiftGlibc -lswiftSwiftOnoneSupport -lswiftDispatch -lBlocksRuntime -lc++_shared -lFoundation \
-L /android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x \
-o /android-on-swift/Build/armv7a/Hello/hello

Android (5058415 based on r339409) clang version 8.0.2 (https://android.googlesource.com/toolchain/clang 40173bab62ec746213857d083c0e8b0abb568790) (https://android.googlesource.com/toolchain/llvm 7a6618d69e7e8111e1d49dc9e7813767c5ca756a) (based on LLVM 8.0.2svn)
Target: armv7a-unknown-linux-android21
Thread model: posix
InstalledDir: /android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin
Found candidate GCC installation: /android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x
Selected GCC installation: /android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x
Candidate multilib: thumb;@mthumb
Candidate multilib: armv7-a;@march=armv7-a
Candidate multilib: armv7-a/thumb;@march=armv7-a@mthumb
Candidate multilib: .;
Selected multilib: armv7-a;@march=armv7-a

 "/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld.gold" \
 -pie -X --enable-new-dtags --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /system/bin/linker \
 -o /android-on-swift/Build/armv7a/Hello/hello \
 /android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../sysroot/usr/lib/arm-linux-androideabi/21/crtbegin_dynamic.o \
 -L/android-on-swift/Build/armv7a/Hello \
 -L/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x \
 -L/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/lib64/clang/8.0.2/lib/linux/arm \
 -L/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/armv7-a \
 -L/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/../lib/armv7-a \
 -L/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../sysroot/usr/lib/arm-linux-androideabi/21 \
 -L/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../sysroot/usr/lib/arm-linux-androideabi \
 -L/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../sysroot/usr/lib/../lib \
 -L/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../sysroot/usr/lib/arm-linux-androideabi/../../lib \
 -L/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/lib/armv7-a \
 -L/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../sysroot/usr/lib /android-on-swift/Install/armv7a-macos/swift/usr/lib/swift/android/armv7/swiftrt.o \
 /android-on-swift/Build/armv7a/Hello/hello-main.o -lswiftCore -lswiftGlibc -lswiftSwiftOnoneSupport -lswiftDispatch -lBlocksRuntime -lc++_shared -lFoundation \
 -lgcc -ldl -lc -lgcc -ldl \
 /android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/../sysroot/usr/lib/arm-linux-androideabi/21/crtend_android.o
 
/android-on-swift/Build/armv7a/Hello/hello-main.o:hello-main.o:function main: error: undefined reference to '$ss27_allocateUninitializedArrayySayxG_BptBwlFyp_Tg5'
/android-on-swift/Build/armv7a/Hello/hello-main.o:hello-main.o:function main: error: undefined reference to '$ss27_allocateUninitializedArrayySayxG_BptBwlFyp_Tg5'
/android-on-swift/Build/armv7a/Hello/hello-main.o:hello-main.o:function main: error: undefined reference to '$ss27_allocateUninitializedArrayySayxG_BptBwlFyp_Tg5'
/android-on-swift/Build/armv7a/Hello/hello-main.o:hello-main.o:function main: error: undefined reference to '$ss27_allocateUninitializedArrayySayxG_BptBwlFyp_Tg5'
/android-on-swift/Build/armv7a/Hello/hello-main.o:hello-main.o:$s5hello6PersonL_V10CodingKeysON: error: undefined reference to '_swift_FORCE_LOAD_$_swiftGlibc'
clang: error: linker command failed with exit code 1 (use -v to see invocation)

v.gorlov · 2019-03-11T23:25:44.113Z

Picking and replacing files libswiftGlibc.so and libswiftCore.so from same toolchain build on Linux fixes link errors. Seems build settings for Cross Compilation for that two libraries incorrect. Other libraries (like libswiftSwiftOnoneSupport.so, libswiftDispatch.so, libFoundation.so, libdispatch.so and libBlocksRuntime.so) seems built correctly.

Here is minimum required linker options:

/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/arm-linux-androideabi/bin/ld.gold \
-o /android-on-swift/Build/armv7a/Hello/hello \
/android-on-swift/Build/armv7a/Hello/hello-main.o \
-L /android-on-swift/Build/armv7a/Hello \
-lswiftCore -lswiftGlibc -lswiftSwiftOnoneSupport -lswiftDispatch -lBlocksRuntime -lFoundation \
-L /android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/arm-linux-androideabi/21 \
-lc

v.gorlov · 2019-03-11T23:56:59.666Z

Seems issue in wrong soname due autogenerated rules.ninja by Cmake.

#############################################
# Rule for linking CXX shared library.

rule CXX_SHARED_LIBRARY_LINKER__swiftGlibc-android-armv7
  command = $PRE_LINK && /android-on-swift/Build/armv7a-macos/llvm/./bin/clang++ $LANGUAGE_COMPILE_FLAGS $ARCH_FLAGS -shared -Wl,-headerpad_max_install_names $LINK_FLAGS -o $TARGET_FILE  $in $LINK_PATH $LINK_LIBRARIES && $POST_BUILD
  description = Linking CXX shared library $TARGET_FILE
  restat = $RESTAT

So, we passing linker option -Wl with value -headerpad_max_install_names

After that ld.gold resolves unknown option -headerpad_max_install_names as below:

  -h FILENAME, -soname FILENAME
                              Set shared library name

Literally soname became eaderpad_max_install_names :0

After reading ELF data readelf --all Build/armv7a/Hello/libswiftCore.so we can see following:

Dynamic section at offset 0x46aed4 contains 32 entries:
  Tag        Type                         Name/Value
...
 0x00000004 (HASH)                       0xac13c
 0x00000001 (NEEDED)                     Shared library: [libm.so]
 0x00000001 (NEEDED)                     Shared library: [libdl.so]
 0x00000001 (NEEDED)                     Shared library: [liblog.so]
 0x00000001 (NEEDED)                     Shared library: [libicudataswift.so.64]
 0x00000001 (NEEDED)                     Shared library: [libicui18nswift.so.64]
 0x00000001 (NEEDED)                     Shared library: [libicuucswift.so.64]
 0x00000001 (NEEDED)                     Shared library: [libc++_shared.so]
 0x00000001 (NEEDED)                     Shared library: [libc.so]
 0x0000000e (SONAME)                     Library soname: [eaderpad_max_install_names]
 0x0000001a (FINI_ARRAY)                 0x461460

While ELF data of library build on Linux as below:

Dynamic section at offset 0x46aed4 contains 32 entries:
  Tag        Type                         Name/Value
...
 0x00000004 (HASH)                       0xac130
 0x00000001 (NEEDED)                     Shared library: [libm.so]
 0x00000001 (NEEDED)                     Shared library: [libdl.so]
 0x00000001 (NEEDED)                     Shared library: [liblog.so]
 0x00000001 (NEEDED)                     Shared library: [libicudataswift.so.64]
 0x00000001 (NEEDED)                     Shared library: [libicui18nswift.so.64]
 0x00000001 (NEEDED)                     Shared library: [libicuucswift.so.64]
 0x00000001 (NEEDED)                     Shared library: [libc++_shared.so]
 0x00000001 (NEEDED)                     Shared library: [libc.so]
 0x0000000e (SONAME)                     Library soname: [libswiftCore.so]
 0x0000001a (FINI_ARRAY)                 0x461460

jrose · 2019-03-12T00:35:27.089Z

It looks like CMake is adding -headerpad_max_install_names because we don't use CMake's cross-compilation support to build the standard library. You can probably manually hack it out of CMAKE_CXX_LINK_FLAGS where it's being added.

v.gorlov · 2019-03-12T01:51:28.352Z

Thanks!
Hacking file rules.ninja does the job.

line = line.gsub('$SONAME_FLAG $INSTALLNAME_DIR$SONAME', '-Wl,-soname,$SONAME')
line = line.gsub('-Wl,-headerpad_max_install_names', '')

Strange that for instance in swift-corelibs-libdispatch repository rules.ninja don't require hacking. Correct rules generated by cmake out of box.

#############################################
# Rule for linking C shared library.

rule C_SHARED_LIBRARY_LINKER__BlocksRuntime
  command = $PRE_LINK && /android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang --target=armv7-none-linux-androideabi21 --gcc-toolchain=/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64 --sysroot=/android-on-swift/Sources/ndk-macos-r19b/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -fPIC $LANGUAGE_COMPILE_FLAGS $ARCH_FLAGS $LINK_FLAGS -shared $SONAME_FLAG$SONAME -o $TARGET_FILE $in $LINK_PATH $LINK_LIBRARIES && $POST_BUILD
  description = Linking C shared library $TARGET_FILE
  restat = $RESTAT

So, non-official CrossCompile toolchain on macOS Host, which builds Swift (including Foundation and Dispatch libraries) for Android platform seems works.

Sample run:

adb shell ls -l /data/local/tmp/hello
total 44792
-rwxrwxrwx 1 shell shell    60504 2019-03-12 02:35 hello
-rwxrwxrwx 1 shell shell   117492 2019-03-12 02:35 libBlocksRuntime.so
-rw-rw-rw- 1 shell shell 14697312 2019-03-12 02:35 libFoundation.so
-rwxrwxrwx 1 shell shell  4842992 2019-03-12 02:35 libc++_shared.so
-rwxrwxrwx 1 shell shell  2619056 2019-03-12 02:35 libcrypto.so.1.1
-rwxrwxrwx 1 shell shell   500140 2019-03-12 02:35 libcurl.so
-rwxrwxrwx 1 shell shell  1268968 2019-03-12 02:35 libdispatch.so
-rwxrwxrwx 1 shell shell     5256 2019-03-12 02:35 libicudataswift.so.64
-rwxrwxrwx 1 shell shell  4522788 2019-03-12 02:35 libicui18nswift.so.64
-rwxrwxrwx 1 shell shell  2739748 2019-03-12 02:35 libicuucswift.so.64
-rwxrwxrwx 1 shell shell   587076 2019-03-12 02:35 libssl.so.1.1
-rw-rw-rw- 1 shell shell  7715976 2019-03-12 02:35 libswiftCore.so
-rw-rw-rw- 1 shell shell   402012 2019-03-12 02:35 libswiftDispatch.so
-rw-rw-rw- 1 shell shell   691028 2019-03-12 02:35 libswiftGlibc.so
-rw-rw-rw- 1 shell shell  1193304 2019-03-12 02:35 libswiftRemoteMirror.so
-rw-rw-rw- 1 shell shell   652704 2019-03-12 02:35 libswiftSIMDOperators.so
-rw-rw-rw- 1 shell shell  1035376 2019-03-12 02:35 libswiftSwiftOnoneSupport.so
-rwxrwxrwx 1 shell shell  2140764 2019-03-12 02:35 libxml2.so

Starting execution of "/data/local/tmp/hello/hello"...
adb shell LD_LIBRARY_PATH=/data/local/tmp/hello /data/local/tmp/hello/hello
SA - SwiftCore: Works!
SA - DispatchQueue: Works!
SA - BlockOperation: Works!
SA - JSONSerialization/JSONDecoder: Works!
SA - URLSession: Currently disabled. Will fail with `Segmentation fault`. Seems something in Foundation classes needs to be fixed.

Execution of "/data/local/tmp/hello/hello" completed.

v.gorlov · 2019-05-09T00:23:42.066Z

Toolchain finally reached alpha state. It is supports arm, aarch64, x86 architectures. Can be downloaded as pre-build package and used alongside other build tools on macOS system (cmake, Android Studio, etc.). x86_64 architecture support comes later.

I collected ~30 patches which is needed to successfully build toolchain on macOS. Hope that upcoming Pull Requests will be processed quickly.

compnerd · 2019-05-09T14:59:44.015Z

Cool! That is pretty interesting.

For the CMake support, what did you do? I have a pretty extensive change for CMake that I am working on to support Swift properly. It adds support to build with CMake + Ninja rather than just Xcode.

What are the items in the patch set? I am curious as I have been working on building the android runtime (standalone) on Windows and have gotten to the point where only a single thing is holding it from actual production - the need to override the C++ compiler with the android toolchain.

v.gorlov · 2019-05-09T17:33:41.864Z

All patches can be found here: swift-everywhere-toolchain/Patches at develop · vgorloff/swift-everywhere-toolchain · GitHub

Additionally to patches several hacks were made such as search/replace in build.ninja file(s). But those hacks will be removed soon since I am going to find proper place in Cmake files where compatibility changes need to be applied.

Typical Cmake patches I made:

cmake/modules/SwiftConfigureSDK.cmake - Added i686 support for Android.
cmake/modules/SwiftAndroidSupport.cmake - Same as above.
cmake/modules/AddSwift.cmake - Same as above.
stdlib/private/CMakeLists.txt - Disabled Swift overlays targets as they don't make sense on Android.
stdlib/public/stubs/CMakeLists.txt - Same as above.
stdlib/CMakeLists.txt - Disabled tests if test not enabled by SWIFT_INCLUDE_TESTS which seems just historical issue.

Hacks (search/replace in build.ninja file(s)):

Replace -dynamiclib with -shared
Replace $SONAME_FLAG $INSTALLNAME_DIR$SONAME with -Wl,-soname,$SONAME
Many more :)

C++ compiler for Dispatch, Foundation builds set by using CMAKE_TOOLCHAIN_FILE file.

# See why we need to use cmake toolchain in NDK v19 - https://gitlab.kitware.com/cmake/cmake/issues/18739
"-DCMAKE_TOOLCHAIN_FILE=#{@ndk.sources}/build/cmake/android.toolchain.cmake"

In dependencies C++ compiler configured like below:

LDFLAGS='-march=armv7-a'
CC=#{@ndk.toolchain}/bin/armv7a-linux-androideabi#{@ndk.api}-clang
CXX=#{@ndk.toolchain}/bin/armv7a-linux-androideabi#{@ndk.api}-clang++
AR=#{@ndk.toolchain}/bin/arm-linux-androideabi-ar
RINLIB=#{@ndk.toolchain}/bin/arm-linux-androideabi-ranlib

In Swift there is also hack applied for AR:

gsub('/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar', "#{@ndk.toolchain}/bin/arm-linux-androideabi-ar")

LLVM and Clang which we building manually used only to build Swift itself. Neither dependencies (icu, XML, ...) nor Dispatch/Foundation using that manually built Clang. Clang which comes with NDK is used for dependencies and libs.

I am going to enable x86_64 support for Android Simulators and start submitting Pull requests with compatibility fixes next week.

compnerd · 2019-05-09T17:36:29.409Z

The AR issue should be solvable by using CMAKE_AR. The CMake toolchain issue is what I'm hitting on the Windows -> android cross-compile as well.