Call for help: Linux (and others) build configurations, including static libraries, with ICU.


(Michael Ilseman) #1

With https://github.com/apple/swift/pull/9684 the Swift standard library depends on ICU on Darwin in addition to Linux, where it has always had that dependency. While our Linux bots have been happy with the changes, I’m not familiar with the build configurations involving the build-your-own-ICU path, nor with FreeBSD, Cygwin, Android, etc. I would like to make sure that all supported configurations still work, can someone help me with this?

Additionally, the standard library can be built as a static library. In this configuration, user programs that link against the static library should also be told explicitly to link against ICU (e.g. “-licu*”). On Darwin, the least evil approach was autolinking by emitting a linker option via inline asm in the shims library. Is there an approach that would work for Linux? An alternative solution could be adding the flags in the Swift driver itself, but that would mean that any build systems that bypass the Swift driver would also need to be updated to pass those same flags.


(Davide C. C. Italiano) #2

It's been quite a while but if I remember correctly when I first did
the port I largely mimicked what Linux was doing, and `icu` was a
dependency (at least I try to remember installing it). I don't have a
FreeBSD machine handy, but I'll consider setting one up in the weekend
to see if anything broke.

Feel free to ping me if I forget.

Thanks,

···

On Thu, May 25, 2017 at 5:33 PM, Michael Ilseman <milseman@apple.com> wrote:

With https://github.com/apple/swift/pull/9684 the Swift standard library depends on ICU on Darwin in addition to Linux, where it has always had that dependency. While our Linux bots have been happy with the changes, I’m not familiar with the build configurations involving the build-your-own-ICU path, nor with FreeBSD, Cygwin, Android, etc. I would like to make sure that all supported configurations still work, can someone help me with this?

--
Davide


(Simon Evans) #3

For building ICU (on Linux at least) you just need the icu directory at the same level as the other swift source directories and then the `—libicu` option should build it and the `—install-libicu` option should install the libs in usr/lib/swift/linux/ and usr/lib/swift_static/linux/

Static stdlib linking on ELF platforms is controlled by the file
$ cat ~/swift/usr/lib/swift_static/linux/static-stdlib-args.lnk
-ldl
-lpthread
-latomic
-lswiftCore
-latomic
-lswiftImageInspectionShared
/usr/lib/x86_64-linux-gnu/libicui18n.a
/usr/lib/x86_64-linux-gnu/libicuuc.a
/usr/lib/x86_64-linux-gnu/libicudata.a
-Xlinker
-export-dynamic
-Xlinker
--exclude-libs
-Xlinker
ALL

This file is created at build time by the script utils/gen-static-stdlib-link-args which tries to work out what libicu paths to use. For the above file it is using the .a files to link as part of `-static-stdlib`. If building ICU as part of the build process, instead of the .a files it should list

-licui18n
-licuuc
-licudata

Since the compile automatically points to the libs in swift_static. I think the generator script might need to be modified for FreeBSD as I dont think -ldl is valid on FreeBSD.

That script is called by lib/Driver/CMakeLists.txt. Although it should work on any ELF platform it does have a check to only allow it to run on Linux, this was because I didnt have any other platform to test against and didnt want to break it for other ELF platforms, so you may need to remove that.

One other thing to check when compiling swift-corelibs-foundation on Linux is you may need the change in this unmerged PR https://github.com/apple/swift-corelibs-foundation/pull/722 which uses the correct header files for ICU if compiling ICU from source using pkgconfig - I dont know what the non-Linux equivalent would be.

Hope that helps

si

···

On 26 May 2017, at 01:33, Michael Ilseman <milseman@apple.com> wrote:

With https://github.com/apple/swift/pull/9684 the Swift standard library depends on ICU on Darwin in addition to Linux, where it has always had that dependency. While our Linux bots have been happy with the changes, I’m not familiar with the build configurations involving the build-your-own-ICU path, nor with FreeBSD, Cygwin, Android, etc. I would like to make sure that all supported configurations still work, can someone help me with this?

Additionally, the standard library can be built as a static library. In this configuration, user programs that link against the static library should also be told explicitly to link against ICU (e.g. “-licu*”). On Darwin, the least evil approach was autolinking by emitting a linker option via inline asm in the shims library. Is there an approach that would work for Linux? An alternative solution could be adding the flags in the Swift driver itself, but that would mean that any build systems that bypass the Swift driver would also need to be updated to pass those same flags.


(Davide C. C. Italiano) #4

I tried this one but I'm afraid the support for FreeBSD bitrot enough
that I didn't even get to the point where this failed.
I have a crash in the compiler/verifier instead.
Please let me know if you need other infos.

[27/6/50] Compiling
/usr/home/davide/mount/.../SwiftPrivate/freebsd/x86_64/SwiftPrivate.o
FAILED: stdlib/private/SwiftPrivate/freebsd/x86_64/SwiftPrivate.o
cd /usr/home/davide/mount/dcci/work/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/stdlib/private/SwiftPrivate
&& /usr/local/bin/python /usr/home/davide/mount/dcci/work/swif$
/utils/line-directive
@/usr/home/davide/mount/dcci/work/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/stdlib/private/SwiftPrivate/lKFOW.txt
-- /usr/home/davide/mount/dcci/w$
rk/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/./bin/swiftc
-c -sdk / -targetx86_64-unknown-freebsd11.0-RELEASE-p2 -resource-dir
/usr/home/davide/mount/dcci/work/build/Ni$
ja-RelWithDebInfoAssert/swift-freebsd-x86_64/./lib/swift -O -g -D
INTERNAL_CHECKS_ENABLED -I
/usr/home/davide/mount/dcci/work/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/$
/lib/swift/freebsd/x86_64 -module-cache-path
/usr/home/davide/mount/dcci/work/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/./module-cache
-no-link-objc-runtime -Xfrontend $
enable-cow-existentials -swift-version 3 -Xfrontend -sil-serialize-all
-module-link-name swiftSwiftPrivate -force-single-frontend-invocation
-parse-as-library -o /usr/home/davide/m$
unt/dcci/work/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/stdlib/private/SwiftPrivate/freebsd/x86_64/SwiftPrivate.o
@/usr/home/davide/mount/dcci/work/build/Ninja-RelWithD$
bInfoAssert/swift-freebsd-x86_64/stdlib/private/SwiftPrivate/lKFOW.txt
/usr/home/davide/mount/dcci/work/swift/stdlib/private/SwiftPrivate/SwiftPrivate.swift:50:12:
warning: simultaneous accesses to var 'result', but modification
requires exclusive acc$
ss; consider copying to a local variable
      swap(&result[i], &result[j])
      ~~~~~^~~~~~~~~~~~~~~~~~~~~~~
      result.swapAt(i, j)
/usr/home/davide/mount/dcci/work/swift/stdlib/private/SwiftPrivate/SwiftPrivate.swift:50:24:
note: conflicting access is here
      swap(&result[i], &result[j])
                       ^~~~~~~~~~
swifterror value when used in a callsite should be marked with
swifterror attribute
  %6 = alloca swifterror %swift.error*, align 8
  %61 = call i1
@_T0s8SequencePsE8containsS2b7ElementQzKc5where_tKFSRys6UInt16VG_Tgq505_T0s6E37VSbs5Error_pIxydzo_ABSbsAC_pIxidzo_TRAHSbs0H0_pIxydzo_Tfq1cn_nTfq4ng_n(%TSRys6UInt16VG
* noalias nocapture dereferenceable(16) %5, i8* bitcast (i1 (i16)*
@_T0s11_StringCoreV22isRepresentableAsASCIISbyFSbs6UInt16VcfU_ to
i8*), %swift.refcounted* null, %swift.refcounted
* undef, %swift.error** nocapture %6), !dbg !1877
<unknown>:0: error: fatal error encountered during compilation; please
file a bug report with your project and the crash log
<unknown>:0: note: Broken function found, compilation aborted!

···

On Thu, May 25, 2017 at 5:33 PM, Michael Ilseman <milseman@apple.com> wrote:

With https://github.com/apple/swift/pull/9684 the Swift standard library depends on ICU on Darwin in addition to Linux, where it has always had that dependency. While our Linux bots have been happy with the changes, I’m not familiar with the build configurations involving the build-your-own-ICU path, nor with FreeBSD, Cygwin, Android, etc. I would like to make sure that all supported configurations still work, can someone help me with this?

Additionally, the standard library can be built as a static library. In this configuration, user programs that link against the static library should also be told explicitly to link against ICU (e.g. “-licu*”). On Darwin, the least evil approach was autolinking by emitting a linker option via inline asm in the shims library. Is there an approach that would work for Linux? An alternative solution could be adding the flags in the Swift driver itself, but that would mean that any build systems that bypass the Swift driver would also need to be updated to pass those same flags.


(Michael Ilseman) #5

That would be awesome! Do you use the static stdlib configuration?

I had to make assumptions about the type of pthread_key_t. If the type is wrong, there’s a nice static_assert with a message telling how to fix it. This came up yesterday for Android builds, and fixes are pretty straight forward: https://github.com/apple/swift/pull/9930/files

···

On May 25, 2017, at 5:36 PM, Davide Italiano <dccitaliano@gmail.com> wrote:

On Thu, May 25, 2017 at 5:33 PM, Michael Ilseman <milseman@apple.com> wrote:

With https://github.com/apple/swift/pull/9684 the Swift standard library depends on ICU on Darwin in addition to Linux, where it has always had that dependency. While our Linux bots have been happy with the changes, I’m not familiar with the build configurations involving the build-your-own-ICU path, nor with FreeBSD, Cygwin, Android, etc. I would like to make sure that all supported configurations still work, can someone help me with this?

It's been quite a while but if I remember correctly when I first did
the port I largely mimicked what Linux was doing, and `icu` was a
dependency (at least I try to remember installing it). I don't have a
FreeBSD machine handy, but I'll consider setting one up in the weekend
to see if anything broke.

Feel free to ping me if I forget.

Thanks,

--
Davide


(Michael Ilseman) #6

Wow, please file the bug report. Thank you for trying it out, at least!

···

On May 29, 2017, at 5:16 PM, Davide Italiano <dccitaliano@gmail.com> wrote:

On Thu, May 25, 2017 at 5:33 PM, Michael Ilseman <milseman@apple.com> wrote:
With https://github.com/apple/swift/pull/9684 the Swift standard library depends on ICU on Darwin in addition to Linux, where it has always had that dependency. While our Linux bots have been happy with the changes, I’m not familiar with the build configurations involving the build-your-own-ICU path, nor with FreeBSD, Cygwin, Android, etc. I would like to make sure that all supported configurations still work, can someone help me with this?

Additionally, the standard library can be built as a static library. In this configuration, user programs that link against the static library should also be told explicitly to link against ICU (e.g. “-licu*”). On Darwin, the least evil approach was autolinking by emitting a linker option via inline asm in the shims library. Is there an approach that would work for Linux? An alternative solution could be adding the flags in the Swift driver itself, but that would mean that any build systems that bypass the Swift driver would also need to be updated to pass those same flags.

I tried this one but I'm afraid the support for FreeBSD bitrot enough
that I didn't even get to the point where this failed.
I have a crash in the compiler/verifier instead.
Please let me know if you need other infos.

[27/6/50] Compiling
/usr/home/davide/mount/.../SwiftPrivate/freebsd/x86_64/SwiftPrivate.o
FAILED: stdlib/private/SwiftPrivate/freebsd/x86_64/SwiftPrivate.o
cd /usr/home/davide/mount/dcci/work/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/stdlib/private/SwiftPrivate
&& /usr/local/bin/python /usr/home/davide/mount/dcci/work/swif$
/utils/line-directive
@/usr/home/davide/mount/dcci/work/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/stdlib/private/SwiftPrivate/lKFOW.txt
-- /usr/home/davide/mount/dcci/w$
rk/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/./bin/swiftc
-c -sdk / -targetx86_64-unknown-freebsd11.0-RELEASE-p2 -resource-dir
/usr/home/davide/mount/dcci/work/build/Ni$
ja-RelWithDebInfoAssert/swift-freebsd-x86_64/./lib/swift -O -g -D
INTERNAL_CHECKS_ENABLED -I
/usr/home/davide/mount/dcci/work/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/$
/lib/swift/freebsd/x86_64 -module-cache-path
/usr/home/davide/mount/dcci/work/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/./module-cache
-no-link-objc-runtime -Xfrontend $
enable-cow-existentials -swift-version 3 -Xfrontend -sil-serialize-all
-module-link-name swiftSwiftPrivate -force-single-frontend-invocation
-parse-as-library -o /usr/home/davide/m$
unt/dcci/work/build/Ninja-RelWithDebInfoAssert/swift-freebsd-x86_64/stdlib/private/SwiftPrivate/freebsd/x86_64/SwiftPrivate.o
@/usr/home/davide/mount/dcci/work/build/Ninja-RelWithD$
bInfoAssert/swift-freebsd-x86_64/stdlib/private/SwiftPrivate/lKFOW.txt
/usr/home/davide/mount/dcci/work/swift/stdlib/private/SwiftPrivate/SwiftPrivate.swift:50:12:
warning: simultaneous accesses to var 'result', but modification
requires exclusive acc$
ss; consider copying to a local variable
     swap(&result[i], &result[j])
     ~~~~~^~~~~~~~~~~~~~~~~~~~~~~
     result.swapAt(i, j)
/usr/home/davide/mount/dcci/work/swift/stdlib/private/SwiftPrivate/SwiftPrivate.swift:50:24:
note: conflicting access is here
     swap(&result[i], &result[j])
                      ^~~~~~~~~~
swifterror value when used in a callsite should be marked with
swifterror attribute
%6 = alloca swifterror %swift.error*, align 8
%61 = call i1
@_T0s8SequencePsE8containsS2b7ElementQzKc5where_tKFSRys6UInt16VG_Tgq505_T0s6E37VSbs5Error_pIxydzo_ABSbsAC_pIxidzo_TRAHSbs0H0_pIxydzo_Tfq1cn_nTfq4ng_n(%TSRys6UInt16VG
* noalias nocapture dereferenceable(16) %5, i8* bitcast (i1 (i16)*
@_T0s11_StringCoreV22isRepresentableAsASCIISbyFSbs6UInt16VcfU_ to
i8*), %swift.refcounted* null, %swift.refcounted
* undef, %swift.error** nocapture %6), !dbg !1877
<unknown>:0: error: fatal error encountered during compilation; please
file a bug report with your project and the crash log
<unknown>:0: note: Broken function found, compilation aborted!


(Davide C. C. Italiano) #7

freebsd sys/ as:

`typedef int pthread_key_t;`

Is this compliant with your definition?

Thanks,

···

On Thu, May 25, 2017 at 5:41 PM, Michael Ilseman <milseman@apple.com> wrote:

That would be awesome! Do you use the static stdlib configuration?

I had to make assumptions about the type of pthread_key_t. If the type is wrong, there’s a nice static_assert with a message telling how to fix it. This came up yesterday for Android builds, and fixes are pretty straight forward: https://github.com/apple/swift/pull/9930/files

From a very quick (non-exhaustive) look I see pthread_key_t defined in

--
Davide


(Michael Ilseman) #8

If that’s the case, we’ll want to add in FreeBSD alongside Android for that.

···

On May 25, 2017, at 5:45 PM, Davide Italiano <dccitaliano@gmail.com> wrote:

On Thu, May 25, 2017 at 5:41 PM, Michael Ilseman <milseman@apple.com> wrote:

That would be awesome! Do you use the static stdlib configuration?

I had to make assumptions about the type of pthread_key_t. If the type is wrong, there’s a nice static_assert with a message telling how to fix it. This came up yesterday for Android builds, and fixes are pretty straight forward: https://github.com/apple/swift/pull/9930/files

From a very quick (non-exhaustive) look I see pthread_key_t defined in
freebsd sys/ as:

`typedef int pthread_key_t;`

Is this compliant with your definition?

Thanks,

--
Davide


(Davide C. C. Italiano) #9

I don't think any platform redefines `pthread_key_t` to something else
in FreeBSD in machine-dependent bits, but I'll take another look
[sorry, I don't have a checkout handy and I haven't hacked on anything
for FreeBSD that's not llvm in quite a while].
Anyway, I'm confident that's how it's defined on x86-64 and given
swift doesn't support any other platforms on FreeBSD (I think) we
should be fine.

···

On Thu, May 25, 2017 at 5:47 PM, Michael Ilseman <milseman@apple.com> wrote:

If that’s the case, we’ll want to add in FreeBSD alongside Android for that.

--
Davide


(Jordan Rose) #10

michael-yuji on GitHub just sent us a patch in https://github.com/apple/swift/pull/9940, so at least that part's not a concern.

···

On May 25, 2017, at 17:47, Michael Ilseman via swift-dev <swift-dev@swift.org> wrote:

If that’s the case, we’ll want to add in FreeBSD alongside Android for that.

On May 25, 2017, at 5:45 PM, Davide Italiano <dccitaliano@gmail.com> wrote:

On Thu, May 25, 2017 at 5:41 PM, Michael Ilseman <milseman@apple.com> wrote:

That would be awesome! Do you use the static stdlib configuration?

I had to make assumptions about the type of pthread_key_t. If the type is wrong, there’s a nice static_assert with a message telling how to fix it. This came up yesterday for Android builds, and fixes are pretty straight forward: https://github.com/apple/swift/pull/9930/files

From a very quick (non-exhaustive) look I see pthread_key_t defined in
freebsd sys/ as:

`typedef int pthread_key_t;`

Is this compliant with your definition?

Thanks,

--
Davide

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev