Static linking on Linux in Swift 5.3.1

Hey @tomerd , I will try to come with a minimal reproducer that I can share. My main.swift looks pretty straight forward:

import Backtrace

Backtrace.install()

ContenttoolCli.main()

@t089 could you type bt in lldb to give us the backtrace from within LLDB? Alongside the bt a register read may be useful.

hm, indeed ...

* thread #1, name = 'contenttoolcli', stop reason = signal SIGSEGV: invalid address (fault address: 0xe0)
  * frame #0: 0x0000fffff7750690
    frame #1: 0x0000fffff74d9a28
    frame #2: 0x0000fffff74dae54
    frame #3: 0x00000000017e17d8 contenttoolcli`getpwuid_r + 296
    frame #4: 0x00000000017e11ac contenttoolcli`getpwuid + 148
    frame #5: 0x0000000001466c18 contenttoolcli`_CFCopyHomeDirURLForUser + 260
    frame #6: 0x000000000130cd40 contenttoolcli`Foundation.NSHomeDirectory() -> Swift.String + 24
    frame #7: 0x0000000000ed7a98 contenttoolcli`globalinit_33_41385DE57462D266592DAC19BD5D46EA_func0 at Login.swift:50:49
    frame #8: 0x000000000120a2ac contenttoolcli`__pthread_once_slow(once_control=0x00000000041fd6b0, init_routine=(contenttoolcli`__once_proxy)) at pthread_once.c:116
    frame #9: 0x00000000011c4100 contenttoolcli`swift_once + 112
    frame #10: 0x0000000000ed7b1c contenttoolcli`contenttoolcli.AuthConfig.configDir.unsafeMutableAddressor : Foundation.URL at Login.swift:50:16
    frame #11: 0x0000000000ed7d94 contenttoolcli`contenttoolcli.AuthConfig.save(name: Swift.String) throws -> Swift.String at Login.swift:61:58
    frame #12: 0x0000000000ee0528 contenttoolcli`contenttoolcli.Login.run() throws -> () at Login.swift:299:42
    frame #13: 0x0000000000ee6dd4 contenttoolcli`protocol witness for ArgumentParser.ParsableCommand.run() throws -> () in conformance contenttoolcli.Login : ArgumentParser.ParsableCommand in contenttoolcli at <compiler-generated>:0
    frame #14: 0x0000000000452238 contenttoolcli`static (extension in ArgumentParser):ArgumentParser.ParsableCommand.main(Swift.Optional<Swift.Array<Swift.String>>) -> () at ParsableCommand.swift:95:19
    frame #15: 0x0000000000452364 contenttoolcli`static (extension in ArgumentParser):ArgumentParser.ParsableCommand.main() -> () at ParsableCommand.swift:105:10
    frame #16: 0x0000000000f014b8 contenttoolcli`main at main.swift:6:16
    frame #17: 0x00000000017a9cbc contenttoolcli`__libc_start_main + 612
    frame #18: 0x00000000004012d8 contenttoolcli`_start + 76
(lldb) register read
General Purpose Registers:
        x0 = 0x0000000000000070
        x1 = 0x0000fffff752e8d0
        x2 = 0x00000000000000e0
        x3 = 0x0000000000000000
        x4 = 0x0000000040100401
        x5 = 0x0000000055545555
        x6 = 0x203a647773736170
        x7 = 0x2020202020202020
        x8 = 0x000000000000000a
        x9 = 0x000000000000001b
       x10 = 0x0000000000000006
       x11 = 0x74616d726f666e69
       x12 = 0x2020202020202020
       x13 = 0x0a7461706d6f6320
       x14 = 0x0000000000000016
       x15 = 0x0000000000000000
       x16 = 0x0000000000000007
       x17 = 0x0000ffffe80010c8
       x18 = 0x0000fffff77b1a70
       x19 = 0x0000fffff77b5000
       x20 = 0x0000fffff77b0000
       x21 = 0x0000fffff74ee310
       x22 = 0x0000fffff74dd000
       x23 = 0x0000000000000000
       x24 = 0x0000fffff752e8b0
       x25 = 0x0000fffff752e680
       x26 = 0x0000fffff74dcf70
       x27 = 0x0000ffffffffd4c0
       x28 = 0x0000ffffffffd4b8
        fp = 0x0000ffffffffd420
        lr = 0x0000fffff7750674
        sp = 0x0000ffffffffd420
        pc = 0x0000fffff7750690
      cpsr = 0x00001000

And actually, if I use a different part of my program, I get a different backtrace, so there seem to be multiple, different? issues ...

* thread #3, name = 'NIO-ELT-1-#0', stop reason = signal SIGSEGV: invalid address (fault address: 0x62)
  * frame #0: 0x0000fffff7cfb610
    frame #1: 0x0000fffff7cfc7c8
    frame #2: 0x00000000017e40bc contenttoolcli`gaih_inet.constprop.7 + 2380
    frame #3: 0x00000000017e4c38 contenttoolcli`getaddrinfo + 216
    frame #4: 0x00000000009463f0 contenttoolcli`NIO.GetaddrinfoResolver.(resolve in _B791BB4B10FE8C41F6F4E349E094F88F)(host: Swift.String, port: Swift.Int) -> () at GetaddrinfoResolver.swift:134:15
    frame #5: 0x0000000000945d88 contenttoolcli`NIO.GetaddrinfoResolver.initiateAAAAQuery(host: Swift.String, port: Swift.Int) -> NIO.EventLoopFuture<Swift.Array<NIO.SocketAddress>> at GetaddrinfoResolver.swift:86:9
    frame #6: 0x000000000094727c contenttoolcli`protocol witness for NIO.Resolver.initiateAAAAQuery(host: Swift.String, port: Swift.Int) -> NIO.EventLoopFuture<Swift.Array<NIO.SocketAddress>> in conformance NIO.GetaddrinfoResolver : NIO.Resolver in NIO at <compiler-generated>:0
    frame #7: 0x000000000094bcc4 contenttoolcli`NIO.HappyEyeballsConnector.(beginDNSResolution in _2EDDDA16464B5E6B93684A67F9A71C90)() -> () at HappyEyeballs.swift:434:49
    frame #8: 0x000000000094b450 contenttoolcli`NIO.HappyEyeballsConnector.(processInput in _2EDDDA16464B5E6B93684A67F9A71C90)(NIO.HappyEyeballsConnector.(ConnectorInput in _2EDDDA16464B5E6B93684A67F9A71C90)) -> () at HappyEyeballs.swift:325:13
    frame #9: 0x000000000094b288 contenttoolcli`closure #1 () -> () in NIO.HappyEyeballsConnector.resolveAndConnect() -> NIO.EventLoopFuture<NIO.Channel> at HappyEyeballs.swift:311:18
    frame #10: 0x00000000009a9300 contenttoolcli`reabstraction thunk helper from @escaping @callee_guaranteed () -> () to @escaping @callee_guaranteed () -> (@out ()) at <compiler-generated>:0
    frame #11: 0x00000000009a9320 contenttoolcli`reabstraction thunk helper from @escaping @callee_guaranteed () -> (@out ()) to @escaping @callee_guaranteed () -> () at <compiler-generated>:0
    frame #12: 0x00000000009a936c contenttoolcli`closure #4 () -> () in NIO.SelectableEventLoop.run() throws -> () at SelectableEventLoop.swift:454:25
    frame #13: 0x00000000004de4b4 contenttoolcli`reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @escaping @callee_guaranteed () -> (@out (), @error @owned Swift.Error) at <compiler-generated>:0
    frame #14: 0x00000000009ac188 contenttoolcli`reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @escaping @callee_guaranteed () -> (@out (), @error @owned Swift.Error)partial apply forwarder with unmangled suffix ".9" at <compiler-generated>:0
    frame #15: 0x00000000009a3a34 contenttoolcli`NIO.withAutoReleasePool<A>(() throws -> A) throws -> A at SelectableEventLoop.swift:26:16
    frame #16: 0x00000000009a7be4 contenttoolcli`NIO.SelectableEventLoop.run() throws -> () at SelectableEventLoop.swift:453:21
    frame #17: 0x0000000000924000 contenttoolcli`static NIO.MultiThreadedEventLoopGroup.(runTheLoop in _D5D78C61B22284700B9BD1ACFBC25157)(thread: NIO.NIOThread, canEventLoopBeShutdownIndividually: Swift.Bool, selectorFactory: () throws -> NIO.Selector<NIO.NIORegistration>, initializer: (NIO.NIOThread) -> (), _: (NIO.SelectableEventLoop) -> ()) -> () at EventLoop.swift:851:22
    frame #18: 0x0000000000924674 contenttoolcli`closure #1 (NIO.NIOThread) -> () in static NIO.MultiThreadedEventLoopGroup.(setupThreadAndEventLoop in _D5D78C61B22284700B9BD1ACFBC25157)(name: Swift.String, selectorFactory: () throws -> NIO.Selector<NIO.NIORegistration>, initializer: (NIO.NIOThread) -> ()) -> NIO.SelectableEventLoop at EventLoop.swift:871:41
    frame #19: 0x0000000000924c18 contenttoolcli`reabstraction thunk helper from @escaping @callee_guaranteed (@guaranteed NIO.NIOThread) -> () to @escaping @callee_guaranteed (@in_guaranteed NIO.NIOThread) -> (@out ()) at <compiler-generated>:0
    frame #20: 0x00000000009fc79c contenttoolcli`closure #1 (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static NIO.ThreadOpsPosix.run(handle: inout Swift.Optional<Swift.UInt>, args: NIO.Box<(body: (NIO.NIOThread) -> (), name: Swift.Optional<Swift.String>)>, detachThread: Swift.Bool) -> () at ThreadPosix.swift:105:13
    frame #21: 0x00000000009fc974 contenttoolcli`@objc closure #1 (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static NIO.ThreadOpsPosix.run(handle: inout Swift.Optional<Swift.UInt>, args: NIO.Box<(body: (NIO.NIOThread) -> (), name: Swift.Optional<Swift.String>)>, detachThread: Swift.Bool) -> () at <compiler-generated>:0
    frame #22: 0x00000000012056a0 contenttoolcli`start_thread(arg=0x0000ffffffffd44f) at pthread_create.c:463
    frame #23: 0x00000000017eee8c contenttoolcli`thread_start + 12


General Purpose Registers:
        x0 = 0x0000000000000031
        x1 = 0x0000000000000062
        x2 = 0x0000000000000000
        x3 = 0x0000fffff74d31d0
        x4 = 0x0000fffff7d5aaf4
        x5 = 0x00000000fbad2488
        x6 = 0x312e302e302e3732
        x7 = 0x6f686c61636f6c09
        x8 = 0x000000000000003f
        x9 = 0x0000000000000001
       x10 = 0x0000000000000000
       x11 = 0x0000000000000038
       x12 = 0x6f6c09312e302e30
       x13 = 0x0000fffff7e9a308
       x14 = 0x0000fffff7e8ce08
       x15 = 0x0000000000000000
       x16 = 0x0000fffff7eb4ff0
       x17 = 0x0000fffff7d120d0
       x18 = 0x0000fffff7fdca70
       x19 = 0x000000007fffffff
       x20 = 0x0000fffff74d5820
       x21 = 0x00000000ffffffff
       x22 = 0x0000fffff7d596f0
       x23 = 0x0000fffff74d31d0
       x24 = 0x00000000000003e0
       x25 = 0x00000000000003e0
       x26 = 0x00000000000003df
       x27 = 0x0000fffff74d35af
       x28 = 0x0000000000000001
        fp = 0x0000fffff74d2c90
        lr = 0x0000fffff7cfb5f4
        sp = 0x0000fffff74d2c90
        pc = 0x0000fffff7cfb610
      cpsr = 0x80001000

Again, when I don't use -static-executable but -static-stdlib both paths through the program work fine.

@t089 this kinda looks like you haven't got the exact same glibc on the build and the runtime system.

Please note that glibc cannot be used as a fully static library. For things that can have plugins (like NSS), glibc will always dynamically load these plugins. For this to work the glibc version of your build system and your runtime system need to be the exact same ones. That is because glibc does have an ABI for its dynamic library but doesn't have an "internal ABI".

Both of the crashes happen inside glibc code that deals with NSS stuff (DNS / getpwuid).

More info here.

I see, I was suspecting something like this, buuuuuuut: I am running the app in the same docker image I used for building. So glibc should be exactly the same ...

Anyhow, I think I understand now, that -static-executable is not really what I want because it anyhow cannot be fully static because I need some functionality from glibc, which will then require dynamic loading anyhow. Does this reasoning make sense?

So in essence this means, the resulting binary is only portable between systems that have (the same?) glibc version, as used during swift build?

I think it would be a bug if -static-executable would include glibc statically, because that cannot be included in the list for the reasons stated. AFAIK the flag is intended to force linking ICU statically.

Yes, it does, and I agree. -static-stdlib is the maximum I would personally recommend.

Yes, essentially this is true. I think if you avoided using NSS entirely in your program then it would work. And just to state the obvious, that's not a bug in Swift bug a bug in glibc. For truly static binaries we'd need to be able to target musl or some other libc that actually supports static linking.

There is one other (unrecommended) option which is to build glibc yourself and ./configure it with --use-static-nss (IIRC).

Well, -static-executable should do what it says on the tin which is "building a static executable". Just like C compilers can also build you static executables that then don't actually work because you use glibc. The -static-stdlib flag is more like what you describe which is "link in statically anything that Swift brings with itself (libswiftCore/libswiftFoundation/ICU/...). Binaries produced with -static-stdlib will however still link glibc and friends dynamically but at least for glibc that's the recommendation anyway.

One argument we could make is that -static-executable should issue a warning about that glibc quirk.

Hmm, this is interesting and definitely sounds like a bug. If you manage to create a repro that you can share, then I'm more than happy to look into it.

+1 cc @drexin

I don't think it is a bug or quirk. Linking glibc statically is a bug in the first place and can't possibly work right for the reasons stated before (it simply depends on dynamic loading).
On Linux you just always have to link glibc dynamically, that's just how the system works. Doesn't Go or Rust link glibc dynamically?

One argument we could make is that -static-executable should issue a warning about that glibc quirk.

In all fairness it already does so ... I was hoping this would be more of a hypothetical problem ...

/app/.build/checkouts/swift-nio/Sources/NIO/GetaddrinfoResolver.swift:134: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

On Linux you just always have to link glibc dynamically, that's just how the system works. Doesn't Go or Rust link glibc dynamically?

Indeed from what I could see Rust and Go always link glibc dynamically. I wonder how are the tools like kubectl, helm, etc distributed as a single "linux" binary? Maybe Go has some non-glibc based implementations?

Go usually targets the syscall ABI directly, so it doesn't go through any libc.

2 Likes

Dear all! Is it possible to use musl gcc toolchain instead of original with glibc? Is there any instruction/documentation how to do that? There is a lot of reasons why some of us may need portable exe.

No, there is no hint of musl support in the Swift stdlib yet, you can find supported C standard libraries here. Depending on your use, you may be able to get away with just linking musl because of its partial ABI compatibility with glibc, but I doubt that would work well for anything more involved.

@drexin in light of this, is there any way for us to query the link flags for Swift static stdlibs such that we can use ld directly? This is because in Bazel's rules_swift, we uses the rest of cc_toolchain for linking, not Swift itself. See issue: Support fully-static linking · Issue #8 · bazelbuild/rules_swift · GitHub

At the moment, I believe we use swift-autolink-extract to extract related dynamic linking information: rules_swift/autolinking.bzl at master · bazelbuild/rules_swift · GitHub any new flags should be aware of to implement for static linking?

Thanks!

Hey @liuliu,

you can use swift-autolink-extract on object files to get the dependencies. The general flags that swiftc uses for static linking can be read from usr/lib/swift_static/linux/static-stdlib-args.lnk and usr/lib/swift_static/linux/static-executable-args.lnk respectively.

Terms of Service

Privacy Policy

Cookie Policy