Hey @tomerd , I will try to come with a minimal reproducer that I can share. My main.swift looks pretty straight forward:
import Backtrace
Backtrace.install()
ContenttoolCli.main()
@t089 could you type bt in lldb to give us the backtrace from within LLDB? Alongside the bt a register read may be useful.
hm, indeed ...
* thread #1, name = 'contenttoolcli', stop reason = signal SIGSEGV: invalid address (fault address: 0xe0)
* frame #0: 0x0000fffff7750690
frame #1: 0x0000fffff74d9a28
frame #2: 0x0000fffff74dae54
frame #3: 0x00000000017e17d8 contenttoolcli`getpwuid_r + 296
frame #4: 0x00000000017e11ac contenttoolcli`getpwuid + 148
frame #5: 0x0000000001466c18 contenttoolcli`_CFCopyHomeDirURLForUser + 260
frame #6: 0x000000000130cd40 contenttoolcli`Foundation.NSHomeDirectory() -> Swift.String + 24
frame #7: 0x0000000000ed7a98 contenttoolcli`globalinit_33_41385DE57462D266592DAC19BD5D46EA_func0 at Login.swift:50:49
frame #8: 0x000000000120a2ac contenttoolcli`__pthread_once_slow(once_control=0x00000000041fd6b0, init_routine=(contenttoolcli`__once_proxy)) at pthread_once.c:116
frame #9: 0x00000000011c4100 contenttoolcli`swift_once + 112
frame #10: 0x0000000000ed7b1c contenttoolcli`contenttoolcli.AuthConfig.configDir.unsafeMutableAddressor : Foundation.URL at Login.swift:50:16
frame #11: 0x0000000000ed7d94 contenttoolcli`contenttoolcli.AuthConfig.save(name: Swift.String) throws -> Swift.String at Login.swift:61:58
frame #12: 0x0000000000ee0528 contenttoolcli`contenttoolcli.Login.run() throws -> () at Login.swift:299:42
frame #13: 0x0000000000ee6dd4 contenttoolcli`protocol witness for ArgumentParser.ParsableCommand.run() throws -> () in conformance contenttoolcli.Login : ArgumentParser.ParsableCommand in contenttoolcli at <compiler-generated>:0
frame #14: 0x0000000000452238 contenttoolcli`static (extension in ArgumentParser):ArgumentParser.ParsableCommand.main(Swift.Optional<Swift.Array<Swift.String>>) -> () at ParsableCommand.swift:95:19
frame #15: 0x0000000000452364 contenttoolcli`static (extension in ArgumentParser):ArgumentParser.ParsableCommand.main() -> () at ParsableCommand.swift:105:10
frame #16: 0x0000000000f014b8 contenttoolcli`main at main.swift:6:16
frame #17: 0x00000000017a9cbc contenttoolcli`__libc_start_main + 612
frame #18: 0x00000000004012d8 contenttoolcli`_start + 76
(lldb) register read
General Purpose Registers:
x0 = 0x0000000000000070
x1 = 0x0000fffff752e8d0
x2 = 0x00000000000000e0
x3 = 0x0000000000000000
x4 = 0x0000000040100401
x5 = 0x0000000055545555
x6 = 0x203a647773736170
x7 = 0x2020202020202020
x8 = 0x000000000000000a
x9 = 0x000000000000001b
x10 = 0x0000000000000006
x11 = 0x74616d726f666e69
x12 = 0x2020202020202020
x13 = 0x0a7461706d6f6320
x14 = 0x0000000000000016
x15 = 0x0000000000000000
x16 = 0x0000000000000007
x17 = 0x0000ffffe80010c8
x18 = 0x0000fffff77b1a70
x19 = 0x0000fffff77b5000
x20 = 0x0000fffff77b0000
x21 = 0x0000fffff74ee310
x22 = 0x0000fffff74dd000
x23 = 0x0000000000000000
x24 = 0x0000fffff752e8b0
x25 = 0x0000fffff752e680
x26 = 0x0000fffff74dcf70
x27 = 0x0000ffffffffd4c0
x28 = 0x0000ffffffffd4b8
fp = 0x0000ffffffffd420
lr = 0x0000fffff7750674
sp = 0x0000ffffffffd420
pc = 0x0000fffff7750690
cpsr = 0x00001000
And actually, if I use a different part of my program, I get a different backtrace, so there seem to be multiple, different? issues ...
* thread #3, name = 'NIO-ELT-1-#0', stop reason = signal SIGSEGV: invalid address (fault address: 0x62)
* frame #0: 0x0000fffff7cfb610
frame #1: 0x0000fffff7cfc7c8
frame #2: 0x00000000017e40bc contenttoolcli`gaih_inet.constprop.7 + 2380
frame #3: 0x00000000017e4c38 contenttoolcli`getaddrinfo + 216
frame #4: 0x00000000009463f0 contenttoolcli`NIO.GetaddrinfoResolver.(resolve in _B791BB4B10FE8C41F6F4E349E094F88F)(host: Swift.String, port: Swift.Int) -> () at GetaddrinfoResolver.swift:134:15
frame #5: 0x0000000000945d88 contenttoolcli`NIO.GetaddrinfoResolver.initiateAAAAQuery(host: Swift.String, port: Swift.Int) -> NIO.EventLoopFuture<Swift.Array<NIO.SocketAddress>> at GetaddrinfoResolver.swift:86:9
frame #6: 0x000000000094727c contenttoolcli`protocol witness for NIO.Resolver.initiateAAAAQuery(host: Swift.String, port: Swift.Int) -> NIO.EventLoopFuture<Swift.Array<NIO.SocketAddress>> in conformance NIO.GetaddrinfoResolver : NIO.Resolver in NIO at <compiler-generated>:0
frame #7: 0x000000000094bcc4 contenttoolcli`NIO.HappyEyeballsConnector.(beginDNSResolution in _2EDDDA16464B5E6B93684A67F9A71C90)() -> () at HappyEyeballs.swift:434:49
frame #8: 0x000000000094b450 contenttoolcli`NIO.HappyEyeballsConnector.(processInput in _2EDDDA16464B5E6B93684A67F9A71C90)(NIO.HappyEyeballsConnector.(ConnectorInput in _2EDDDA16464B5E6B93684A67F9A71C90)) -> () at HappyEyeballs.swift:325:13
frame #9: 0x000000000094b288 contenttoolcli`closure #1 () -> () in NIO.HappyEyeballsConnector.resolveAndConnect() -> NIO.EventLoopFuture<NIO.Channel> at HappyEyeballs.swift:311:18
frame #10: 0x00000000009a9300 contenttoolcli`reabstraction thunk helper from @escaping @callee_guaranteed () -> () to @escaping @callee_guaranteed () -> (@out ()) at <compiler-generated>:0
frame #11: 0x00000000009a9320 contenttoolcli`reabstraction thunk helper from @escaping @callee_guaranteed () -> (@out ()) to @escaping @callee_guaranteed () -> () at <compiler-generated>:0
frame #12: 0x00000000009a936c contenttoolcli`closure #4 () -> () in NIO.SelectableEventLoop.run() throws -> () at SelectableEventLoop.swift:454:25
frame #13: 0x00000000004de4b4 contenttoolcli`reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @escaping @callee_guaranteed () -> (@out (), @error @owned Swift.Error) at <compiler-generated>:0
frame #14: 0x00000000009ac188 contenttoolcli`reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @escaping @callee_guaranteed () -> (@out (), @error @owned Swift.Error)partial apply forwarder with unmangled suffix ".9" at <compiler-generated>:0
frame #15: 0x00000000009a3a34 contenttoolcli`NIO.withAutoReleasePool<A>(() throws -> A) throws -> A at SelectableEventLoop.swift:26:16
frame #16: 0x00000000009a7be4 contenttoolcli`NIO.SelectableEventLoop.run() throws -> () at SelectableEventLoop.swift:453:21
frame #17: 0x0000000000924000 contenttoolcli`static NIO.MultiThreadedEventLoopGroup.(runTheLoop in _D5D78C61B22284700B9BD1ACFBC25157)(thread: NIO.NIOThread, canEventLoopBeShutdownIndividually: Swift.Bool, selectorFactory: () throws -> NIO.Selector<NIO.NIORegistration>, initializer: (NIO.NIOThread) -> (), _: (NIO.SelectableEventLoop) -> ()) -> () at EventLoop.swift:851:22
frame #18: 0x0000000000924674 contenttoolcli`closure #1 (NIO.NIOThread) -> () in static NIO.MultiThreadedEventLoopGroup.(setupThreadAndEventLoop in _D5D78C61B22284700B9BD1ACFBC25157)(name: Swift.String, selectorFactory: () throws -> NIO.Selector<NIO.NIORegistration>, initializer: (NIO.NIOThread) -> ()) -> NIO.SelectableEventLoop at EventLoop.swift:871:41
frame #19: 0x0000000000924c18 contenttoolcli`reabstraction thunk helper from @escaping @callee_guaranteed (@guaranteed NIO.NIOThread) -> () to @escaping @callee_guaranteed (@in_guaranteed NIO.NIOThread) -> (@out ()) at <compiler-generated>:0
frame #20: 0x00000000009fc79c contenttoolcli`closure #1 (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static NIO.ThreadOpsPosix.run(handle: inout Swift.Optional<Swift.UInt>, args: NIO.Box<(body: (NIO.NIOThread) -> (), name: Swift.Optional<Swift.String>)>, detachThread: Swift.Bool) -> () at ThreadPosix.swift:105:13
frame #21: 0x00000000009fc974 contenttoolcli`@objc closure #1 (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static NIO.ThreadOpsPosix.run(handle: inout Swift.Optional<Swift.UInt>, args: NIO.Box<(body: (NIO.NIOThread) -> (), name: Swift.Optional<Swift.String>)>, detachThread: Swift.Bool) -> () at <compiler-generated>:0
frame #22: 0x00000000012056a0 contenttoolcli`start_thread(arg=0x0000ffffffffd44f) at pthread_create.c:463
frame #23: 0x00000000017eee8c contenttoolcli`thread_start + 12
General Purpose Registers:
x0 = 0x0000000000000031
x1 = 0x0000000000000062
x2 = 0x0000000000000000
x3 = 0x0000fffff74d31d0
x4 = 0x0000fffff7d5aaf4
x5 = 0x00000000fbad2488
x6 = 0x312e302e302e3732
x7 = 0x6f686c61636f6c09
x8 = 0x000000000000003f
x9 = 0x0000000000000001
x10 = 0x0000000000000000
x11 = 0x0000000000000038
x12 = 0x6f6c09312e302e30
x13 = 0x0000fffff7e9a308
x14 = 0x0000fffff7e8ce08
x15 = 0x0000000000000000
x16 = 0x0000fffff7eb4ff0
x17 = 0x0000fffff7d120d0
x18 = 0x0000fffff7fdca70
x19 = 0x000000007fffffff
x20 = 0x0000fffff74d5820
x21 = 0x00000000ffffffff
x22 = 0x0000fffff7d596f0
x23 = 0x0000fffff74d31d0
x24 = 0x00000000000003e0
x25 = 0x00000000000003e0
x26 = 0x00000000000003df
x27 = 0x0000fffff74d35af
x28 = 0x0000000000000001
fp = 0x0000fffff74d2c90
lr = 0x0000fffff7cfb5f4
sp = 0x0000fffff74d2c90
pc = 0x0000fffff7cfb610
cpsr = 0x80001000
Again, when I don't use -static-executable but -static-stdlib both paths through the program work fine.
@t089 this kinda looks like you haven't got the exact same glibc on the build and the runtime system.
Please note that glibc cannot be used as a fully static library. For things that can have plugins (like NSS), glibc will always dynamically load these plugins. For this to work the glibc version of your build system and your runtime system need to be the exact same ones. That is because glibc does have an ABI for its dynamic library but doesn't have an "internal ABI".
Both of the crashes happen inside glibc code that deals with NSS stuff (DNS / getpwuid).
More info here.
I see, I was suspecting something like this, buuuuuuut: I am running the app in the same docker image I used for building. So glibc should be exactly the same ...
Anyhow, I think I understand now, that -static-executable is not really what I want because it anyhow cannot be fully static because I need some functionality from glibc, which will then require dynamic loading anyhow. Does this reasoning make sense?
So in essence this means, the resulting binary is only portable between systems that have (the same?) glibc version, as used during swift build?
I think it would be a bug if -static-executable would include glibc statically, because that cannot be included in the list for the reasons stated. AFAIK the flag is intended to force linking ICU statically.
1 Like
Yes, it does, and I agree. -static-stdlib is the maximum I would personally recommend.
Yes, essentially this is true. I think if you avoided using NSS entirely in your program then it would work. And just to state the obvious, that's not a bug in Swift bug a bug in glibc. For truly static binaries we'd need to be able to target musl or some other libc that actually supports static linking.
There is one other (unrecommended) option which is to build glibc yourself and ./configure it with --use-static-nss (IIRC).
Well, -static-executable should do what it says on the tin which is "building a static executable". Just like C compilers can also build you static executables that then don't actually work because you use glibc. The -static-stdlib flag is more like what you describe which is "link in statically anything that Swift brings with itself (libswiftCore/libswiftFoundation/ICU/...). Binaries produced with -static-stdlib will however still link glibc and friends dynamically but at least for glibc that's the recommendation anyway.
One argument we could make is that -static-executable should issue a warning about that glibc quirk.
1 Like
Hmm, this is interesting and definitely sounds like a bug. If you manage to create a repro that you can share, then I'm more than happy to look into it.
+1 cc @drexin
I don't think it is a bug or quirk. Linking glibc statically is a bug in the first place and can't possibly work right for the reasons stated before (it simply depends on dynamic loading).
On Linux you just always have to link glibc dynamically, that's just how the system works. Doesn't Go or Rust link glibc dynamically?
One argument we could make is that
-static-executableshould issue a warning about that glibc quirk.
In all fairness it already does so ... I was hoping this would be more of a hypothetical problem ...
/app/.build/checkouts/swift-nio/Sources/NIO/GetaddrinfoResolver.swift:134: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
On Linux you just always have to link glibc dynamically, that's just how the system works. Doesn't Go or Rust link glibc dynamically?
Indeed from what I could see Rust and Go always link glibc dynamically. I wonder how are the tools like kubectl, helm, etc distributed as a single "linux" binary? Maybe Go has some non-glibc based implementations?
Go usually targets the syscall ABI directly, so it doesn't go through any libc.
2 Likes
No, there is no hint of musl support in the Swift stdlib yet, you can find supported C standard libraries here. Depending on your use, you may be able to get away with just linking musl because of its partial ABI compatibility with glibc, but I doubt that would work well for anything more involved.
@drexin in light of this, is there any way for us to query the link flags for Swift static stdlibs such that we can use ld directly? This is because in Bazel's rules_swift, we uses the rest of cc_toolchain for linking, not Swift itself. See issue: Support fully-static linking · Issue #8 · bazelbuild/rules_swift · GitHub
At the moment, I believe we use swift-autolink-extract to extract related dynamic linking information: rules_swift/autolinking.bzl at master · bazelbuild/rules_swift · GitHub any new flags should be aware of to implement for static linking?
Thanks!
Hey @liuliu,
you can use swift-autolink-extract on object files to get the dependencies. The general flags that swiftc uses for static linking can be read from usr/lib/swift_static/linux/static-stdlib-args.lnk and usr/lib/swift_static/linux/static-executable-args.lnk respectively.