Suggestions for what is breaking Linux master-next bots?

I've been keeping an eye on the macOS master-next bots recently but haven't paid so much attention to the Linux bots. I noticed today that they're all failing when linking libswiftCore.so with:

/usr/bin/ld.gold: error: stdlib/public/core/linux/x86_64/Swift.o: requires dynamic R_X86_64_PC32 reloc against '_swiftEmptyArrayStorage' which may overflow at runtime; recompile with -fPIC

(e.g.: https://ci.swift.org/view/swift-master-next/job/oss-swift-incremental-RA-linux-ubuntu-16_04-master-next/3654/)

The bot logs don't go back more than a few days so I can't tell when this first started to occur. The "Long Test" bots passed as recently as Feb. 8 so it must have been introduced sometime in the last few weeks. I don't see any changes directly involving _swiftEmptyArrayStorage. Did something change in the way we generate code for Builtin.addressof?

cc @Joe_Groff , who I think has run into this before

I haven't seen this particular case. Is _swiftEmptyArrayStorage still visible to be linked from the runtime? Does it have hidden or protected visibility? You might get that error if _swiftEmptyArrayStorage is missing, or if it has default visibility (which with ELF's flat namespace means it can be replaced by a different definition from an LD_PRELOAD or other .so file). Everything the Swift runtime and standard library export ought to have protected visibility.

It looks like the definitions of _swiftEmptyArrayStorage, _swiftEmptyDictionaryStorage, and _swiftEmptySetStorage are not marked with SWIFT_RUNTIME_STDLIB_INTERFACE in GlobalObjects.cpp. Let me try adding that and seeing if it helps.

Here's a PR that attempts to address these issues: Visibility fixes for ELF. by jckarter · Pull Request #14831 · apple/swift · GitHub

It also occurred to me (duh!) that since this isn't happening on master, it must be somehow connected to an upstream LLVM change. I skimmed over all the commits going back to Feb. 8 and this one looked like it might be related:

Login Intrinsics calls should avoid the PLT when "RtLibUseGOT" metadata is present

There were also a few dso_local-related changes (r325514, r324551) but those seemed less likely to be involved.

It's possible that anything in LLVM that lets it more aggressively avoid PLT/GOT overhead for dso-local things could have caused problems with us if there were mismatches in the visibility between the Swift code in the standard library and C++ code in the runtime. swiftc emits everything it can as protected by default, which would make it dso-local, whereas it looks like we declare exported symbols as having default visibility in the C++ runtime code. That link error probably came up because we were emitting code that assumed _swiftEmptyArrayStorage was dso-local, but the default visibility declaration from the runtime headers may have prevented gold from resolving those dso-local relocations.

I just merged Visibility fixes for ELF. by jckarter · Pull Request #14831 · apple/swift · GitHub, which should give those symbols consistently protected visibility. Hopefully that clears up the issue on master-next.

I think it helped. Now it is failing to build libswiftGlibc.so with a similar error about "__environ":

/usr/bin/ld.gold: error: stdlib/public/Platform/linux/x86_64/Glibc.o: requires dynamic R_X86_64_PC32 reloc against '__environ' which may overflow at runtime; recompile with -fPIC

https://ci.swift.org/view/swift-master-next/job/oss-swift-incremental-RA-linux-ubuntu-16_10-master-next/3037

I wonder if we give symbols imported from C or the shims headers the wrong visibility or dso_local-ness.

Maybe. I filed a bug report a few weeks ago suggesting that we need to check that we're setting dso_local appropriately: [SR-7016] Swift should set dso_local for symbols · Issue #49564 · apple/swift · GitHub

1 Like