Linking libswiftCore and a slightly different lib/Demangling in the same binary

Development Compiler
1

tl;dr: I have a situation where swift/lib/Demangling being part of the Swift runtime means I can't link parts of the compiler (which also depend on Demangling) into the same binary as the Swift runtime if they don't match exactly. (Or maybe even if they do?) How can we fix this?

Long version:

  • We regularly import the Swift toolchain sources into our monorepo (tracking master*, not a specific release). (*technically the tensorflow branch, but that's not relevant here)

  • From time to time, we build a Linux toolchain from those sources and check in the binaries and static libraries as a stable "release". libswiftCore.a is one of these, which has swift/lib/Demangling compiled into it. Let's say the toolchain was built at commit A.

  • We don't always release when we sync. Let's say the sources are now at commit B.

  • We want to build things that have both Swift code and also parts of the compiler linked into the same binary. As an example, let's imagine a service that uses swift-format on Linux. We build a binary that contains swift/lib/SyntaxParse and its C++ dependencies from the compiler, plus all the Swift code from swift-format and its dependencies.

    • Binaries that we deploy to production are standalone statically linked binaries. We don't use shared libraries. Let's assume that won't change.

    • We don't prebuild individual compiler libraries like swift/lib/SyntaxParse or Demangling; those targets would be built from source, in this case, at commit B, whenever we build our tool.

  • Just about everything in the Swift compiler eventually depends on swift/lib/Demangling. For example, the syntax parser depends on it via SyntaxParse > Parse > AST > Basic > Demangling. (Trying to excise that dependency wasn't fruitful, and would only solve this specific case.)

All of this means that our binary is going to pull in lib/swift/Demangling twice: once from libswiftCore.a (commit A) and once from the compiler sources (commit B).

Up until now, we've gotten lucky and for some reason lld hasn't been giving us duplicate symbol errors. Today's sync did, so I thought "do I really need the demangler for swift-format?" and tried to -Wl,-zmuldefs my problems away :grimacing:. That caused a crash at runtime:

failed to demangle superclass of _ContiguousArrayStorage from mangled name 's28__ContiguousArrayStorageBaseC'

I'm assuming I'm the victim of undefined behavior here for violating ODR. As an experiment, I looked at my linker inputs file, and libdemangling.a was listed before libswiftCore.a. When I moved it down below and re-linked, the binary ran correctly. Not a real solution, of course.

Random idea: Demangling is a nice standalone library that only depends on header-only LLVM support. Could conditional compilation be used so that swift/lib/Demangling, when compiled for the Swift runtime, could be in a different namespace than when it's compiled for the compiler? That would prevent them from colliding and allow Swift code+runtime and compiler components to be statically linked together, even if the Demangling API/ABI differed slightly between the versions.

One concern would be whether we could then end up building a tool where the Swift code and C++ code could end up with slightly incompatible (de)mangling implementations, and that we'd somehow end up depending on the behavior of both. I suppose that's possible, but I can't think of a better answer right now.

We'd prefer not to have to guarantee keeping the prebuilt compiler and the source in sync at all times, because that would mean increasing our toolchain build and testing load, or syncing less frequently.

2 Likes