Swift Linux layout considerations (aka Linux is difficult, lets go shopping)

jrose · September 7, 2019, 12:34am

@compnerd and I met in person to discuss this last week, along with @alblue, @johannesweiss, @kevints, @beccadax, @millenomi, and @Rostepher. We discovered we're in agreement about nearly everything, and that most of the remaining questions are around terminology (i.e. "we know what we want to do, but what do we call the things and how do you write it on the command line?").

(Please correct me on anything I've misremembered, misrepresented, or left out!)

Groundwork

We agreed on some starting goals:

The Swift compiler should have first-class support for cross-compilation.
Open-source toolchains must continue to work with Xcode.
Using Swift must not require modifying any platform content, either in / or in Xcode.app or in a development package you got from Microsoft.
(I added this one late) There should be downloads on swift.org where you download one thing and then swift run works on a package you just checked out.

During his work on Windows, @compnerd broke down "the stuff we ship on swift.org" into four components:

Toolchain: The minimum set of tools needed to compile a Swift program. In its fullest form this would include swiftc, clang, lld, and llvm-ar, providing full multi-platform compile-and-link support.
Swift-based tools: SwiftPM, SwiftSyntax, SourceKit-LSP, and anything else that's used for Swift development that requires a Swift runtime itself. (It's a little unfortunate SwiftPM is in this category since that's how many Swift libraries are distributed, but, well, SwiftPM's written in Swift, so we can't get around it.)
Swift compile-time dependencies: This includes whatever is needed to compile and link a Swift program for a specific platform. For example, the Swift stdlib swiftmodule would be in here.
Swift run-time dependencies: This is the content that needs to be present to run a Swift program for a specific platform. On Apple platforms, for example, this is installed in the OS by default; for Windows you'd distribute it alongside your app; and on Linux...well, it's complicated and we haven't really figured it out yet.

Making Goal 4 work requires putting all of these in one package, but that doesn't have to be the only way to distribute them. In particular, cross-compilation should work with your existing dev tools and toolchain, and only rely on swapping out your compile-time and run-time dependencies.

Okay, so we all agree on the goals. How do we get there?

System content

At the start of the meeting we started to list out which content was distributed where, for both Linux and Apple platforms. We then started discussing the "system" content for both Android and Windows, and how those differ from Linux and Apple's nice, neat "sysroot" model. (This is what @compnerd's previous post covered.) The key insight from this discussion was that each of these platforms has one or more directories with known layouts that can be turned into search paths based on the -target you're compiling for, possibly with some extra information gleaned from other command-line arguments or environment variables. (For Windows, the WinSDK version is typically an environment variable.)

Proposal: swiftc should accomodate multiple system directories, and automatically add appropriate search paths within them (the same way -sdk foo.sdk/ searches foo.sdk/System/Library/Frameworks/ on Apple platforms). We'll have to audit how else SDKPath is used today in the compiler and where it's assumed to be unique, but we think it'll work.

Swift compile-time content

For Apple platforms, the system directory (what Xcode calls an SDK) also contains the cross-compilation Swift content for that platform (starting in Xcode 11, anyway). But other platforms don't have Swift included ahead of time. Where do they get that information? It must be an external download, which would contain:

The stdlib, obviously
Probably the corelibs as well
Anything needed to massage the system content into a good shape (module maps, API notes, maybe more?)
Additional config files for search paths and such (CMake modules, pkgconfig, SwiftPM's destination.json)

During the meeting we called this the cross-compilation support directory. It's also roughly the "Swift compile-time dependencies" mentioned above. To cross-compile, then, you should be able to point the compiler at a particular system directory (or set of directories) and a particular cross-compilation support directory.

Proposal: This directory should have the same layout as the system directories, so that we can use the same mechanism described in the previous section for the Swift content as well as the system content. This also implicitly handles the default Xcode case, where the two directories are the same.

Multi-arch cross-compilation support

It would kind of stink if we had to have separate cross-compilation support directories for each ABI we wanted to support—that would duplicate a lot of content needlessly. However, we already have the notion of ABI-specific subdirectories in the system content directories for link dependencies (see @compnerd's examples above). Is that enough for the Swift content as well, if laid out in the same way? We can certainly put libraries-for-linking in such a layout, so it's "just" the swiftmodules and other content that matters.

Proposal: Allow the cross-compilation support directories to use the multi-target layout for swiftmodules, like Apple platforms already do. Also, don't bother encoding differences that wouldn't result in exposing different Swift API, like different libc implementations or hard-float / soft-float.

usr/
  lib/
    swift/
      Swift.swiftmodule/
        armv7-unknown-linux.swiftdoc
        armv7-unknown-linux.swiftmodule
        x86_64-unknown-linux.swiftdoc
        x86_64-unknown-linux.swiftmodule
  armv7-unknown-linux-eabi/
    lib/
      swift/
        libswiftCore.so
  armv7-unknown-linux-eabihf/
    lib/
      swift/
        libswiftCore.so
  x86_64-unknown-linux/
    lib/
      swift/
        libswiftCore.so

(exact directory layout not proposed; also forgive me for my fake Linux link ABI names)

Host == build

Okay, so we've accomplished goal 1 ("first-class cross-compilation support") and goal 3 ("without modifying system content directory"). But can we pack all the content into one downloadable thing so that people can get started with swift run easily? That is, can we have a default search path that's compiler-relative when not cross-compiling?

We didn't quite converge to a proposal for this, but I think we ended with two main options:

Assume that the compiler is in $DIR/usr/bin/ inside a cross-compilation support directory $DIR. In order for this to work for Xcode, we'd have to allow $DIR/usr/lib/swift/$PLATFORM as a standard search path as well as the sensible $DIR/usr/lib/swift/. (This is roughly what we have implemented today, just cleaned up if we want.)

# Apple toolchain
usr/
  bin/
    swiftc
  lib/
    swift/
      macosx/
        libswiftCore.dylib
        Swift.swiftmodule/
      iphoneos/
        libswiftCore.dylib
        Swift.swiftmodule/

# Linux toolchain - x86_64 only
usr/
  bin/
    swiftc
  lib/
    swift/
      linux/
        libswiftCore.so
        Swift.swiftmodule

Put entire cross-compilation support directories somewhere relative to the compiler, and have the compiler find them. This is more uniform but probably more work to move everything around, and might be more disruptive for external build systems that have come to depend on Swift's toolchain layout.

usr/
  bin/
    swiftc
  lib/
    swift/
      macosx/
        usr/
          lib/
            swift/
              libswiftCore.dylib
              Swift.swiftmodule/
      iphoneos/
        usr/
          lib/
            swift/
              libswiftCore.dylib
              Swift.swiftmodule/

Next steps

Agree on a name for the stackable platform directories. I personally think we should stick with -sdk and just allow stacking SDKs, but we didn't focus on resolving this in the meeting.
Decide on how to lay out the non-cross downloads.
Figure out exactly what it'll take to make downloadable Xcode toolchains prefer their stdlib over the one in Xcode (to preserve goal 2).
Get everything in the compiler currently using -resource-dir to look in stacked SDKs instead, unless it really is tied to the compiler. (That's mostly just Clang's intrinsic headers.)
Actually implement the stackable SDK thing and all the search paths that go with it.
Consider moving or symlinking the Swift libraries for the Swift-based Tools out of the platform-specific subdirectory; they need to be found relative to those tools anyway.
(Apple folks) Deal with how this all interacts with Catalyst. (To this end it would be great to hold off on a bunch of this work until we Apple folks can upstream the Catalyst support, at least the work we had to do for search paths. Trust me, it will make a bunch of things easier.)

Future work

Consider moving the C dispatch implementation to usr/{include,lib}/ within the cross-compilation support directory instead of usr/lib/swift/ (at least on Linux). There's no real reason to do this besides consistency, but, well, consistency.
Consider using symbol versioning on ELF to defend against mixing unstable Swift ABIs. (Someone needs to go off and investigate this.)
Decide if the Swift toolchain's copy of Clang and the other LLVM tools should be prefixed (swift-clang) to not conflict with platform tools. (Discussed somewhat in Shipping clang with the Swift toolchains, but with no action taken yet.)
Anything to do with installing a Linux toolchain into /, or even multiple Linux runtimes.