Swift Linux layout considerations (aka Linux is difficult, lets go shopping)

[compnerd (Saleem Abdulrasool)]

Hello again Swift developers,

Welcome back to another round of warts in the Linux support. I've been working on the driver a bit recently, trying to wrestle with some of the layout assumptions that are baked into the driver. The motivation behind this is to help separate the standard library development and the compiler development as well as cross-compilation.

Starting off with the original state of the world, the standard library and the compiler builds were intertwined completely, the swift (SDK) content being placed into the compiler resource directory and this path to relative to the compiler.

With additional architectures being added, the swift content gained an additional level of indirection - the architecture. This made the resource directory change from 'usr/lib/swift//...to a split of the content acrossusr/lib/swift// which contained the swift content (.swiftmodule, .swiftdoc) and usr/lib/swift/` which contained the architecture dependent runtime components. This means that a single build can only target a single architecture of the platform. It also pretty much meant that everything was built as a single unit (or you would have to copy and paste content across machines to build on a different build machine).

The reasonable approach to deal with this was to split up the runtime compilation and the compiler builds. This is the work that I finished a little while ago, permitting building just the toolchain or building just the standard library. With this work complete, it was possible to build just the standard library for a different target than the compiler allowing building the compiler for Windows, macOS, or Linux with the standard library for android (or Windows, Linux).

With that in place, the separation of the content of the SDK (standard library, libdispatch/swiftDispatch, Foundation) was now possible. However, the behaviour of -sdk was odd in that it was treated as a sysroot and did not actually cascade the search for the runtime support (swiftrt.o/swiftrt.obj), glibc.modulemap, or the swift runtime bits for linking. That brings me to the set of changes that I am currently working on - allowing cascading of the searches. This unfortunately also brings to light that we need a separation of Swift SDK search and the system content (i.e. sysroot). The reason for this is that repackaging another vendor's SDK may be more involved than what I would like to get into (e.g. can I bundle in the content of MacOSX.sdk?) Additionally, swift's build system currently builds the content as a sysroot more than a SDK (the platform is referenced again in the SDK where it should not since the SDK is platform dependent by definition).

I assume that the problem of the generated layout is possible to solve (just some work which needs to be done on the build system). But, that brings me to the point of future work that I am not entirely certain on how to best address. This work would solve the issue of the architecture dependent bits in the architecture independent location, as that is the last piece necessary to complete the SDK layout support. However, there is another wrinkle here - the current layout really does not provide sufficient granularity to be able to solve the problem. For concreteness, let us consider a specific case: "Linux ARM". Let us restrict ourselves to the 32-bit variant of ARMv7 (is anyone actually working with ARMv6 hardware? Anything less than ARMv5TE in LLVM is pretty suspect I imagine). Well, the problem is that ARMv7 is immensely fragmented, largely incompatible, and all the same architecture: armv7. That is, 'usr/lib/linux/armv7` does not work to home the platform. The platform may be any of the following:

1. -target armv7-unknown-linux-gnueabi -mfloat-abi=softfp (okay, I doubt that anyone actually uses this anymore, but I'm not particularly right in the head I've been told)
2. -target armv7-unknown-linux-gnu -mfloat-abi=soft (this is deprecated and shouldn't be used for modern targets)
3. -target armv7-unknown-linux-gnu -mfloat-abi=hard (lets just engorge ourselves in hatred for ourselves)
4. -target armv7-unknown-linux-gnueabi -mfloat-abi=soft (this is extremely common still)
5. -target armv7-unknown-linux-gnueabi -mfloat-abi=hard or -target armv7-unknown-linux-gnueabihf (many distributions are starting to support this, yes, they are subtly different, lets just ignore that and be friends?)
6. -target armv7-unknown-linux-eabi -mfloat-abi=soft (embedded environment, non-glibc or no libc)
7. -target armv7-unknown-linux-eabi -mfloat-abi=hard or -target armv7-unknown-linux-eabihf (embedded environment, non-glibc or no libc, hard float)
8. -target armv7-unknown-linux-musleabi (musl libc instead of glibc)
9. -target armv7-unknown-linux-musleabihf (musl libc instead of glibc, hard float)

They are all ARMv7 variants, all of which should be part of the same SDK, but we have no possible way to organise these currently. There are at least four options that I can come up with:

1. make the SDK more of a swiftroot - organised like a sysroot with a layout like usr/<target>/lib
2. make the SDK more like a SDK and expand the <arch> directory to <arch>-<environ>
3. make the SDK more like a SDK and add an <environ> component under the <arch>
4. make the SDK target/environment dependent and require different SDKs for each of the architecture/environment combination (at the cost of the combinatorial explosion of SDKs)

My personal preference would be the third of changing the layout to support this with multiple architecture/environment variables for each platform in a single SDK. This would make cross-compilation to different architectures extremely easy and convenient with only needing a single SDK per platform. At that point, I believe that we will have enough flexibility to address the multitude of Linux environments as well as android environments and we would be easily able to cross-compile swift code for a variety of targets.

This still leaves the fact that we need two separate roots - one for the Swift SDK and one for the system SDK which is needed for the C library, math library, threading library, etc. I think that we should cascade the search for the system libraries through the SDK enabling someone to create a SDK for a distribution which contains the system libraries, but does not require it so that it is possible to be able to target systems which may not be fully under our control.

Saleem

CC: @jrose @Michael_Gottesman @millenomi @graskind

[lukasa (Cory Benfield)]

CC: @johannesweiss @kevints

[jrose (Jordan Rose)]

I have such a different perspective on this, and a lot of it centers around this sentence in the intro:

Starting off with the original state of the world, the standard library and the compiler builds were intertwined completely, the swift (SDK) content being placed into the compiler resource directory and this path to relative to the compiler.

"SDK", as used by Apple, has always been a sysroot; it has never been something that you use on top of your base system, or against a separate sysroot, or with "other SDKs" (third-party frameworks calling themselves SDKs notwithstanding). Apple certainly didn't invent the term "SDK", but our use of -sdk to replace --sysroot / -isysroot when designing the Swift driver interface wasn't an accident. Unfortunately, it doesn't match up with how, say, Microsoft uses it for Windows or (?) Google for Android.

So, my suggestion is that we have this discussion without using the word "SDK". The important point is that there's different sets of content that the compiler needs to be able to find, and that that content is necessarily not distributed together.

On @compnerd's recent PR, we listed out several things the compiler uses and tried to classify them (before deciding to take it to the forums for broader discussion). Here's an updated version of that list based on what I found in Xcode 11 and the 5.0 release toolchain:

	Platform-dependent?	Compiler-dependent?	Current location in official builds	Classification	Explanation
Standard library	Yes	If no stable ABI	Sysroot in Xcode (as of Xcode 11), per-platform compiler resource dir otherwise	Platform Swift content	Needed for both compilation (the Swift module / interface) and linking (so / dylib / TBD)
Static standard library	Yes	If no stable ABI	Per-platform next to the compiler resource dir	Platform Swift content	Not supported on Apple platforms anymore. We should move this into the resource dir, at least for now…
Stdlib "shim" headers	Can get away with the same content	No	Top-level compiler resource dir, but everyone agrees it belongs with the stdlib	Platform Swift content	C headers used to support the stdlib in various ways
Overlay libraries	Yes	If no stable ABI	Sysroot in Xcode (as of Xcode 11), per-platform compiler resource dir otherwise	Platform Swift content	Swift augmentation of C headers in existing sysroot
Overlay "shim" headers	Can get away with the same content	No	Top-level compiler resource dir, but everyone agrees it belongs with the overlays	Platform Swift content	Hopefully will become implementation-only and thus disappear from distributions altogether
Clang resource directory	*	Yes	Top-level compiler resource dir	Compiler content	Contains both headers and static runtime libraries
Swift runtime libraries / objects	Yes	Yes	Per-platform compiler resource dir	Compiler content	Includes the new backwards-deployment support libraries on ABI-stable platforms
System headers	Yes	No	Sysroot	Platform non-Swift content	We usually don't control the packaging of these
System modules and API notes	Yes	No	Sysroot in Xcode, per-platform compiler resource dir elsewhere	Platform Swift content	Note that Apple platforms have some bonus API notes in the compiler resource dir, but I hope those will go away
SwiftPM resources	No	If no stable ABI	Top-level compiler resource dir	Compiler content	Used to parse Package.swift files, so run on the current machine rather than the one you're compiling for
CF and CDispatch headers	Can get away with the same content	No	Top-level compiler resource dir	Platform Swift content	Non-Apple platforms only. Ideally these would eventually go away completely behind implementation-only imports, but we're not sure we can pull that off in practice (especially Dispatch)
Migrator info	Yes	May also depend on compiler changes	Top-level compiler resource dir	?
Backwards-deployment layouts	Yes	No	Per-platform compiler resource dir	Platform Swift content	Only relevant for ObjC interop on Apple platforms
Prebuilt module cache	Yes	Yes	Per-platform compiler resource dir	?	Stable ABI platforms only. Faster access to module interfaces found in the SDK, er, sysroot. Not semantics-breaking to remove, but the stdlib is particularly slow to import otherwise.

I left off a bunch of things too: LLDB resources, header files for using the toolchain, ICU install resources, and a bunch of empty folders that probably shouldn't go into our toolchains, and some Xcode-only stuff to support CreateML that'll move into the Apple SDKs soon.

From this list, I do see a pretty clear separation between what I called "platform Swift content" and what I called "Compiler content". I think my "platform Swift content" maps to @compnerd's "SDK", too. But we've got a few holes even if we could magically move everything to a new layout:

• We want to be able to test new stdlibs on Apple platforms, which means swift.org downloads need to win over the Xcode SDK, er, sysroot-with-Swift content. (But we need to find things in both places.)

• But a cross-compile should win over the swift.org download, since any host resources aren't going to be relevant.

• The migrator info and the prebuilt module cache are based on both the compiler and the platform Swift content (and the non-Swift platform content as well).

• Ideally the downloads from swift.org would be self-contained: one folder with everything you need and no extra configuration.

• As @compnerd points out, "platform" is important but in a lot of cases you need full-on distinction by target triple. Apple platforms are the odd ones out with fat binaries and somewhat shared interfaces.

• Nearly all of the "platform Swift content" except for the Apple backwards-deployment layout files can be handled by normal search paths if need be.

Given this last point, I think it's worth investigating whether we can avoid introducing a new concept of "SDK" and instead use normal search paths for "platform Swift content that should override the resource dir". That's roughly what we have implemented, which is convenient: a search order of "explicit user search paths > compiler resource dir > sysroot". (But it does mean passing both -I and -L and is at odds with us ever getting rid of -I being both Swift and Clang search paths.)

Some alternatives in the "search path" solution space:

• Use frameworks for platform content, because then you just need one path, -F. Isn't really the intended model for module maps or swiftrt.o, though.
• Come up with a new concept (maybe it's called "SDK", maybe something else) for "a bundle of Platform Swift Content". (Can you have more than one of them?)

[compnerd (Saleem Abdulrasool)]

@jrose

So, my suggestion is that we have this discussion without using the word "SDK".

You have a good idea here that using "SDK" is this discussion is confusing and we should aim to avoid the term.

@jrose

"SDK", as used by Apple, has always been a sysroot;

Wait, this is at odds with what was on the PR - since the path contained the platform name duplicated:
/Developer/Platforms/<PLATFORM>.platform/Developer/SDKs/<?>.sdk/usr/lib/swift/<PLATFORM>/<ARCH>.
The desire here is to remove the second platform since it is repetitive (it obviously is the SDK for that platform.

@jrose

Use frameworks for platform content, because then you just need one path, -F . Isn't really the intended model for module maps or swiftrt.o, though.

I would love to do this, but, I don't think it is entirely possible. The platform may not normally package items in frameworks, and I don't think that we should have a solution that requires repackaging the existing platform content. For the Swift libraries, yes, we could absolutely go with this (sidenote, this is already well tested by Foundation).

I do think that you covered the full set of content categories. I think that one tweak we should apply: we should consider "native" compilation to be a degenerate case of cross-compilation. The distinction is largely unimportant/uninteresting and treating it the same means that we have fewer paths to worry about.

Is my understanding correct here that this should be the order:

• User Specified -[flag]
• Toolchain embedded path (/Applications/Xcode.app/Contents/Developer)

Sounds like the migrator info and prebuilt module cache should also order the same way as the previous case, and there is nothing to consider here.

Yes, ideally, the swift.org content would be self-contained, but I don't believe that it is: it does not include the platform content (i.e. libSystem headers, tbd, etc).

The full target triple is need to do the distinction but that seems like it can be contained entirely in the structure layout? Rather than doing /usr/lib/swift/<platform>/<arch> we can easily do /usr/lib/swift/<normalized triple> and solve that it seems (there are alternative approaches: components of the triple form directory structure, but that seems overly deep for no gain. Am I missing something?)

Why can the Apple backwards deployment layout files not follow the same search order?

@jrose

Given this last point, I think it's worth investigating whether we can avoid introducing a new concept of "SDK" and instead use normal search paths for "platform Swift content that should override the resource dir"

Hmm, I'm torn. It is way easier as a developer and way more flexible as a user. At the same time, I really think that the convenience of a single -[flag] option is something that many users would like. I also am somewhat weary of the flexibility being too much - consider the ordering of -I, -isystem, -resource-dir, -isystem-cxx. It is a mess, and having the management of the search order be user controlled seems like we would be stuck with that permanently.

[jrose (Jordan Rose)]

@compnerd

Wait, this is at odds with what was on the PR - since the path contained the platform name duplicated:
/Developer/Platforms/<PLATFORM>.platform/Developer/SDKs/<?>.sdk/usr/lib/swift/<PLATFORM>/<ARCH> .
The desire here is to remove the second platform since it is repetitive (it obviously is the SDK for that platform.

That's just not correct. The path where the stdlib is found inside an Apple SDK is $SDKROOT/usr/lib/swift/, not $SDKROOT/usr/lib/swift/$PLATFORM/ (and certainly not $SDKROOT/usr/lib/swift/$PLATFORM/$ARCH, which we're not even using in the resource dir anymore for Apple platforms).

@compnerd

The platform may not normally package items in frameworks, and I don't think that we should have a solution that requires repackaging the existing platform content.

Agreed! I was going off your idea of saying the non-Swift platform content is the "sysroot" (even if it's not exactly a root, like in the Windows case), and wondering if we could get all the Swift content into normal frameworks. It definitely feels weird to do that for things like Glibc.modulemap, though.

@compnerd

I think that one tweak we should apply: we should consider "native" compilation to be a degenerate case of cross-compilation. The distinction is largely unimportant/uninteresting and treating it the same means that we have fewer paths to worry about.

I don't quite agree here. I'd be happy to say that if anything lives in the compiler resource dir and it matches the current platform, it's for native compilation. That fits the story better for swift.org downloads having everything you need.

@compnerd

Is my understanding correct here that this should be the order:

• User Specified -[flag]
• Toolchain embedded path ( /Applications/Xcode.app/Contents/Developer )

No. -sdk is how you compile for anything on macOS (whether for the host or for another Apple platform), but the toolchain should still win if it's a downloadable toolchain from swift.org. Otherwise you wouldn't be able to test new stdlibs with macOS or iOS.

@compnerd

The full target triple is need to do the distinction but that seems like it can be contained entirely in the structure layout? Rather than doing /usr/lib/swift/<platform>/<arch> we can easily do /usr/lib/swift/<normalized triple> and solve that it seems (there are alternative approaches: components of the triple form directory structure, but that seems overly deep for no gain. Am I missing something?)

This seems fine for non-Apple platforms, and it'd solve the minor problem of "I'm compiling for Linux but it's a different Linux", though at the cost of making it harder to use the downloadable builds of Swift on non-Ubuntu platforms that are "close enough". (For Apple and any other platforms with fat binaries it's important to keep the dylibs as multi-architecture, and then we might as well use the multi-arch swiftmodule directories as well.)

@compnerd

Why can the Apple backwards deployment layout files not follow the same search order?

Yeah, ignore these, they can move into the SDKs wholesale. Let's come up with the long-term plan.

---

I almost like the "explicit -resource-dir, then SDK/sysroot, then ../lib/swift" plan, but I don't see how it fits in with "download this toolchain for macOS and try it out", where we need the compiler's default resource dir to win over the SDK. I suppose we could give up on all-in-one configurations for cross-compiles compiles not supported by the platforms in ../lib/swift and always require an explicit resource dir in those cases. What do you think?

cc also @alblue, who's been interested in this in the past

[compnerd (Saleem Abdulrasool)]

@jrose

That's just not correct. The path where the stdlib is found inside an Apple SDK is $SDKROOT/usr/lib/swift/ , not $SDKROOT/usr/lib/swift/$PLATFORM/ (and certainly not $SDKROOT/usr/lib/swift/$PLATFORM/$ARCH , which we're not even using in the resource dir anymore for Apple platforms).

I think that we are both saying the same thing: we want $SDKROOT/usr/lib/swift/<ARCH> rather than $SDKROOT/usr/lib/swift/<PLATFORM>/<ARCH>.

@jrose

Agreed! I was going off your idea of saying the non-Swift platform content is the "sysroot" (even if it's not exactly a root, like in the Windows case), and wondering if we could get all the Swift content into normal frameworks. It definitely feels weird to do that for things like Glibc.modulemap, though.

compnerd:

Oh, I see what you mean. Actually, that sounds like a really interesting idea. If we do this for just the Swift content, this could be totally viable. It would simplify things and actually be really easy to compose as well.

@jrose

No. -sdk is how you compile for anything on macOS (whether for the host or for another Apple platform), but the toolchain should still win if it's a downloadable toolchain from swift.org. Otherwise you wouldn't be able to test new stdlibs with macOS or iOS.

This is what I was worried about. How do we differentiate when it should go one way rather than the other? I suppose that we could do something like -nostdlib/-nostdinc++ to explicitly opt into this mode. That also makes sense to me from the perspective that this is specifically for testing and so you are explicitly specifying that you want the testing approach.

@jrose

we might as well use the multi-arch swiftmodule directories as well

Yeah, I think that changing the other platforms to the same layout makes sense.

@jrose

I suppose we could give up on all-in-one configurations for cross-compiles compiles not supported by the platforms in ../lib/swift and always require an explicit resource dir in those cases.

I'm not so keen on this. The resource dir should be largely platform agnostic. Especially given the way that the content is split up. Could you remind me again what the content in the resource directory do we need to give precedence and in what case you would have something that is platform dependent in the resource directory?

[jrose (Jordan Rose)]

@compnerd

I think that we are both saying the same thing: we want $SDKROOT/usr/lib/swift/<ARCH> rather than $SDKROOT/usr/lib/swift/<PLATFORM>/<ARCH> .

Hm. Why the <ARCH>? It's an anti-goal for Darwin, but is it really the case that an SDK is going to be multi-architecture anywhere else?

@compnerd

This is what I was worried about. How do we differentiate when it should go one way rather than the other? I suppose that we could do something like -nostdlib / -nostdinc++ to explicitly opt into this mode. That also makes sense to me from the perspective that this is specifically for testing and so you are explicitly specifying that you want the testing approach.

Yeah, we might be able to live with that on the command line, and have Xcode do it by default. Weird for, say, SwiftPM though.

@compnerd

we might as well use the multi-arch swiftmodule directories as well

Yeah, I think that changing the other platforms to the same layout makes sense.

Oh. I don't, actually. I think the directory layout makes sense when you have fat libraries but not otherwise.

@compnerd

I'm not so keen on this. The resource dir should be largely platform agnostic. Especially given the way that the content is split up. Could you remind me again what the content in the resource directory do we need to give precedence and in what case you would have something that is platform dependent in the resource directory?

The main reason here is in service of the "you download one thing from swift.org and it works" goal. I don't see why you're against having platform-dependent content in the resource directory if we can figure out how to not have it get in the way. Maybe we should mark it more clearly, though (like moving the headers into the platform directories even though they'll always be the same).

[compnerd (Saleem Abdulrasool)]

@jrose

Hm. Why the <ARCH> ? It's an anti-goal for Darwin, but is it really the case that an SDK is going to be multi-architecture anywhere else?

Yes, I would like for it to be multiarch everywhere. I think that I would say that for Darwin, I would prefer that we do something slightly silly - have symlinks that point back to the fat binary. If that is not really desirable, then we can just avoid the on Darwin targets. I don't think that is too big of a deal.

@jrose

we might as well use the multi-arch swiftmodule directories as well

Yeah, I think that changing the other platforms to the same layout makes sense.

Oh. I don't, actually. I think the directory layout makes sense when you have fat libraries but not otherwise.

The problem is that the module actually is tied to the architecture, and it would be desirable to support multi-architecture layouts for non-Darwin targets. I'm not tied to the directory approach. I'm just as happy with the <ARCH>/<MODULE>.swiftmodule approach. I think I actually prefer the latter. This was to avoid the two different code paths really. I would say that this is in the same category as the <ARCH> layer in the first point.

@jrose

if we can figure out how to not have it get in the way. Maybe we should mark it more clearly, though (like moving the headers into the platform directories even though they'll always be the same).

I think that marking it this way is enough to satisfy my issue with this.

[jrose (Jordan Rose)]

I guess I really want a concrete example: what platforms have non-fat-binary multi-arch SDKs, and what are the arch's?

[compnerd (Saleem Abdulrasool)]

Well, if you want a shipping example - Windows. The Windows SDK is a multi-architecture non-fat-binary SDK. It targets ARM, ARM64, X86, X64 architectures.

If you want a Linux example - exherbo Linux does multi-architecture on a single root - so any installation can be ARMv7 (hf/softfp, glibc or musl) , ARM64 (glibc or musl), x86, x86_64 (or any subset).

[jrose (Jordan Rose)]

What does that look like? …oh, is it the "lib64" thing, different link paths for different targets? Okay, I'd forgotten about that.

So, hm. Darwin is definitely not going to use $SDKROOT/usr/lib/swift/<ARCH> because it uses fat libraries. But we could do that everywhere else, and make it conditional on the use of fat libraries. Or we could search both paths (we already have a way to search multiple paths in an SDK because of Mac Catalyst, which has both $SDKROOT/usr/lib/swift/ and $SDKROOT/System/iOSSupport/usr/lib/swift/).

So, to summarize:

• Explicit -resource-dir wins over -sdk wins over toolchain, except when, say, -toolchain-stdlib is passed.
• -sdk continues to mean "the place where headers and libraries are", not "the Swift content"
• -resource-dir gets smart enough to know whether it needs to append a platform component or not, so people don't have to think about it.
• The Clang resource directory and the SwiftPM info becomes compiler-relative always (the latter we don't even really have a choice about, since it's really swift-package-relative). Other resources are found relative to -resource-dir if explicitly passed.

Does this handle all our use cases?

---

@Ben_Cohen should probably be brought in at this point too, since he's been thinking about how to do testing snapshots going forward. This -toolchain-stdlib idea neatly handles the "do we distribute the stdlib with the compiler" question, if we can get it all to work with Xcode.

[compnerd (Saleem Abdulrasool)]

I don't think that this works very well. The split layout needs to be considered - that is what started this thread in the first place. What you have described is pretty much what exists today and only suffices for Darwin, not the other platforms. We need to have a nice way to address the case of system headers and libraries and swift pieces are separate (Windows and android both need this). Additionally, both of them are furthermore interesting with libraries and headers being in different locations. We need a way to consolidate the various approaches.

[jrose (Jordan Rose)]

Right, I thought that was -resource-dir.

[compnerd (Saleem Abdulrasool)]

No, the resource dir is the compiler resource directory, which contains the path to the SwiftShims, the clang resource directory for the clang importer. sdk can be the flag which contains the path to the Swift standard library, runtime support (swiftrt.obj), and additional Swift core libraries but that still leaves the piece of the system headers and libraries which are in different locations needing to be cobbled together a separate flags. The standard library and core libraries can be built separately, and we need a way to cascade the search through the two.

[jrose (Jordan Rose)]

sdk can be the flag which contains the path to the Swift standard library, runtime support ( swiftrt.obj ), and additional Swift core libraries but that still leaves the piece of the system headers and libraries which are in different locations needing to be cobbled together a separate flags.

I guess I shouldn't have tried to reintroduce the word "SDK" just yet. I still think "SDK" should be "the thing the platform provides", not "the place with Swift content".

SwiftShims goes with the stdlib, not with the compiler. We just haven't made that happen yet. It's only the Clang resource dir that's really tied to the compiler, and AFAICT there's no reason for that to ever not be found relative to the compiler.

[compnerd (Saleem Abdulrasool)]

An example of how this can possibly work:

-resource-dir S:\b\swift\lib -sdk S:\b\Library\Developer\Platforms\android.platform\Developer\SDKs\android.sdk -Xclang --sysroot=C:\Microsoft\AndroidNDK64\android-ndk-r16b\sysroot -Xclang-linker --sysroot=C:\Microsoft\AndroidNDK64\android-ndk-r16b\platforms\android-21\arch\arm

[compnerd (Saleem Abdulrasool)]

Okay, if -sdk remains the thing that the platform provides, that is fine. Lets work that out first.

In the case of android, for compilation, you need the compilation sysroot, e.g. C:\Microsoft\AndroidNDK64\android-ndk-r16b\sysroot, and for linking you need the linking sysroot, e.g. C:\Microsoft\AndroidNDK64\android-ndk-r16b\platforms\android-21\arch-[arm|x86|arm64].

In the case of Windows, for compilation (lets assume ARM), you need the compilation roots, e.g. -I C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\include\14.22.27905\include -I C:\Program Files (x86)\Windows Kits\10\10.0.18362.0\ucrt -I C:\Program Files (x86)\Windows Kit\10\10.0.18362.0\shared -I C:\Program Files (x86)\Windows Kits\10\10.0.18362.0\um and for the linking you need the linking roots, e.g. -L C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.22.27905\lib\arm -L C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\ucrt\arm -L C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\um\arm

For Linux, well, lets select a distribution, lets say exherbo. Then, lets select an architecture, lets say ARMv7. Then lets select a FP ABI, lets say hard float. Next, lets select a libc, lets say musl. Then, because its exherbo, its really simple: you need the sysroot: /usr/armv7-unknown-linux-musleabihf. Ubuntu does something similar as well.

This effectively boils down to the following:

Android:

[compile]

• <NDKRoot>\sysoot

[link]

• <NDKRoot>\platforms\android-<PlatformLevel>\arch-<PlatformArch>

Windows:

[compile]

• <MSVCRoot>\include\<MSVCVersion>\include
• <WinSDKRoot>\Include\<WinSDKVersion>\ucrt
• <WinSDKRoot>\Include\<WinSDKVersion>\shared
• <WinSDKRoot>\Include\<WinSDKVersion>\um

[link]

• <MSVCRoot>\<MSVCVersion>\lib\<Arch>
• <WinSDKRoot>\Lib\<WinSDKVersion>\ucrt\<Arch>
• <WinSDKRoot>\Lib\<WinSDKVersion>\um\<Arch>

Linux:

[compile]

• --sysroot=/usr/<triple>

[link]

• --sysroot=/usr/<triple>

Additionally, all the content in these paths are effectively non-modifiable, and possibly disjoint from the Swift standard library and core libraries.

[jrose (Jordan Rose)]

@compnerd and I met in person to discuss this last week, along with @alblue, @johannesweiss, @kevints, @beccadax, @millenomi, and @Rostepher. We discovered we're in agreement about nearly everything, and that most of the remaining questions are around terminology (i.e. "we know what we want to do, but what do we call the things and how do you write it on the command line?").

(Please correct me on anything I've misremembered, misrepresented, or left out!)

Groundwork

We agreed on some starting goals:

1. The Swift compiler should have first-class support for cross-compilation.
2. Open-source toolchains must continue to work with Xcode.
3. Using Swift must not require modifying any platform content, either in / or in Xcode.app or in a development package you got from Microsoft.
4. (I added this one late) There should be downloads on swift.org where you download one thing and then swift run works on a package you just checked out.

During his work on Windows, @compnerd broke down "the stuff we ship on swift.org" into four components:

• Toolchain: The minimum set of tools needed to compile a Swift program. In its fullest form this would include swiftc, clang, lld, and llvm-ar, providing full multi-platform compile-and-link support.

• Swift-based tools: SwiftPM, SwiftSyntax, SourceKit-LSP, and anything else that's used for Swift development that requires a Swift runtime itself. (It's a little unfortunate SwiftPM is in this category since that's how many Swift libraries are distributed, but, well, SwiftPM's written in Swift, so we can't get around it.)

• Swift compile-time dependencies: This includes whatever is needed to compile and link a Swift program for a specific platform. For example, the Swift stdlib swiftmodule would be in here.

• Swift run-time dependencies: This is the content that needs to be present to run a Swift program for a specific platform. On Apple platforms, for example, this is installed in the OS by default; for Windows you'd distribute it alongside your app; and on Linux...well, it's complicated and we haven't really figured it out yet.

Making Goal 4 work requires putting all of these in one package, but that doesn't have to be the only way to distribute them. In particular, cross-compilation should work with your existing dev tools and toolchain, and only rely on swapping out your compile-time and run-time dependencies.

Okay, so we all agree on the goals. How do we get there?

System content

At the start of the meeting we started to list out which content was distributed where, for both Linux and Apple platforms. We then started discussing the "system" content for both Android and Windows, and how those differ from Linux and Apple's nice, neat "sysroot" model. (This is what @compnerd's previous post covered.) The key insight from this discussion was that each of these platforms has one or more directories with known layouts that can be turned into search paths based on the -target you're compiling for, possibly with some extra information gleaned from other command-line arguments or environment variables. (For Windows, the WinSDK version is typically an environment variable.)

Proposal: swiftc should accomodate multiple system directories, and automatically add appropriate search paths within them (the same way -sdk foo.sdk/ searches foo.sdk/System/Library/Frameworks/ on Apple platforms). We'll have to audit how else SDKPath is used today in the compiler and where it's assumed to be unique, but we think it'll work.

Swift compile-time content

For Apple platforms, the system directory (what Xcode calls an SDK) also contains the cross-compilation Swift content for that platform (starting in Xcode 11, anyway). But other platforms don't have Swift included ahead of time. Where do they get that information? It must be an external download, which would contain:

• The stdlib, obviously
• Probably the corelibs as well
• Anything needed to massage the system content into a good shape (module maps, API notes, maybe more?)
• Additional config files for search paths and such (CMake modules, pkgconfig, SwiftPM's destination.json)

During the meeting we called this the cross-compilation support directory. It's also roughly the "Swift compile-time dependencies" mentioned above. To cross-compile, then, you should be able to point the compiler at a particular system directory (or set of directories) and a particular cross-compilation support directory.

Proposal: This directory should have the same layout as the system directories, so that we can use the same mechanism described in the previous section for the Swift content as well as the system content. This also implicitly handles the default Xcode case, where the two directories are the same.

Multi-arch cross-compilation support

It would kind of stink if we had to have separate cross-compilation support directories for each ABI we wanted to support—that would duplicate a lot of content needlessly. However, we already have the notion of ABI-specific subdirectories in the system content directories for link dependencies (see @compnerd's examples above). Is that enough for the Swift content as well, if laid out in the same way? We can certainly put libraries-for-linking in such a layout, so it's "just" the swiftmodules and other content that matters.

Proposal: Allow the cross-compilation support directories to use the multi-target layout for swiftmodules, like Apple platforms already do. Also, don't bother encoding differences that wouldn't result in exposing different Swift API, like different libc implementations or hard-float / soft-float.

usr/
  lib/
    swift/
      Swift.swiftmodule/
        armv7-unknown-linux.swiftdoc
        armv7-unknown-linux.swiftmodule
        x86_64-unknown-linux.swiftdoc
        x86_64-unknown-linux.swiftmodule
  armv7-unknown-linux-eabi/
    lib/
      swift/
        libswiftCore.so
  armv7-unknown-linux-eabihf/
    lib/
      swift/
        libswiftCore.so
  x86_64-unknown-linux/
    lib/
      swift/
        libswiftCore.so

(exact directory layout not proposed; also forgive me for my fake Linux link ABI names)

Host == build

Okay, so we've accomplished goal 1 ("first-class cross-compilation support") and goal 3 ("without modifying system content directory"). But can we pack all the content into one downloadable thing so that people can get started with swift run easily? That is, can we have a default search path that's compiler-relative when not cross-compiling?

We didn't quite converge to a proposal for this, but I think we ended with two main options:

1. Assume that the compiler is in $DIR/usr/bin/ inside a cross-compilation support directory $DIR. In order for this to work for Xcode, we'd have to allow $DIR/usr/lib/swift/$PLATFORM as a standard search path as well as the sensible $DIR/usr/lib/swift/. (This is roughly what we have implemented today, just cleaned up if we want.)

   # Apple toolchain
   usr/
     bin/
       swiftc
     lib/
       swift/
         macosx/
           libswiftCore.dylib
           Swift.swiftmodule/
         iphoneos/
           libswiftCore.dylib
           Swift.swiftmodule/
   

   # Linux toolchain - x86_64 only
   usr/
     bin/
       swiftc
     lib/
       swift/
         linux/
           libswiftCore.so
           Swift.swiftmodule
   

2. Put entire cross-compilation support directories somewhere relative to the compiler, and have the compiler find them. This is more uniform but probably more work to move everything around, and might be more disruptive for external build systems that have come to depend on Swift's toolchain layout.

   usr/
     bin/
       swiftc
     lib/
       swift/
         macosx/
           usr/
             lib/
               swift/
                 libswiftCore.dylib
                 Swift.swiftmodule/
         iphoneos/
           usr/
             lib/
               swift/
                 libswiftCore.dylib
                 Swift.swiftmodule/
   

Next steps

• Agree on a name for the stackable platform directories. I personally think we should stick with -sdk and just allow stacking SDKs, but we didn't focus on resolving this in the meeting.

• Decide on how to lay out the non-cross downloads.

• Figure out exactly what it'll take to make downloadable Xcode toolchains prefer their stdlib over the one in Xcode (to preserve goal 2).

• Get everything in the compiler currently using -resource-dir to look in stacked SDKs instead, unless it really is tied to the compiler. (That's mostly just Clang's intrinsic headers.)

• Actually implement the stackable SDK thing and all the search paths that go with it.

• Consider moving or symlinking the Swift libraries for the Swift-based Tools out of the platform-specific subdirectory; they need to be found relative to those tools anyway.

• (Apple folks) Deal with how this all interacts with Catalyst. (To this end it would be great to hold off on a bunch of this work until we Apple folks can upstream the Catalyst support, at least the work we had to do for search paths. Trust me, it will make a bunch of things easier.)

Future work

• Consider moving the C dispatch implementation to usr/{include,lib}/ within the cross-compilation support directory instead of usr/lib/swift/ (at least on Linux). There's no real reason to do this besides consistency, but, well, consistency.

• Consider using symbol versioning on ELF to defend against mixing unstable Swift ABIs. (Someone needs to go off and investigate this.)

• Decide if the Swift toolchain's copy of Clang and the other LLVM tools should be prefixed (swift-clang) to not conflict with platform tools. (Discussed somewhat in Shipping clang with the Swift toolchains, but with no action taken yet.)

• Anything to do with installing a Linux toolchain into /, or even multiple Linux runtimes.

Whiteboard

[compnerd (Saleem Abdulrasool)]

Thanks for the write up @jrose. I believe that all the points discussed are covered.

Do you have a rough estimate of when the changes for catalyst would be merged? I would rather get some sense of the time line so that progress can be made here. If the time horizon is on the order of months, that may be a bit too long of a wait I think (although I may be misunderstanding the suggestion on the wait and it applies to a specific portion?). Some of the items might be able to be executed upon in the mean time, e.g. restructuring the swiftmodule layout, and shouldn't really be impacted by the search order changes, so it should be possible to make some progress there in the mean time.

[JetForMe (Rick M)]

This may be out of place here, but skimming through this discussion, I'm reminded of an issue I come up against a lot when developing for iOS: building a universal library (framework or module or just a .a with headers) that let my Xcode iOS project build binaries for both Device and Simulator. Apple treats these as separate platforms, making it quite tedious. As a developer, I want to be able to seamlessly switch between a Device and the Simulator targets, and not have to go rebuild third-party dependencies I may have. I want to just drop them into my project as a monolithic entity (e.g. Framework or .a) and not sweat the implementation details.