Misplacement of runtime libraries on Linux?

stevapple · 2022-03-12T16:44:36.808Z

As we all know, Swift runtime libraries should vary with arch…

…But they’re placed in arch-unrelated directories /usr/lib/swift/(linux|android|freebsd|…) instead of /usr/lib/swift/{os}/x86_64.

I feels like the initial design of Swift compiler, which is based on macOS, gains out-of-the-box support for fat binaries (which is called Universal now). However, this is not true for most of the platforms. I believe @compnerd has noticed it when designing Swift on Windows layout, so Windows users have the (import) libs in {SDKROOT}/usr/lib/swift/windows/x86_64.

I believe this can be fixed with ease (because both directories are in the search path), but I would like to hear any feedback on this change.

Finagolfin · 2022-03-13T16:58:35.364Z

@compnerd raised this and other linux SDK layout issues a couple years ago, and I think some of those have since been fixed but not this one. I agree that we should place the Swift libraries in architecture-specific directories on all ELF platforms, since they don't have fat libraries.

I don't know if the Swift compiler already links against that architecture-specific path at compile-time as you say, but it certainly doesn't add that architecture-specific path to the RUNPATH invoked at runtime, so that will need to be changed too. This layout issue is the main reason I distribute three separate architecture-specific Android SDKs, rather than rolling them into one.

@3405691582, do you agree for OpenBSD too?

3405691582 · 2022-03-14T14:02:28.852Z

I would need to refresh my mental cache a little about this, so this might be slightly inaccurate, but I do remember that glibc's dynamic loader will automatically search architecture-named subdirectories of RPATH, however other platforms' loader will not, so to maintain portability any changes have to be mindful of that fact.

It doesn't seem like OpenBSD has a kind of multiarch solution, so I don't know if subdirectories makes sense -- an off the cuff suggestion might be to use the entire platform triple, but that may not be standardized.

stevapple · 2022-03-17T03:19:44.448Z

@Finagolfin I opened apple/swift#41850 for testing this, could you build a Linux toolchain with CI for further investigation? I haven't got CI access yet.

ktoso · 2022-03-17T07:11:42.623Z

I guess folks won't mind if I kick off a toolchain build for you there -- done.

stevapple · 2022-03-17T14:08:54.683Z

I will instead suggest we hardcode that in RPATH for future releases. Currently none of these platforms are ABI stable, so you’ll never have to get the program work with a lower version runtime.

To maintain some basic compatibility, the legacy (non-suffixed) path will still be searched, and if the libraries are there, the linker behavior will be exactly the same as it is now.

Michael_Gottesman · 2022-03-18T20:09:41.307Z

Whatever we do here... lets be sure not to rush our discussion here. For changes like this in the build we have to be careful and think through it. I think it is important to get @compnerd's perspective and also I am going to need a bit of time to actually engage with this and page back in all the information so I can give the best information here.

One other thing to consider is that this may make differences in between the Linux/Darwin/Windows build. That may require us to introduce a bunch of code to solve this in the short term into a build that is already problematic to say the least. I would need to remember how this all fits together though before I commit on this further. Regardless, even if we think in the short term that this would be disruptive, at minimum, we need a long term plan if this is the correct layout. My long term vision for Swift in this area is that it should not be special on any platform, we should follow the platform defaults as much as possible... that being said we have to balance that against complexity in the build.

Michael_Gottesman · 2022-03-21T18:28:32.116Z

So I talked with a few people/@compnerd and my memory was correct: the reason for the current state of the world is due to the amount of work it would take in the build to fix this problem. Instead compnerd at the time just kept everything the same across all platforms and taught the driver how to find libraries in these locations.

That says to me this is probably a bit of work in the build system that may not be worth investing into at this point in time.

stevapple · 2022-04-18T02:10:38.900Z

Michael_Gottesman:

One other thing to consider is that this may make differences in between the Linux/Darwin/Windows build. That may require us to introduce a bunch of code to solve this in the short term into a build that is already problematic to say the least.

Linux and Windows SDK layout should be aligned regarding runtime libraries. That is, if a platform supports fat binary, it can use the fat library under /usr/lib/swift/{platform}; if it don’t (or we just don’t want a fat binary), it can use the one under /usr/lib/swift/{platform}/{arch}. So by this time every platform should share the same build step, and this work will eventually reduce the unnecessarily duplicated code shared by Linux and Windows CMake scripts.

Michael_Gottesman:

Regardless, even if we think in the short term that this would be disruptive, at minimum, we need a long term plan if this is the correct layout. My long term vision for Swift in this area is that it should not be special on any platform, we should follow the platform defaults as much as possible... that being said we have to balance that against complexity in the build.

I agree that we need to consider more here, but it comes out that even /usr/lib/swift/linux itself is not regular to Linux and we’re now using RPath and custom search path to deal with it. Using the same technique for /usr/lib/swift/linux/x86_64 is acceptable IMO.

After ABI stability, I would suggest symlinking host runtime libraries into /usr/lib and then use @rpath/*.so for primary search directory instead.

stevapple · 2022-04-18T02:11:18.363Z

SDK (and what it makes possible — cross-compilation) is definitely something requiring joined efforts from multiple persons as well as communities. I’m working on an initiative of a related workgroup, and would like to hear your opinions first from everyone who, I know, is already distributing toolchains or SDKs (possibly on behalf of his/her community or company).

Overall: @compnerd @Michael_Ilseman
Android: @Finagolfin
Darwin: Apple
Linux: @futurejones @tomerd
Wasm: @Max_Desiatov
Windows: @compnerd

And also from everyone who’s working on new ports, which is not so familiar for me to list though.

Finagolfin · 2022-04-18T03:27:10.232Z

The workgroup sounds like a good idea. I've been considering writing up some documentation on cross-compilation for the main Swift repo, now that my Android port is winding down.

As for moving these libraries to a new arch-specific path, I don't think it'd be much work and is something we should do, since you can't assume x86 anymore because many platforms are moving to ARM.

stevapple · 2022-09-17T09:17:49.489Z

The initiative for Swift Platform Work Group is live now!

Finagolfin · 2023-03-09T02:00:33.289Z

I've put together a pull for this, which passed the CI and produced a linux toolchain build that people can download and test out.

The advantage of this change is that packagers like @Ron_Olson, @futurejones, and me can distribute multi-arch Swift SDKs in the same Swift resource directory, rather than having to distribute separate lib/swift resource directories per architecture currently. For example, the official linux x86_64 toolchain could also contain the linux aarch64 SDK after this change, for easy cross-compilation to other architectures.

@ktoso and @0xTim, would be good to get some testing from the SSWG people. This should be a drop-in change for most everyone, as the compiler generates these paths automatically, and will only require changes from those writing toolchain-related scripts or something expecting the runtime libraries in the previous location, ie packagers like me.

@Michael_Gottesman, if you would review the pull, would like to get this in before the 5.9 branch.