[Pitch] Package Manager: Statically link Swift runtime libraries by default on supported platforms

Evolution Pitches
1

Hello folks,

Swift 5.3.1 introduced statically linking the Swift runtime libraries on Linux. With this feature, users can set the --static-swift-stdlib flag when invoking SwiftPM commands (or the long form -Xswiftc -static-stdlib) in order to statically link the Swift runtime libraries into the program.

On some platforms, such as Linux, this is often the preferred way to link programs, since the program is easier to deploy to the target server or otherwise share.

This proposal explores making it SwiftPM's default behavior when building executable programs on such platforms.

Motivation

Darwin based platform ship with the Swift runtime libraries in the dyld shared cache. This allows building smaller Swift programs by dynamically linking the Swift runtime libraries. The shared cache keeps the cost of loading these libraries low.

Other platforms, such as the standard Linux distributions, do not ship with the Swift runtime libraries.
Hence, a deployment of a Swift program (e.g. a web service built in Swift) on such platform requires one of three options:

  1. Package the application with a "bag of shared objects" (the libswift*.so files making up Swift's runtime libraries) alongside the program.
  2. Statically link the runtime libraries using the --static-swift-stdlib flag described above.
  3. Use a "runtime" docker image that contain the correct version of the the runtime libraries (matching the compiler version exactly).

Out of the three options, the most convenient is #2 given that #1 requires manual intervention and/or additional wrapper scripts that use ldd, readelf or similar tools to deduce the correct list of runtime libraries. #3 is convenient but version sensitive. #2 also has cold start performance advantage because there is less dynamic library loading. However #2 comes at a cost of bigger binaries.

Stepping outside the Swift ecosystem, deployment of statically linked programs is often the preferred way on server centric platforms such as Linux, as it dramatically simplifies deployment of server workloads. For reference, Go and Rust both chose to statically link programs by default for this reason.

Proposed solution

We propose to make statically linking of the Swift runtime libraries SwiftPM's default behavior when building executables on platforms that support such linking, with an opt-out way to disable this default behavior.

Note this does not mean the resulting program is fully statically linked - only the Swift runtime libraries (stdlib, Foundation, Dispatch, etc) would be statically linked into the program, while external dependencies will continue to be dynamically linked and would remain a concern left to the user when deploying Swift based programs. Such external dependencies include:

  1. Glibc (including libc.so, libm.so, libdl.so, libutil.so): On Linux, Swift relies on Glibc to interact with the system and its not possible to fully statically link programs based on Glibc. In practice this is usually not a problem since most/all Linux systems ship with a compatible Glibc.
  2. libstdc++ and libgcc_s.so: Swift on Linux also relies on GNU's C++ standard library as well as GCC's runtime library which much like Glibc is usually not a problem because a compatible version is often already installed on the target system.
  3. At this time, non-Darwin version of Foundation (aka libCoreFoundation) has two modules that rely on system dependencies (FoundationXML on libxml2 and FoundationNetworking on libcurl) which cannot be statically linked at this time and require to be installed on the target system.
  4. Any system dependencies the program itself brings (e.g. libsqlite, zlib) would not be statically linked and must be installed on the target system.

Detailed design

The changes proposed are focused on the behavior of SwiftPM. We propose to change SwiftPM's default linking of the Swift runtime libraries when building executables as follows:

Default behavior

  • Darwin-based platforms used to support statically linking the Swift runtime libraries in the past (and never supported fully static binaries).
    Today however, the Swift runtime library is shipped with the operating system and can therefore not easily be included statically in the binary.
    Naturally, dynamically linking the Swift runtime libraries will remain the default on Darwin.

  • Linux and WASI support static linking and would benefit from it for the reasons highlighted above.
    We propose to change the default on these platforms to statically link the Swift runtime libraries.

  • Windows may benefit from statically linking for the reasons highlighted above but it is not technically supported at this time.
    As such the default on Windows will remain dynamically linking the Swift runtime libraries.

  • The default behavior on platforms not listed above will remain dynamically linking the Swift runtime libraries.

Opt-in vs. Opt-out

SwiftPM's --static-swift-stdlib CLI flag is designed as an opt-in way to achieve static linking of the Swift runtime libraries.

We propose to deprecate --static-swift-stdlib and introduce a new flag --disable-static-swift-runtime which is designed as an opt-out from the default behavior described above.

Users that want to force static linking (as with --static-swift-stdlib) can use the long form -Xswiftc -static-stdlib.

Example

Consider the following simple program:

$ swift package init --type executable
Creating executable package: test
Creating Package.swift
Creating README.md
Creating .gitignore
Creating Sources/
Creating Sources/test/main.swift
Creating Tests/
Creating Tests/testTests/
Creating Tests/testTests/testTests.swift

$ cat Sources/test/main.swift
print("Hello, world!")

Building the program with default dynamic linking yields the following:

$ swift build -c release
[3/3] Build complete!

$ ls -la --block-size=K .build/release/test
-rwxr-xr-x 1 root root 17K Dec  3 22:48 .build/release/test*

$ ldd .build/release/test
	linux-vdso.so.1 (0x00007ffc82be4000)
	libswift_Concurrency.so => /usr/lib/swift/linux/libswift_Concurrency.so (0x00007f0f5cfb5000)
	libswiftCore.so => /usr/lib/swift/linux/libswiftCore.so (0x00007f0f5ca55000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0f5c85e000)
	libdispatch.so => /usr/lib/swift/linux/libdispatch.so (0x00007f0f5c7fd000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0f5c7da000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0f5c7d4000)
	libswiftGlibc.so => /usr/lib/swift/linux/libswiftGlibc.so (0x00007f0f5c7be000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0f5c5dc000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0f5c48d000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0f5c472000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f0f5d013000)
	libicui18nswift.so.65 => /usr/lib/swift/linux/libicui18nswift.so.65 (0x00007f0f5c158000)
	libicuucswift.so.65 => /usr/lib/swift/linux/libicuucswift.so.65 (0x00007f0f5bf55000)
	libicudataswift.so.65 => /usr/lib/swift/linux/libicudataswift.so.65 (0x00007f0f5a4a2000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0f5a497000)
	libBlocksRuntime.so => /usr/lib/swift/linux/libBlocksRuntime.so (0x00007f0f5a492000)

Building the program with static linking of the Swift runtime libraries yields the following:

$ swift build -c release --static-swift-stdlib
[3/3] Build complete!

$ ls -la --block-size=K .build/release/test
-rwxr-xr-x 1 root root 35360K Dec  3 22:50 .build/release/test*

$ ldd .build/release/test
	linux-vdso.so.1 (0x00007fffdaafa000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdd521c5000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdd521a2000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fdd51fc0000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fdd51e71000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fdd51e56000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdd51c64000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fdd54211000)

These snippets demonstrates the following:

  1. Statically linking of the Swift runtime libraries increases the binary size from 17K to ~35M.
  2. Statically linking of the Swift runtime libraries reduces the dependencies reported by ldd to core Linux libraries.

This jump in binary size may be alarming at first sight, but since the program is not usable without the Swift runtime libraries, the actual size of the deployable unit is similar.

$ mkdir deps

$ ldd ".build/release/test" | grep swift | awk '{print $3}' | xargs cp -Lv -t ./deps
'/usr/lib/swift/linux/libswift_Concurrency.so' -> './deps/libswift_Concurrency.so'
'/usr/lib/swift/linux/libswiftCore.so' -> './deps/libswiftCore.so'
'/usr/lib/swift/linux/libdispatch.so' -> './deps/libdispatch.so'
'/usr/lib/swift/linux/libswiftGlibc.so' -> './deps/libswiftGlibc.so'
'/usr/lib/swift/linux/libicui18nswift.so.65' -> './deps/libicui18nswift.so.65'
'/usr/lib/swift/linux/libicuucswift.so.65' -> './deps/libicuucswift.so.65'
'/usr/lib/swift/linux/libicudataswift.so.65' -> './deps/libicudataswift.so.65'
'/usr/lib/swift/linux/libBlocksRuntime.so' -> './deps/libBlocksRuntime.so'

$ ls -la --block-size=K deps/
total 42480K
drwxr-xr-x 2 root root     4K Dec  3 22:59 ./
drwxr-xr-x 6 root root     4K Dec  3 22:58 ../
-rw-r--r-- 1 root root    17K Dec  3 22:59 libBlocksRuntime.so
-rw-r--r-- 1 root root   432K Dec  3 22:59 libdispatch.so
-rwxr-xr-x 1 root root 27330K Dec  3 22:59 libicudataswift.so.65*
-rwxr-xr-x 1 root root  4030K Dec  3 22:59 libicui18nswift.so.65*
-rwxr-xr-x 1 root root  2403K Dec  3 22:59 libicuucswift.so.65*
-rwxr-xr-x 1 root root   520K Dec  3 22:59 libswift_Concurrency.so*
-rwxr-xr-x 1 root root  7622K Dec  3 22:59 libswiftCore.so*
-rwxr-xr-x 1 root root   106K Dec  3 22:59 libswiftGlibc.so*

Impact on existing packages

The new behavior will take effect with a new version of SwiftPM, and packages build with that version will be linked accordingly.

  • Deployment of applications using "bag of shared objects" technique (#1 above) will continue to work as before (though would be potentially redundant).
  • Deployment of applications using explicit static linking (#2 above) will continue to work and emit a warning that its redundant.
  • Deployment of applications using docker "runtime" images (#3 above) will continue to work as before (though would be redundant).

Alternatives considered and future directions

The most obvious question this proposal brings is why not fully statically link the program instead of statically linking only the runtime libraries. Go is a good example for creating fully statically linked programs, contributing to its success in the server ecosystem at large. Swift already offers a flag for this linking mode: -Xswiftc -static-executable, but in reality Swift's ability to create fully statically linked programs is constrained. This is mostly because today, Swift on Linux only supports GNU's libc (Glibc) and GNU's C++ standard library which do not support producing fully static binaries. A future direction could be to look into supporting the musl libc and LLVM's libc++ which should be able to produce fully static binaries.

Further, Swift has good support to interoperate with C libraries installed on the system. Whilst that is a nice feature, it does make it difficult to create fully statically linked programs because it would be necessary to make sure each and every of these dependencies is available in a fully statically linked form with all the common dependencies being compatible. For example, it is not possible to link a binary that uses the musl libc with libraries that expect to be statically linked with Glibc. As Swift's ability to create fully statically linked programs improves, we should consider changing the default from -Xswiftc -static-stdlib to -Xswiftc -static-executable.

A more immediate future direction which would improve programs that need to use of FoundationXML and FoundationNetworking is to replace the system dependencies of these modules with native implementation. This is outside the scope of this proposal which focuses on SwiftPM's behavior.

Another alternative is to do nothing. In practice, this proposal does not add new features, it only changes default behavior which is already achievable with the right knowledge of build flags. That said, we believe that changing the default will make using Swift on non-Darwin platforms easier, saving time and costs to Swift users on such platforms.

The spelling of the new flag --disable-static-swift-runtime is open to alternative ideas, e.g. --disable-static-swift-runtime-libraries.

Original text:

23 Likes
2

+1

I think overall this is a good direction outlined - anyone who requires dynamic can opt out. Will decrease the initial deployment hurdle for many common use cases.

1 Like
3

How does this affect a package like SwiftPM itself, which has a half dozen executables? Won’t that mean a significant increase in size for the default build? Would it be better for “automatic” products to statically link whatever is unique to that product, but dynamically link whatever is shared with other products? Would it work to have a single executable package statically link the runtime, but a dual executable package install the runtime dynamic libraries into the package’s “bag of shared objects” and then dynamically link them accordingly?

4

I'm +1 on this idea: I think it will produce good outcomes and is a sensible default for the platform.

For 99% of consumers SwiftPM is a part of the toolchain, and is distributed accordingly. In that use-case, SwiftPM can and should be dynamically linked with the runtime. After all, the toolchain also contains the runtime, and you're unlikely to ever want a copy of SwiftPM on a machine that does not contain the runtime, given that you'll never be able to build anything with it.

I think this kind of changing behaviour is extremely hard to predict, so I'd be opposed to it.

3 Likes
5

I'm cheerleading this pitch / change since the initial floating around so just chiming in here again:

+1, definitely the right thing to do on Linux platforms. It'll make using, learning, and deploying Swift much easier and lower the perceived complexity to people coming over to Swift from other languages.

We're not taking away the dynamic approach, so if someone wants to and is experienced enough to know they want it, they can still get the dynamic behavior.

1 Like
6

I would suggest we add “until statical linking is available” here so that we can enable by default as soon as we implement it. IMO statical linking would be useful for Windows too.

1 Like
7

+1 from me, my thoughts from the pre-pitch stand but I think this is a good direction for the Swift ecosystem overall

3 Likes
8

+1 from me. To clarify, this has been the default behavior for WASI by default since the first stable SwiftWasm release (5.3.0), and this tweak was in upstream SwiftPM as WASI-specific for some time already. WebAssembly doesn't really support dynamic linking (yet).

5 Likes
9

I don’t mean the SwiftPM instance in the toolchain, I mean any old package that has that many products. We already have the silly situation where a package with two executables and a library grows in size if you explicitly mark the library dynamic because the executables cannot be hooked up to the dynamic product, and so they statically link its target. You end up with the library three times over instead of reduced to one.

I think the default for SwiftPM on Linux should be that it “does what it heuristically deems best” with no guarantees about the choice it will make*, which corresponds to how things already work on Apple platforms (and to many users’ understanding of what Linux was doing all along). Then specific flags can be available to manually force this or that strategy, such as --static-swift-runtime, --bundle-dynamic-swift-runtime, --external-dynamic-swift-runtime, or whatever else turns out to be useful.

I do not object to blanket static linking for now, if that is seen as a general improvement to the heuristics that benefits most cases. What I don’t really like is codifying the default behaviour in a way that prevents future heuristics improvements or forces such improvements to go through the evolution process.

*except of course the guarantee that everything you need will end up in the “bag of products” in some form or other

1 Like
10

I think this is an orthogonal question: whether SwiftPM should respect the request for dynamic linkage for in-package executable targets is unrelated to whether we statically link runtime libraries. If you have explicitly asked for dynamic linkage, then you should get it.

I think that's what the proposal is doing: changing the heuristic to explicitly prefer static linkage.

Any change to the default, implicit linkage behaviour absolutely must go through evolution, IMO. The linkage behaviour is an observable property of the system, and changing it without letting people know is likely to trigger breakage. In particular, moving from any form of static linkage to implicit dynamic linkage is going to break build systems all over the place that assumed they did not need to copy something that they now do. I don't think we should make any such change without the evolution process to justify it.

3 Likes
11

How is it unrelated? If I want to install a package that has six executables, and the heuristic chooses to statically link the runtime into each one, then I end up with the runtime six times over, which is wasteful. (It might be less of a nuisance than the status quo, which is why I am okay with it if we leave the door open to further improvements to the heuristics, which simply requires not promising the default to be static.)

I do not think there really is any official support for any kind of installation at the moment. swift run is supported in a stable manner, but for anything else the only API you have is to query --show-bin-path. You still have to manually sort through which out of the hundreds of bits and pieces in that directory are actually relevant. The contents of that directory have been constantly changing as Swift has evolved and none of it has gone through the evolution process.

The lack of a stable installation model has been a major gap in SwiftPM since the beginning. The original future directions document envisioned an install command, but it has never materialized.

A smaller step that might fall within the scope of this pitch would simply be an output directory of “all the installation bits”. It could either be at a fixed location, or specified with something like --installation-components /somewhere/specific. Then the contract of the default heuristic would simply be that everything needed would end up in there by the time the build completed (without any other junk). Then the installer (whether human or script) knows everything in there needs to be handled. For now that is executables, dynamic libraries (including the runtime if not statically linked) and resource bundles. If anything is left over, then the install is known to be incomplete. Overrides such as for linkage modes would still use that folder in the same arrangement, but would add corresponding guarantees about which pieces would or wouldn’t be present.

Another question: What happens if a dependency product is a dynamic library? Does the default still statically link the runtime into both the dynamic library and the executable? Does that even work at runtime?

Unless I am wrong and that situation magically works, the default must be able to fall back to other strategies. A --static-swift-runtime flag could reasonably object that the current dependency graph does not support it, but I don’t think we want a default swift build to object that “error: dependency such‐and‐such does not support the default build mode; use --disable-static-swift-runtime”.

12

I agree that this is orthogonal, but just to clarify, it is currently not possible to ask for this, because only package products support specifying a linkage type and you cannot link products from the same package, only targets. So it isn't possible by definition to dynamically link any built products from the same package.

4 Likes
13

Running the same commands with the last trunk snapshot toolchain from Dec. 6, the statically-linked executable is surprisingly much smaller, around 7.5 MB (I don't know what optimization is done differently in trunk, or if that small size would stick for non-trivial executables). Further, a subsequent trunk pull removed the libicu dependency for the stdlib, though not for swift-corelibs-foundation, so the static linking penalty keeps diminishing.

How does this affect a package like SwiftPM itself, which has a half dozen executables?

@SDGGiesbrecht, while this does not affect your question in any way, I just wanted to mention that your chosen example of SPM no longer ships a half-dozen executables but a single combo executable, swift-package, and several symbolic links to it. Run this command with recent trunk snapshots to see what I mean, ls -l swift-DEVELOPMENT-SNAPSHOT-2021-12-06-a-centos8/usr/bin/swift-*.

4 Likes
14

Yes, I am aware that fancy post‐processing goes on when constructing the official toolchain. But none of that happens when you just do swift build, like we are discussing here. Everyone’s goal here is to make the defaults smarter so that fewer packages have to monkey around with those sorts of extra steps.

(P.S. Generalized manifest support for “alias products” like that would be absolutely awesome for other reasons.)

2 Likes
15

Yep, ICU has been dropped from the standard library, and I've noticed @kubamracek recently added a hermetic-seal-at-link flag which enables more aggressive LTO-based dead-stripping. Dead-stripping is currently a big weakness in Swift, because it's very high-level so there's a lot of metadata (e.g. witness tables for protocol conformances) floating around. It's difficult for the compiler to know whether those things are really used or not, so they can't be stripped.

The tests indicate that this feature isn't quite ready for non-Apple platforms yet, but once it is, hopefully we'll be able to get those binary sizes right down. That's nice for package dependencies, which are already statically linked, but the standard library is designed to make heavy use of generics so I expect we'll also see big benefits there.

LTO isn't a thing for dynamic libraries (AFAIK), because... you know, they're linked at runtime by the dynamic linker. I think we'd really want to take advantage of this everywhere that we can, so defaulting to static linking the runtime libs is a good move. +1.

5 Likes
16

+1 from me as well, this makes total sense for Linux environments. Additionally, the deployment story for server side swift applications becomes way better without us having to version the environment.

17

It sounds like the goal here is for programmers of container-deployed executables (chiefly server programmers) to be able to get a simple swift build to build their binaries with --static-swift-stdlib without having to explicitly specify that every time. I assume swift build can pick up preferred build-configuration options from the package it's building. Should there just be a swift package init --type X that defaults to building things in the best way for a container-deployed executable, which would include --static-swift-stdlib?

18

There is no place today to capture such preference. Are you implying that instead of changing the default behavior we should add a new "build settings" section of sorts to Package.swift to capture preferences of this kind and it will be the package author responsibility to set it?

19

Hmm. I assume it must've been a philosophical decision to not have that. I find that pretty surprising; it certainly seems to assume that there's one standard way that executables should be built on a platform, which puts us in the unfortunate position of having to pick that globally. Dynamic linking is a more reasonable default for a lot of use cases even on Linux; it's mostly containerization that prefers statically linking every non-system dependency. I understand that server development, and thus containers, is the central driver of our current adoption on Linux; server developers shouldn't be required to pass a specific command-line flag to swift build as the only way to get acceptable output. But this proposal seems to be just flipping a switch such that, in a year or two, we'll be forcing a different set of users to pass a different command-line flag to swift build as the only way to get acceptable output.

It sure seems to me like the right fix is to recognize that there's something we're failing to express in packages that the build system ought to be aware of, one way or another. If we philosophically don't want a build-settings section, maybe the best approach is to distinguish between kinds of executables, so that executables destined for containers are a different basic kind of product. The build system could then default to different build settings for that kind of product, like statically linking every non-system dependency.

19 Likes
20

I do not know why such was not added originally. cc @ddunbar

This focus of the proposal is about what would be considered a sensible default. Even if we decide to add some sort of build settings section to the manifest to capture the user's preference (which seems like a good but additive suggestion), SwiftPM would still need to have a default behavior when such are not specified. Right now, that default behavior across all platforms is dynamic linking [of the runtime libraries], but one could argue the default should be different across platforms, reflecting how the Swift runtime is distributed on those platforms.

Given that no Linux distribution ships the Swift runtime libraries, nor is there a stable ABI for Swift on Linux, the proposal argues that the sensible default on Linux should be static linking [of the runtime libraries]. If and when a Linux distribution ships with a runtime, the toolchain on that distribution could change this default to reflect that, or we could decide to further refine the behavior.

9 Likes