SE-0342: Statically link Swift runtime libraries by default on supported platforms

Evolution Proposal Reviews
1

The review of SE-0342: Statically link Swift runtime libraries by default on supported platforms begins now and runs through February 25, 2022.

The proposal is authored by @NeoNacho and @tomerd.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager by DM. When messaging me directly, please keep the proposal link at the top of the message and put "SE-0342" somewhere in the subject line.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  • What is your evaluation of the proposal?
  • Is the problem being addressed significant enough to warrant a change to Swift?
  • Does this proposal fit well with the feel and direction of Swift?
  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at https://github.com/apple/swift-evolution/blob/master/process.md.

Thanks for helping review this proposal!

Ted Kremenek
Review Manager

16 Likes
2

nit: SE-0342 (instead of SE-342)

2 Likes
3

only the Swift runtime libraries (stdlib, Foundation, Dispatch, etc)

Is Foundation to be viewed as such a runtime library - as far as I could tell the sample later in the proposal does not link against Foundation (and I personally would not want to be forced to link it in- it is not uncommon for portable server swift code to avoid depending on foundation as been discussed in other threads).

Otherwise I am +1 on the change of default linking behavior - as long as the opt out exists I think it makes sense from a pragmatic point of view.

I’ve followed the pitch thread and fairly quickly read through the final proposal.

3 Likes
4

AFAIK libFoundation will be statically linked as long as Foundation is depended. That means server-side developers should avoid Foundation by their own. I would like to see some flag to warn Foundation usage, but that’s off-topic for this review thread.

5

yes, foundation and dispatch will be linked if used

4 Likes
6

Big +1 on this for the reasons stated around static linking generally being the preferred approach on Linux.

Definitely, this is a small change that should improve the onboarding for non Apple platform developers wanting to use Swift since it's more similar to other programming languages.

I read all the related threads.

Similar to this comment [Pre-Pitch] Statically linking the Swift runtime libraries by default on Linux - #26 by NeoNacho I don't love the -runtimes portion of the suggested flag. I think stdlib fits a bit better in here, especially since I think runtime is an overloaded word with things like the clang runtime libraries. But I understand that a lot of these words are overloaded.

I think this is likely out of scope but I think it would be great if part of the solution here is putting in some effort to solve the potential sharp edges with this feature like SwiftSyntax does not work with --static-swift-stdlib so users didn't hit issues like this in libraries we control.

3 Likes
7

Definitely for it. This change primarily benefits Swift on Linux. The arguments about it not notably helping in a general linux distribution preferring shared libraries to preserve space are relevant, but it accelerates anyone trying to use server-side swift within a container, which is the more beneficial to a broader audience in my view.

Yes, especially given that common deployment patterns are leaning more and more into statically linked binaries for containers. This reduces (doesn't fully resolve) the number libraries that a developer needs to replicate in order to run their binary.

Yes

I've found them hugely beneficial, especially for fast deployment of (comparatively) slim container images.

Followed the conversations, and have repeatedly used the previously-not-default option to generate static binaries for multiple projects.

8

Overall; -1

The premise of this change feels off to me.

Just saying “Linux is a server-centric OS” is a bit of a misnomer. There are certainly distributions of Linux, like Alpine, that are server-centric, but there are also distributions that are desktop-oriented, like Ubuntu Desktop and Fedora. On Alpine, it makes sense to statically link everything, especially given that it’s a statically linked musl-based distribution. Ubuntu, Arch, and Fedora usually dynamically link runtimes. It really only at the distribution level that it makes sense to determine the default behavior.

In terms of “shipping a runtime”, the normal means of distributing files on most (at least all of the distros I've worked with) is via a package in the package manager. Creating a package like swift-runtime so that one could run sudo apt install swift-runtime on Ubuntu, for example, would satisfy the missing runtime and standard library issues in a way that is consistent with how other bits are installed. Then one would make that swift-runtime package a dependency of any swift-based programs.

In terms of propagating bug fixes; distributions regularly update the dynamic libraries, especially when a security vulnerability is found. For servers running a single program used to serve webpages, this shouldn’t be a problem if the runtimes are statically linked. The person hosting the server can re-compile with the fixed library and re-distribute it and bob’s your uncle. In a desktop context, the person using the machine will likely have more than one program. Are we expecting them to update every executable?

Additionally, in a server context with a minimal set of programs, the binary size overhead of packing around multiple copies of the same runtimes should be negligible, but once you have 100 apps installed, all of which are statically linked, that’s going to get expensive pretty quickly. As noted in the proposal,

Statically linking of the Swift runtime libraries increases the binary size from 17K to ~35M.
...
This jump in binary size may be alarming at first sight, but since the program is not usable without the Swift runtime libraries, the actual size of the deployable unit is similar.

The difference is that you only pay for the runtimes once when they're dynamically linked, whereas you will pay for each executable when it's statically linked. Once you have two copies of your program, you're paying for that runtime twice, so it will be ~70M instead of ~35M + 34K. (It looks like the cost of the program itself is fairly negligible compared to the size of the runtime + stdlib, so including the extra dead-code-elmination you get from statically linking, I'm estimating the runtime to be about 35M.) Three would be 105M instead of 35M + 51K. And once you get up to 100 apps, you're looking at 3 gigabytes in copies of the runtime.

I think my last concern is ABI. If an App links Foundation statically and another dependency in the OS contains a copy of Foundation, can objects be passed between them? If everyone dynamically links the same copy of Foundation, the answer is yes (or at least should be). With statically linked libraries, it’s unclear. I can’t tell if Jordan was inspired by this proposal when he wrote this blog post, but Dynamic Linking Is Bad For Apps And Static Linking Is Also Bad For Apps // -dealloc covers a lot of interesting points, and comments on why you really can’t mix dynamic and static linking safely.
Then my last ABI concern is surrounding copies of global variables coming from different statically linked copies of libraries. It’s the same problem that LLVM runs into when you try to statically link the library multiple times. The cl::opt global variables conflict.

Linux isn’t just a container or server platform, so the premise of treating as such by default is incorrect. It really depends on what distribution you’re targeting. So to conclude, if you only ever want to have a single Swift program installed on someone’s computer, this is absolutely the right way to go. If we want more than one Swift program installed, then dynamically linking is still the correct behavior, and should continue to be the default on distributions that are used for desktop applications.

Evaluation: -1
Is the problem big enough: Yes, Swift should be able to statically link programs, but I don't think it should be by default.
Effort into review: I've read the proposal through at least 4 times and thought about my answer last night while sleeping before writing it up.

4 Likes
9

Given the ABI isn't stable on non-Apple platforms today my understanding is that you cannot update the libraries without updating the binaries (or more realistically it might work but isn't guaranteed), so I believe the answer to this is yes.

3 Likes
10

+1

  • Windows may benefit from statically linking for the reasons highlighted above but it is not technically supported at this time. As such the default on Windows will remain dynamically linking the Swift runtime libraries.

This is unfortunate, windows could really use static linking. Could we stablish that the default should be static linking on windows too once the technical blockers are removed so that choice does not have to be hashed out again in Swift Evolution? Could you also attach a work item that tracks windows static linking status?

Edit: some background on this Swift binary with embedded runtime and standard library? - #2 by compnerd

11

i use linux as my normal desktop platform, but i just don’t think there are enough swift apps floating around on a typical linux user’s machine for this to be a problem right now. on a platform like iOS where users might have 30+ swift apps installed on the same device, static linking is a problem. but i just don’t think that’s where swift on linux is at yet.

8 Likes
12

Huge +1 on this. The current situation is almost user hostile because we require the user to distribute a number of .so files that however have to be perfectly matched with the compiler version that the binary was compiled with.

6 Likes
13

In favour. Swift should be a good citizen on every platform where it is deployed, so if a given platform prefers static linking, so should Swift.

14

I think this is the correct and pragmatic view. Unless I’m underestimating the enthusiasm from distro maintainers to keep Swift up to date in their package trees, statically linking probably best serves all current Linux clients of Swift apps.

15

Big +1 from my side. As mentioned in lot's of other comments, as long as Swift does not guarantee ABI stability on Linux, the safe option is statically linking the runtime libs. I strongly agree, that Swift should default to the safe option.

Besides that: In my opinion this change will make Swift server side development easier to pick up. As a server app compiled with a Swift toolchain for a given linux distribution, will likely work on the same distribution and not require a toolchain to be installed first. Those benefits can be seen today, when developing for AWS Lambda and using the static runtime flags.

Further this leads us to be able to potentially drop the slim docker images – that were never that slim :wink:. Statically linking the runtime libs is a way better option here.

Lastly as LTO support is something that is looked into, the overall binary size will decrease once LTO support has landed, as only the parts of the runtime libs remain in the executable that are actually needed.

1 Like
16

Strong +1, been reading all the related threads and proposals since this first came up.

All the points "for" I have are the same as Fabian and Johannes already articulated :slight_smile:

17

This is really a good thing...configuring static linking is a pain...
In addition, there should be some work on the linker because current result is 50 MB+ for simple Hello World....

18

A program without Foundation will be a little slimmer in Swift 5.6 thanks to @Alejandro’s work on native String implementations. I believe the size problem will finally be solved by LTO which also should be the new default once it’s stable.

1 Like
19

I think that Evan sumarised all my personal concerns with static linking pretty effectively.

For Windows we would want to keep defaulting to dynamic linking for the most part (the loader model is different, and although there is a small cost associated with it, the dirtied pages are fewer due to the representation) IMO. While there are some cases where static linking does make sense, I'm not so convinced about the default behaviour.

I realize that the proposal is speaking to the executable case only, but due to the inability to always safely mix static and dynamic linking (which Evan points to Jordan's blog post for, and I won't bother reiterating the technical details of), it concerns me that this enables a position which makes it easier for individual developers to accidentally do the wrong thing with mixed mode linking for the executable.

A way to mitigate that would be to enforce no dynamic linking permitted on the executable (requiring everything to be fully statically linked) which is currently not possible to glibc's requirement of dynamic linking for at least nss.

2 Likes
20

Too late, but -1 as well, for the reasons outlined by @etcwilde and @compnerd. The effort is driven by a very specific use case and doesn't fit the goal of using Swift as a general dev language on Linux.

To make it worse, doing this might also proactively lower the priority on the (presumably non-existent) effort to provide a stable Linux ABI, which in turn kinda prohibits inclusion of Swift in Linux distributions.

1 Like