That would also most definitely be nice, but is as you point somewhat orthogonal to this pitch which is an easy “bridging” win even if pursuing such a new setting, to decrease friction in the short term (with no major long term tech debt really). Given the current state server side on Linux, there are a number of already fairly strong reasons to flip the default for Linux already mentioned (for the runtime libraries).
1 Like
My position is that static linking is a sensible default only for executables destined for containers. The standard platform behavior for non-containerized executables on Linux and the BSDs, as defined by platform maintainers through their standard package systems, is to dynamically link your dependencies. You don't need dependencies to have a stable ABI or be distributed with the OS for that to be useful.
I understand that containerized server development is driving most of our usage on Linux right now, and I definitely sympathize with wanting to give those developers the immediate win of swift build just building correctly out of the box without needing any extra command-line flags. However, we've also talked about a number of other build-time configurations that probably ought to be different for containerized server executables, such as linking in a non-standard crash handler / backtracer, or even someday building with crash unwinding enabled in Swift. I'm sure an actual server developer could identify any number of other things along these lines.
If we want those things to also just work out of the box, we're going to have two alternatives:
-
We can continue to recognize only one kind of executable. In this case, we will need to change the defaults to be ever more specific to whatever we've decided to bless as the default use-case for the platform. People who want to build differently will have to override those defaults. As we add features to the system, they may need to override more and more.
-
We can recognize different kinds of executables. As we add features to the system, we can make intelligent decisions about the right defaults for different kinds of executables. Any particular version of swiftpm will make the best decision for what it supports.
I don't think the latter entails having to make a million design decisions immediately. We just pick a name for it, and initially it means exactly the same as executable except using static linking of non-system dependencies. As we grow the system, it can evolve.
I guess executable would mean something like "platform CLI executable". IIUC, Windows would benefit if we also distinguished GUI programs from CLI, because we ought to be emitting winmain instead of main. That would be another example of a standard kind of executable; I don't think it's mandatory for the initial proposal.
14 Likes
Recognizing different kinds of executable products is a great direction and a design space the SwiftPM team has been exploring in depth as part of the plugins effort. Given the breadth of type of products that could fall into this category (CLI, web-service, daemon, windows GUI application, Linux GUI application, etc) we believe the most fitting design is plugin based API that would allow product specific type safe configuration. Said differently, having all these product types hard coded into SwiftPM would not be scalable, and the SwiftPM plugin mechanism was designed to provide infrastructure for such feature.
Orthogonal to this, there is the question of what is the sensible default for platforms that do not ship with the Swift runtime libraries, and the Linux platform is one key example. Right now the default is dynamic linking and the proposal is focused on changing this default to static. IIUC you seem to suggest that dynamic is the sensible default:
Taking a concrete non-server/non-containerized example, how would a user distribute a Swift based CLI tool (an executable) on Linux today? The Swift project does not provide a Linux native packages that contains the runtime libraries so is there is no way to express a dependency on the Swift runtime libraries "through their standard package systems". As such the only choices I am aware of are to either:
- use static linking
- package these libraries along side the executable (tarball / zip file etc)
Are you aware of other options? If not, I would claim that option 1 is obviously the more sensible of the options, and as such the proposal is to make it the default behavior on Linux. If and when Linux distributions start shipping Swift runtime libraries and/or there is a way to express dependency on version-specific Swift runtime libraries through Linux native package systems we can revisit this default.
6 Likes
I feel like this is generalizing the feature to the point that it has to be solved in a way that no longer addresses the original problem of wanting a correct build "out of the box". It's common for IDEs and build systems to make coarse-grained distinctions like console vs. GUI applications. Supporting 3-4 such configurations covering all the major kinds of executables (console, GUI, containerized, maybe embedded) is not unscalable.
Given that this is something we want to do, I don't see why we should be making irrevocable decisions based on not having done it yet. And yes, the typical platform answer for this would be to make a package for your program that depends on a Swift package.
Let's look at it from another angle. If we agree that we want to have fine-grained distinctions between kinds of executable, whether by builtin logic or by plugins, then the question is what executable will mean in that world. My proposal is that we should pick something arbitrary but consistent for it, probably the thing with minimal assumptions, like a platform console application. Your proposal is essentially that we should pick whatever we decide is the favored use case for the target, and that containerized executables should be favored outside of Apple platforms. To me, that seems like it will lead to a world in which it is essentially wrong to ever use executable instead of a fine-grained alternative, at least if you care about portability.
2 Likes
More precisely, I believe this is the kind of decision that distro maintainers make. Dynamic linking is important for getting security bug-fixes out to desktop clients. A world in which every package had to be recompiled whenever a new C runtime was released by the gcc team would be a world in which most packages stay broken because they are not actively maintained.
4 Likes
Note that linux distributions provide a way to install just the runtime for a language (e.g. libstdc++) and even have the ability to have multiple parallel versions (with SO versioning) (e.g. exherbo's slots for libstdc++, libgomp, etc). The packages then depend on the runtime at the particular version that they need.
2 Likes
The point of the proposal is to have a sensible default until all these other ideas materialize, which could take a while. Incidentally, I am involved in the effort to create native Linux packages for Swift so know first hand it will take time to get this to the point where Swift programs can express such runtime dependency. For example, Swift's lack of ABI on Linux as well as it's evolution velocity and versioning scheme (introducing major features in semantically minor releases) make it harder to do and we would need to find a way to install many versions of Swift side-by-side on the same system which conflicts with some of the assumptions the Swift toolchain makes. This is not to say we should not work to solve these problems, but to emphasize that it will take time to get there.
From where I stand, the current default is hurting Swift users on Linux and the adoption of Swift on Linux. In that context, its worth mentioning that go and rust, both pretty popular on Linux, choose static linking based approaches by default. Further, in no way is changing the default irrevocable - we have one default today and this proposal suggests we change it. We can change it again if and when all the pre-requisites for dynamic linking are in place on a critical mass of Linux distributions, or when Linux distributions start shipping Swift they can have a different default on their version of the toolchain.
8 Likes
I think for Linux there are two perspectives that it might be useful to consider:
First, that of a user (who might not be familiar with Swift, or necessarily a developer at all) wanting to use some Swift software that is not packaged for their distro. They'd install a Swift toolchain, download the code, run a build command and copy the binary somewhere in their PATH.
Second, someone looking to package a piece of Swift software for some distro, which involves writing some variation of build script and metadata/dependency declaration.
The packager would surely want to declare a dependency on some Swift runtime package, and dynamically link against it, to reap all the benefits provided by that.
The user, on the other hand, would probably be best served by static linking, as that means they don't really need to know anything about Swift and its runtime. They can copy the binary to different machines, and change or remove the Swift toolchain from the machine they built on without consequence.
I would assume that more people build and use a non-package-managed executable of any particular software than there are people creating a package for the same. Thus I think a default that serves the user rather than the packager (who should also be more familiar with the concept of static/dynamic linking and thus able to make their own decision) is the right choice.
I'd also argue that static linking is a "safer" default more in keeping with progressive disclosure. The statically linked binary might waste disk space and ram, but the dynamically linked when mishandled can become completely unusable.
1 Like
Go literally doesn't support dynamic linking to other Go code at all; it's only capable of dynamically linking to C. Rust's support for dynamic linking is a bit better, but I get the impression that it's pretty marginal.
We absolutely wouldn’t change it back. It would just break the build process for every containerized executable that people make from this point on. This proposal argues that the proposed change is okay because nobody who matters actually wants dynamic linking, but the reverse would clearly not be true. That is why I'm saying that this proposal is essentially codifying that Swift for Linux is meant for containerized executables.
5 Likes
It codifies that Swift for Linux via end-user invocation of spm is primarily meant for containerized executables. As long as there’s an easy way for distro maintainers and DIYers to override the default and link the target and all its dependencies dynamically, that may be the right default.
2 Likes
I don't think this is true. I think it's saying that naively typing swift build on Linux is meant for static binaries.
If distro maintainers have opinions about how to distribute their source, SwiftPM should absolutely support those use-cases. It should without a doubt be possible to say swift build --dynamic-swift-runtime or whatever the flag ends up being. The question is not whether this should be possible, merely whether it should be the default. It seems strange to me to say that our default should be optimised for a use-case we do not have and have no timeline to reach, while it should be pessimised for 100% of the use-cases we do have, and will continue to have.
12 Likes
I think it would be feasible to change the default in the future by tying it to the tools-version, so that existing packages would get the old default until they upgrade.
2 Likes
I think the central disagreement here is actually about how much complexity and customization is necessary to support different patterns of building executables.
I believe we all agree that statically linking the Swift libraries is just one thing among many that server deployments will want to do differently by default. And I believe we all agree that it would be reasonable for server packages to say, hey, this is a server package, you should build it with those different rules. The difference is that:
-
I am suggesting that it would be reasonable for swiftpm to be taught to recognize that innately. The package can say it’s a server, and swiftpm will know what that means, and it will default to the best rules it knows for building a server on the target platform.
-
The counter-argument seems to be that this is not reasonable for swiftpm to do innately. This kind of customization must be done via a complex, yet-to-be-designed plugin mechanism. So in the short term, we will still teach swiftpm all of those best rules for building a server, and we will make them the default for all executables except on Darwin.
Given that containers can easily handle complex configuration options, I don’t really see why the defaults should cater to them. The defaults should cater to manual invocation and best practices for those environments.
That being said, there should almost certainly be official recommendations on how to configure SPM when containerizing.
I use Docker to compile vapor. If you use static, you need more memory. Otherwise it will fail.
Yes, I think this basic use case is a good argument for changing the default behaviour to static linking for a release output. Distributing a compiled Swift program with dynamic linking on Linux or Windows feels like distributing a Java program where you need to also distribute a runtime with it or the user has to install it (in our case with the exact version). (*) Using Swift or a compiled Swift program should be very simple for simple use cases. This is not really a technical argument, but more in the direction of “how to sell Swift to new users”.
Besides this basic argument (which also is the main argument of the original proposal if I understand it correctly), probably all said in this thread in somehow true, we need to be able to build for different use cases, and making this easy for common cases seems to be a good thing.
(*) More would have to be said about this for Windows, as the Unicode Components are quite big which currently would make a Windows executable with static linking (if possible) quite big in my understanding, maybe @compnerd could say something about this issue. For Linux, I am not sure what static linking actually means, e.g. would libxml2 be then included or not if needed, and what about possible license issues.
1 Like
Static linking support on Windows is still broken yet. I would suggest to enable it by default (when it’s done) unless we find a simple way to automatically judge and guide users to download the standalone runtime.
AFAIK the clumsy ICU dependency has been removed from the standard library, so there would only be plain text now. That should be much smaller, and takes up the same space across different OSes.
For this pitch, default “static linking” is only applied to the Swift standard library and Foundation (excluding FoundationXML and FoundationNetworking), so libxml2 and libcurl won’t be statically linked.
This is exactly how C++ programs on Windows work.
Static linking is not yet officially supported on Windows (in theory, it should be possible to do, but currently SPM does not properly support building artifacts such that static linking is possible - components used for static linking cannot be used for shared linking and vice-versa). @abertelrud has some thoughts on how to make those adjustments in SPM, which would then allow us to make further progress towards static linking.
However, you are correct, that static linking with Foundation and the use of unicode functions would likely drag in the ICU data, which would be a penalty in the ~25MiB range.
1 Like
I still disagree on that point. For Windows, DLLs are very common, and you cannot have more than one copy of the standard libraries (swift, C) in the address space. So if you statically link the application, you need to make sure that your code is designed such that it can support partial static and partial dynamic linking for all dependencies. You will need to ensure that there is no swift code anywhere else (e.g. SwiftCOM generates a DLL, which means you cannot use that), etc.
The Swift runtime is already packaged as a redistributable component, just like ucrt, and you can add that MSI to the packaging (installer) to ensure that it is available. I think that for Windows, dynamic linking is and will remain the preferred approach.
3 Likes