Upcoming changes to Windows Swift SDKs

Platform Windows
, ,
21

Correct (at least if they depend on it directly/indirectly through Concurrency).

libclosure and libdispatch are not Swift libraries, they are C libraries. They can and are used outside of Swift. If you are hosted within an application which uses these, they should not be duplicated in the address space. As such, dynamically linked ensures that a C program can host a statically linked Swift binary.

No, the static runtime will still dynamically link against msvcrt, msvcprt, vcruntime, ucrt, kernel32, and ntdll at the very least.

A fully static binary is not entirely possible on Windows - kernel32 and ntdll are nearly impossible to get away from. It is possible to statically link against libdispatch and libclosure, but that would require an additional SDK build.

As it is, currently, it is not possible to link against any of the debug variant of the standard library. That currently is something that is a TODO and requires 2 additional SDK builds at least (swiftCored.dll and libswiftCored.lib as the generated artifacts and the associated libraries for them). However, in order to control the explosion of possibilities, I expect that they will also require the debug variant of the C/C++ runtimes as there is no ABI compatibility between the debug and release variants of the MS runtimes.

Currently, we have 3 architectures x {static, dynamic} x {debug, release} or 12 SDK builds (of which we are currently building 6 as the debug builds are not possible today).

Adding in the additional configuration that you are suggesting - static libclosure and libdispatch variant would add in an additional 6 builds. But it would still not be fully static (as you cannot statically link against the system libraries), and so I am not sure how much value that brings.

Ultimately, Windows as a platform really prefers dynamic linking, much like macOS does. It is just that macOS does not incur the same penalties of distribution given that these are considered system libraries.

2 Likes
22

Is the complaint actually about dynamic linkage, or is it about distributing the DLLs? Nobody has to distribute kernel32.dll or ntdll.dll. But now every Swift app has to distribute the Swift DLLs and mitigate the chances of DLL hell.

1 Like
23

A combination of the two I feel. I think that people want some of the DCE opportunities that static linking enables for a single module (DLL/EXE) package offers. The dependency on 2 (relatively small) DLLs seems to not be too bad IMO and enables a broader set of use case models.

On Windows, applications generally are distributed as installers, where we can create a tiny MSM for the 2 libraries only for the static SDK, versus the full MSM that we generate for the dynamic linking. Integrating that into the installer makes it easy.

The alternate distribution model is a zip file (portable packages), where files are trimmed. Adding the 2 files to the zip shouldn’t add any bulk or real complications.

The bigger change that I think that is being ignored here is the fact that the new experimental SDK build enables a whole new major feature: Win32 SxS installs. This actually is likely to complicate things a bit for the toolchain in order to ease developer/user lives. The Win32 SxS functionality means that the runtime is possible to install with multiple versions simultaneously and all applications are able to share that installation, without concern about ABI stability because the application will run against the specific version that they are built against. Effectively, we are getting ABI stability like functionality without actually guaranteeing an ABI.

24

I chuckled at the thought that “DLL” may read as “DL hell”.

@compnerd yeah, I agree that the inconvenience is only about distribution rather than the linking details. Having been out of the loop since Windows XP, I wonder if WinGet could pull the Swift runtime as a dependency package, like apt or pacman would do on Linux with a binary library. For example, how do .NET apps streamline this with the .NET framework?

25

I think that this is the point that I am failing to convey. With the experimental SDK, the runtime is SxS capable, so once bugs are ironed out, this is reduced to packaging issues:

1. how do we build a MSI (hint: its already something that we have to do to build the toolchain)
2. how do we host the MSI (@mishal_shah would be needed to help here - we need to host an additional separate MSI in addition to the installer exe)
3. how do we ensure that winget is updated regularly (hint: this problem exists with the toolchain too)

Basically, this work is enabling the idea that you can simply winget install Swift.Runtime.6.3.0.12345as we can finally have all the parallel versions and not have only Path as the control mechanism.

2 Likes
26

@compnerd FYI I installed the latest snapshot (x64) to attempt at the static build, but I couldn't even get past swift build with it. You can see I'm trying with the most basic package. 6.1.2 works as expected on the same machine:

PS C:\Users\keeshux\Documents\repos> cd .\foopkg\
PS C:\Users\keeshux\Documents\repos\foopkg> swift package init --type executable
Creating executable package: foopkg
Creating Package.swift
Creating .gitignore
Creating Sources
Creating Sources\foopkg\foopkg.swift
PS C:\Users\keeshux\Documents\repos\foopkg> swift run
error: 'foopkg': Invalid manifest (compiled with: ["C:\\Swift\\x64\\Toolchains\\0.0.0+Asserts\\usr\\bin\\swiftc.exe", "-vfsoverlay", "C:\\Users\\keeshux\\AppData\\Local\\Temp\\TemporaryDirectory.N7JTw1\\vfs.yaml", "-L", "C:\\Swift\\x64\\Toolchains\\0.0.0+Asserts\\usr\\lib\\swift\\pm\\ManifestAPI", "-lPackageDescription", "-sdk", "C:\\Swift\\x64\\Platforms\\0.0.0\\Windows.platform\\Developer\\SDKs\\Windows.sdk", "-libc", "MD", "-I", "C:\\Swift\\x64\\Platforms\\0.0.0\\Windows.platform\\Developer\\Library\\XCTest-0.0.0\\usr\\lib\\swift\\windows", "-I", "C:\\Swift\\x64\\Platforms\\0.0.0\\Windows.platform\\Developer\\Library\\XCTest-0.0.0\\usr\\lib\\swift\\windows\\x86_64", "-L", "C:\\Swift\\x64\\Platforms\\0.0.0\\Windows.platform\\Developer\\Library\\XCTest-0.0.0\\usr\\lib\\swift\\windows\\x86_64", "-I", "C:\\Swift\\x64\\Platforms\\0.0.0\\Windows.platform\\Developer\\Library\\Testing-0.0.0\\usr\\lib\\swift\\windows", "-L", "C:\\Swift\\x64\\Platforms\\0.0.0\\Windows.platform\\Developer\\Library\\Testing-0.0.0\\usr\\lib\\swift\\windows\\x86_64", "-use-ld=lld", "-swift-version", "6", "-I", "C:\\Swift\\x64\\Toolchains\\0.0.0+Asserts\\usr\\lib\\swift\\pm\\ManifestAPI", "-package-description-version", "6.2.0", "C:\\Users\\keeshux\\Documents\\repos\\foopkg\\Package.swift", "-o", "C:\\Users\\keeshux\\AppData\\Local\\Temp\\TemporaryDirectory.At8Oxx\\foopkg-manifest.exe"])
error: compile command failed due to exception 3 (use -v to see invocation)
error: 'foopkg': Invalid manifest (compiled with: ["C:\\Swift\\x64\\Toolchains\\0.0.0+Asserts\\usr\\bin\\swiftc.exe", "-vfsoverlay", "C:\\Users\\keeshux\\AppData\\Local\\Temp\\TemporaryDirectory.mA9SlR\\vfs.yaml", "-L", "C:\\Swift\\x64\\Toolchains\\0.0.0+Asserts\\usr\\lib\\swift\\pm\\ManifestAPI", "-lPackageDescription", "-sdk", "C:\\Swift\\x64\\Platforms\\0.0.0\\Windows.platform\\Developer\\SDKs\\Windows.sdk", "-libc", "MD", "-I", "C:\\Swift\\x64\\Platforms\\0.0.0\\Windows.platform\\Developer\\Library\\XCTest-0.0.0\\usr\\lib\\swift\\windows", "-I", "C:\\Swift\\x64\\Platforms\\0.0.0\\Windows.platform\\Developer\\Library\\XCTest-0.0.0\\usr\\lib\\swift\\windows\\x86_64", "-L", "C:\\Swift\\x64\\Platforms\\0.0.0\\Windows.platform\\Developer\\Library\\XCTest-0.0.0\\usr\\lib\\swift\\windows\\x86_64", "-I", "C:\\Swift\\x64\\Platforms\\0.0.0\\Windows.platform\\Developer\\Library\\Testing-0.0.0\\usr\\lib\\swift\\windows", "-L", "C:\\Swift\\x64\\Platforms\\0.0.0\\Windows.platform\\Developer\\Library\\Testing-0.0.0\\usr\\lib\\swift\\windows\\x86_64", "-use-ld=lld", "-swift-version", "6", "-I", "C:\\Swift\\x64\\Toolchains\\0.0.0+Asserts\\usr\\lib\\swift\\pm\\ManifestAPI", "-package-description-version", "6.2.0", "C:\\Users\\keeshux\\Documents\\repos\\foopkg\\Package.swift", "-o", "C:\\Users\\keeshux\\AppData\\Local\\Temp\\TemporaryDirectory.DvTFEs\\foopkg-manifest.exe"])
error: compile command failed due to exception 3 (use -v to see invocation)
PS C:\Users\keeshux\Documents\repos\foopkg>
27

Interesting, I’ll try to see if I can replicate that. The output with -v might lend some clues, the exception 3 doesn’t immediately indicate what is happening.

28

I wouldn’t bother, the latest one (today) works correctly. I wouldn’t even exclude I messed up for having multiple hack-ish installs.

29

Well, hopefully that is a limited time thing! I am hoping that this work finally enables us to have multiple parallel installations (well, at least any number of stable versions and 1 development version) going forward.

30

An update on the status of this work - the last snapshot (8/14) should have most of the experimental SDK, the missing pieces are the C++ interop libraries and the SwiftRemoteMirror. The former has been merged but we do not have a snapshot yet. For the SwiftRemoteMirror, I am working towards the packaging for that as well. Note that the Foundation restriction should also be lifted at this point - you should be able to use Foundation for static linking with the new static SDK.

There are still a few pieces that are being shuffled around, but we are pretty close to completion. I am curious about the overall performance that people are experiencing. There is one optimization that we are missing in the new builds due to the need for the early swift-driver to enable properly. Again, that would be a transient state, as the new SDK marches towards becoming the default, we will have a path to re-enable the optimization and the early swift-driver.

The packaging for the redistributable for the static runtime is something which needs to be completed (that would contain only dispatch.dll and BlocksRuntime.dll).

3 Likes
31

Another week, another update.

The CI is currently broken, though a fix for that is in flight ( build.ps1: add a workaround for Android build break by compnerd · Pull Request #83852 · swiftlang/swift · GitHub ). Once that is merged, I am hoping that we get another snapshot shortly thereafter. That snapshot should contain the complete static SDK! :partying_face:

More progress has been made though. The complete SDK is now being built (i.e. we are building both the static and the dynamic experimental SDK) in CI. This at least should prevent us from regressing on the work here. The static SDK build has been adjusted to be in the same shape as what I expect it to look like. That is, I expect that your additional flags should now be:

-static-stdlib

I am working on the packaging for the dynamic SDK still. This includes the packaging for the MSM for the runtime redistributables as well.

Once the dynamic SDK and the runtime redistributables are packaged, I intend to start working towards removing the legacy SDK. I would like to quickly move towards removing the legacy SDK as it adds unnecessary time to the build and maintenance burden.

7 Likes
32

The battle towards the experimental SDK continues.

It turns out that the progress that I thought had been made isn’t really entirely correct. There are some changes to dispatch which are still outstanding. So, this actually makes the static SDK slightly more tricky to use for the time being (and more fragile). However, the latest snapshots from swift.org should have a complete static SDK available for use.

The current SPM invocation expected to work is:

swift build -Xswiftc -sdk -Xswiftc "${env:SDKROOT}\..\WindowsExperimental.sdk" -Xswiftc -static-stdlib -Xcc -Ddispatch_STATIC -Xcc -static-libclosure

The -Xswiftc -sdk -Xswiftc “${env:SDKROOT}\..\WindowsExperimental.sdk is required due to the dynamic SDK not being available. That is currently predicated on WiX: package up the experimental dynamic SDK and the runtime redistributables by compnerd · Pull Request #456 · swiftlang/swift-installer-scripts · GitHub which packages up the dynamic SDK.

The packaging PR referenced above includes the MSM and the MSIs for the runtime and dynamic SDK. The single dispatch/closure issue remains on the static SDK, but otherwise, the shape of this is correct.

I would really appreciate it if people could test and validate this for their various use cases.

6 Likes
33

The ongoing saga is now …

We have the dynamic and static SDKs packaged and the redistributable packaged as well. I have been experimenting with the packaging and found a few small issues (BlocksRuntime was missing in the dynamic SDK, SwiftRemoteMirror was not properly built). These are being worked through, so I am hoping to be able to get a working SDK soon.

I managed to actually use the static SDK to build a product on Windows in GHA: vigil/.github/workflows/release.yml at main · compnerd/vigil · GitHub including cross-compilation for Windows ARM64.

This is probably the best time to speak up if you are finding issues with this new SDK. I am going to be moving towards removing the “legacy” SDK from the build and packaging shortly.

2 Likes
34

Can you please elaborate on this a bit? Is it a separate installer that we should distribute along with our Swift-built executable files?

1 Like
35

The redistributable is a MSM that you can integrate into your MSI for distribution. The allows you to avoid having to manually collect the libraries and package them up for applications. Microsoft does the same thing for their runtimes.

3 Likes
36

I thought you were saying earlier in the thread that you'd still expect clients to still require a dynamically linked libdispatch and libclosure, but this seems to imply there would be static variants now?

37

Yes, that is the intent, however, the problem is that there are still some build issues to work through to make everything work properly. I think that switching over to the new SDK is more pressing though as it offers a way to noticeably reduce the CI load for Swift.

38

If that configuration is being supported after all, then doesn't that mean we're doubling the number of build configurations as you suggested earlier?

39

The builds aren’t being doubled per se. We need this number of build anyway, swiftDispatch will be available statically or dynamically. Until a split of libclosure, libdispatch, and the dispatch swift overlay is done/possible, all this is at the packaging side and we must pay the build time penalty.

1 Like
40

I tested the SDK by downloading the latest build from the browser company. It worked both statically and dynamically, until this code:

import Foundation
import FoundationNetworking
let url = URL(string: "https://example.com")!
let (data, response) = try await URLSession.shared.data(from: url)
print(data.count)

Static compilation:

swiftc -static-stdlib http.swift                                                                                  
error: link command failed with exit code 1104 (use -v to see invocation)                                               
LINK : fatal error LNK1104: cannot open file 'curl.lib'                                                                 
clang: error: linker command failed with exit code 1104 (use -v to see invocation)

Dynamic linking compiled, but when I ran it, I got this:
The procedure entry point $sScM6sharedScMvau could not be located in the dynamic link library C:\py\http.exe.
It worked with the non-experimental Swift SDK though.