Enabling Static Linking on Windows

Development Windows
1

Windows has definitely come a far way from when it started. However, one feature that still is not available on Windows is static linking. Everything currently requires dynamic linking, which is a bit unfortunate. Although, technically, the Microsoft linker can resolve the imported symbol, it comes at a cost (binary size and runtime overheads, and unnecessary warnings). It would be wonderful to get this issue resolved. Even then, it results in over-exposure of the interfaces, which is another source of issues. However, going the other way has even bigger problems, and given that dynamic linkage is generally preferable, it is what I focused on initially.

Let me summarise the behaviour just to be extremely clear about what is needed even though this is probably known to many people already. The primary issue is that calls need to indicate their DLL storage, that is indicate whether they are local to the module or not. This is normally done via __declspec(dllexport) and __declspec(dllimport).

  • __declspec(dllexport) indicates that the function participates in global symbol resolution and should be made available to all modules in the address space
  • __declspec(dllimport) indicates that the function resides in an external module in the address space.

We currently do a pretty good job about identifying module local symbols by identifying the module being built, and anything outside of the module is marked for import. Any public interface is marked as export. This has resolved nearly all of the references to functions across module boundaries.

For static linking, we need to indicate to the driver that the module is meant for static linking. We can easily require that when building a static module that -static is passed to the frontend via the driver (or swift-package-manager). This would prevent the public interfaces from being marked as __declspec(dllexport). This would resolve the over-exported API surface issue.

The problem however arises for the functions that live outside the current module. How could we tell that a module is to be linked statically and calls to it should not be marked as __declspec(dllimport)?

One solution that I can think of, which I really am not sure is the best that we can do, is to serialize to the swiftmodule to indicate whether it is static or dynamic. I'd love to get other opinions and thoughts and see if we can come up with a better solution.

Thanks,
Saleem

CC: @John_McCall @Joe_Groff @Douglas_Gregor

3 Likes
2

I think we should design a slightly more general feature. We want to know two things:

  • Which of our dependencies are we going to be statically linked with? Note that this is useful even if we're generally linking dynamically, because we might be linked statically with some libraries.
  • If we're building a library, is it expected to linked statically or dynamically with its clients?
1 Like
3

Hey @John_McCall,

Sorry, this is what I envisioned - not something specific to Windows, more a generic solution that would work across all the platforms.

Right, this is the easy part to resolve. It is what I address in the third paragraph (counting backwards). We have a means for providing this information with the -emit-library -static. -emit-library is processed as -emit-library -shared.

Correct - this is the part of the problem that I am trying to find a solution to: which dependencies are going to be linked against statically (which on windows requires them to be built specifically for static linking). More specifically, how to know that set when building the "consuming" library. The solution that could work (I don't think its great, but it is a solution) is to serialize that into the Swiftmodule which would tell the consuming library that it should expect static linking for this module. Note that this would allow improvements for ELFish platforms as well (as that can then use protected visibility on those interfaces rather than public) which is why I was thinking that this should be a general facility.

1 Like
4

Is it possible to dllimport things that are dllexported from the same library? (Is that what you were referencing with "binary size and runtime overheads, and unnecessary warnings"?) If so, I think -static can be an "expert" flag, because everything will work "correctly" for static or dynamic linking if you omit it from a library, but if you include it you must link statically. If not…can we fake that somehow?

I agree that the mechanism for this would be to store in the swiftmodule a "guaranteed to be linked statically" flag, derived from whether -static is present when compiling.

5

Technically, no. However, the Microsoft linker will realize that is what happened and do so, emitting either warning 4217, 4826, or 4049 in the process. This will increase the binary size due to the indirect reference, the additional rebase entry in the IAT, the additional rebase itself, and the indirect call.

I really would rather this not be an expert mode flag. This is already needed to generate a static library, and its already plumbed through. This is not a new flag that I am proposing. But, yes, this was part of the reason that my original work focused entirely on getting the shared linking model working. The rest becomes much easier once that path works. Furthermore, with Swift Package Manager, this information is readily available to the build and we can pass down the information to the driver as appropriate.

6

I dont know if this will help at all, and i know this is a little out of topic, but right in the beginning of my mixed C++ and Swift codebase i have settled for use GN as a meta builder for Ninja avoiding using the swift builder, because i've just wanted to define the build rules for C++ and Swift in one place.

So i've just built a custom builder generator for swift on GN and now looking at this im kind of glad to have taken that route.

GN is Bazel inspired, but with a smaller and right to the point C++ codebase.

With it you have to define shared_library("library") or static_library("library") right on the module or you can have a variable to replace it for you according to some circunstances..

Like:
if (some_condition)
module = shared_library
else
module = static_library

than you can just wrap it with:
module("library")

Also its also easier to change the compiler to build, so it would be easy to try clang-cl + lld on Windows instead of the vanilla cl and link (you define this in a compile tool module that is parsed from the build manifest and used for everything, making it variable and with some flags that can be evaluated in runtime).

So back to the matter here, i wonder if part of those troubles would be solved if the IDL language used on Swift pm were more explicit about some things and powerful enough to build complex C++ codebases that happened to be dependencies of a swift project.

7

What I mean by "expert mode" is that you don't have to know about it to use Swift on Windows. (Maybe I should call it "intermediate mode".) The only people who have to know about it will be those building libraries (1) that aren't using SwiftPM (2), and because that's a subset of a subset I feel like it's okay if it's a bit subtle.

I don't love the idea of having to get it right, but the more I think about it the less the concrete use cases for trying to work around it hold up:

  • Reuse of object files between static and dynamic library targets. People probably never do this on Windows because of these issues, but it does come up sometimes on ELF and Mach-O systems. Even there, though, it has drawbacks: the very things that -static would help with. We're just living with those drawbacks today whenever we build static libraries.

  • Statically linking several modules' object files into a dynamic library, while still exporting their API. In that case we wouldn't want the other effects of -static anyway, so it's not actually a valid objection.

So maybe it's fine to just go with -static as a driver option, stored as a flag in swiftmodules (and probably turned into a per-item flag at the SIL level), and give up on the idea that the default "non-static" objects can still be linked statically if you want.

8

Yes, these flags matter purely to those that are either building Swift Package Manager itself or are building without Swift Package Manager at which point the assumption should be that they are aware of what it takes to build Swift code manually.

Right, this is not done on Windows in general. In the case that someone does that, we should treat the objects as if they were built for dynamic linking, and you live with the drawbacks. More importantly, we have given you the tools to handle this properly, and I would say that it is a safe assumption that Swift Package Manager will handle that correctly (once we have what it is nailed down) if you used Swift Package Manager. So this devolves into, you are not using the correct tools (which is the same as someone invoking the linker - do not do that, use the linker driver), use swift package manager to build.

This however is and interesting point. I didn't consider this aspect at all. How often do you expect this to occur? To educate me, when is this useful? That is, when do you want to build a library as static but re-export its interfaces? I suppose that this problem can be solved (via a DEFs file or an explicit set of /EXPORT: to the linker), so it isn't preventative, just inconvenient.

I failed to make this clear in the original post, so I apologize. This is not a new option. This is an existing option being used to inform code generation.

9

People keep asking about doing this on Apple platforms to break up their top-level framework into multiple modules, but still be able to distribute a single .framework bundle. (Most likely in that scenario you'd have a top-level module that re-exports the others.) In that case you're only grouping them for your client's convenience, which makes more sense for framework bundles anyway. If you're already distributing a DLL and a .lib and a .swiftmodule or .swiftinterface, distributing several of them doesn't seem like a terrible inconvenience.

And my response wasn't very clear either :-) I meant "given that we don't have a reason to change the model, we don't need to change the option, and if we're having the option, writing it into the swiftmodule seems fine".

10

In case it’s useful: some details for the implementation of this were discussed a while back in this thread (although no progress was made beyond that as far as I know): SPM Windows: Link issues during build - #24 by jrose

1 Like
11

Sorry to necromance this thread, but, I think that there was one last piece here that wasn’t explicitly discussed and I believe should be as it turns out to change the behavior slightly.

I think that the key insights needed here are that there exists a module that exports the sub-modules. This would be the module which imports the sub-modules. We always have the syntax that communicates the semantics: @_exported. The final piece is that this requires compiler support to orchestrate.

When an exported static module is identified (the swift module has the static bit already), the compiler would enumerate the public interfaces and mark them for export. This would add the symbols to the import library (tbd) and the dll (dylib). The additional swift interface/module files could be either distributed or can be merged. I’m not sure which of the two is preferable. However, this would fully allow the use of the static libraries to create a shared library or even export the symbols from an executable for something like -rdynamic.

Thank you for the discussion here, I think that the final result of this is going to be a really amazing experience on windows (and the results should benefit other platforms as well). Please let me know if I’m overlooking anything as I hope to try to finish this off soon. This is one of the few last pieces that I know that needs work.

2 Likes
12

Apologies for reviving this thread. Is there a work item that tracks the static linking for windows? I took a look at the GitHub issues under Apple/swift but could not find it.

Related: SE-0342: Statically link Swift runtime libraries by default on supported platforms - #10 by masters3d

13

I think that there might be an issue for that on the Swift repository. Support building static libraries on Windows · Issue #5692 · apple/swift-package-manager (github.com) tracks the SPM support work, but that currently requires some input from @abertelrud.

If you are just interested in the current state:

If you are looking for building and using static Swift libraries, not a static standard library, and if you are using CMake or bazel you can use static libraries.

1 Like
14

I have been reading through the examples for quite a while.

What I never figured out is how to statically link to a Swift static library without having the source of that library.

I am building a game engine in Swift and users will need to import the engine library so they can use it. I don't want to share the sources for multiple reasons, so I need a way to just grab the module and lib files and link them.

15

That involves a bit more effort - namely you need to explicitly pass along the path information to the swift invocation (e.g. swift build -Xlinker -I %SourceRoot%\third_party\library\include -L%SourceRoot%\third_party\library\lib). The long term desire here is to enable this through nuget integration so that a library can be fetched from a nuget repository and integrated into the build directly.

16

Is there any example for this ?

17

Sure, swift-format, swift-driver, swift-package-manager, sourcekit-lsp are all great examples for this IMO. IIRC, it is best documented in swift-driver though (at least that is where I believe I had written some documentation with explanations, but maybe it was sourcekit-lsp?).

18

I’ll look into that. Thanks.

Edit: Didn’t really find what I need. I’d love to see some project that actually imports a Swift static lib without having the source. Maybe vis .swiftmodule ?

19

For Swift static libraries, in addition to the .lib you need the .swiftmodule directory to be redistributed. The latter is passed via -I during the build.

20

Would that also enable code completion to work with sourcekit on Windows via SPM ?