Swift C++ Interoperability & Windows DLLs

[hwjeremy (Hugh Jeremy)]

Hi everyone! Swift represents a tremendous opportunity for the game development community. Its characteristics make it very well suited to game engine and gameplay code. So imagine my excitement when I saw the news about C++ interoperability.

A practical example of the potential of C++ interoperability in games is the use of Swift to write games with Godot, a C++ game engine. SwiftGodotKit is a tremendously exciting project, and works brilliantly on macOS. C++ Godot symbols are available in Swift and it is magical to behold.

However - Swift will not reach its potential as a game development language if it does not work well on Windows. Much as I love the productivity improvements Swift brings, limiting a game release to macOS only is not often commercially viable.

Right now, it appears that the limiting factor for Swift game development is an inability to perform Swift/C++ interop with Windows DLLs.

To use the SwiftGodotKit example, libgodot.xcframework makes the underlying .dylib available to Swift using all the behaviours mentioned in Swift.org documentation, such as a modulemap. However, I can't find any information about whether such behaviours can be emulated in the Windows DLL context.

Is anyone able to share more information or knowledge about the current state of interop with Windows DLLs, or any plans for the future? While Windows DLLs are certainly not my strong suit (as evidenced by my clumsy phrasing in this post), I would be thrilled to help out any way I can.

• Hugh

[Max_Desiatov]

cc @Alex_L @compnerd

[compnerd (Saleem Abdulrasool)]

@hwjeremy

Right now, it appears that the limiting factor for Swift game development is an inability to perform Swift/C++ interop with Windows DLLs.

This is a rather strong statement. What is the basis for this claim?

@hwjeremy

To use the SwiftGodotKit example, libgodot.xcframework makes the underlying .dylib available to Swift using all the behaviours mentioned in Swift.org documentation, such as a modulemap. However, I can't find any information about whether such behaviours can be emulated in the Windows DLL context.

I think that there’s a fair amount of mischaracterization of technical details here. The dylib is not “made available”. The module map provides context to the C++ compiler (clang) as to how the headers relate to the module. These are then bridged to swift. The dylib does not take part in this at all. The dylib is used at runtime (assuming a fully modern framework with a TBD, otherwise, yes, technically the dylib is also used at link time).

There’s no need for emulation of any of this for windows. Windows is properly supported as a platform. Dynamic linkage works just fine on windows.

@hwjeremy

Is anyone able to share more information or knowledge about the current state of interop with Windows DLLs, or any plans for the future?

Hopefully the above details are helpful in providing additional knowledge about C++ interoperability. As to the state on windows, there are some small issues yet, but for the most part, is usable.

You may be interested in Interoperability: Swift’s Super Power which was also reposted to the swift blog at Swift.org - Swift Everywhere: Using Interoperability to Build on Windows.

While the examples used static linking for other purposes, dynamic linking also sites function, but requires that the user configure the module and the build of the C++ code properly.

Additionally, the way that swift works on windows is that it heavily relies on the C/C++ interoperability to bridge the OS functions. Things like
GitHub - compnerd/DXSample: Sample Program for DirectX 12 + Swift directly link to the windows SDK libraries dynamically.

You may also be interested in GitHub - migueldeicaza/SwiftGodot: New Godot bindings for Swift which some to improve the ergonomics of gadot in swift.

[hwjeremy (Hugh Jeremy)]

@compnerd I really appreciate your reply. If you can help me understand how to make this Windows DLL interop work, I truly think it could be a really big deal for Swift adoption - Cross platform game development in Swift would become possible in a very practical sense.

Blockqoute There’s no need for emulation of any of this for windows. Windows is properly supported as a platform. Dynamic linkage works just fine on windows.

Could you provide a practical example of this in package.swift form? Suppose I have a DLL /some/bin/example_library.dll compiled by MSVC. How can I make that DLL available to Swift code? Is there no need for a module map?

In the case of SwiftGodotKit, we can add the pre-compiled binary libgodot .xcframework created with clang and like so:

        .binaryTarget (
            name: "libgodot",
            url: "https://github.com/migueldeicaza/SwiftGodotKit/releases/download/v4.1.99/libgodot.xcframework.zip",
            checksum: "c8ddf62be6c00eacc36bd2dafe8d424c0b374833efe80546f6ee76bd27cee84e"
        ),

How can I do the same thing with a pre-compiled Windows DLL create with MSVC?

I appreciate this question might sound simplistic, even silly, to those well versed in Windows DLLs, linkers, cross-platform Swift and the depths of compilers. But if it is indeed possible, those of us operating on a lesser plane (such as mysef), a practical example would be very much appreciated.

Blockquote You may also be interested in GitHub - migueldeicaza/SwiftGodot: New Godot bindings for Swift which some to improve the ergonomics of gadot in swift.

You've linked back to the library which I provided as the example in the second paragraph of the original post. Miguel's amazing work is the direct impetus for my post. It's pure magic on macOS. I want to be able to use the library you've linked on Windows. To do so, I need to be able to call into a Windows DLL.

Thank you again for replying - I'm so very hopeful we'll get this working and the nirvana of cross-platform Swift game development can be achieved.

[compnerd (Saleem Abdulrasool)]

@hwjeremy

You've linked back to the library which I provided as the example in the second paragraph of the original post. Miguel's amazing work is the direct impetus for my post. It's pure magic on macOS. I want to be able to use the library you've linked on Windows. To do so, I need to be able to call into a Windows DLL.

Ah, sorry, I was reading on a mobile device and didn't realize that the link was back to Miguel's project. I know that others have experimented successfully with building the project on Windows (but does require the latest toolchain).

@hwjeremy

Could you provide a practical example of this in package.swift form? Suppose I have a DLL /some/bin/example_library.dll compiled by MSVC. How can I make that DLL available to Swift code? Is there no need for a module map?

There is nothing special about C++ here, it is the same as C. All the examples for interoperability with C libraries applies. The complication here is not the C/C++ interop but rather SPM. The XCFramework functionality is limited to macOS., and there is still a hole in SPM support as we do not have support for nuget for the distribution of the prebuilt binaries that is common on Windows (e.g. WinUI and the WinSDK are both distributed as nuget packages).

The example that I have on hand does not use SPM but rather CMake. For SPM, you would have to manually replicate the operations. Effectively, you are responsible for selecting the location of the dependency (e.g. %UserProfile%\SourceCache). Once you have the library, you will need to identify the root of the include tree and implant the module.modulemap for the module. This prepares the module for consumption. At that point, when you build with SPM, you will need to pass along the include path and the linker search path for the associated import libraries, e.g.

swift build -I %UserProfile%\SourceCache\dependency-development\usr\include -Xlinker -L%UserProfile%\SourceCache\dependency-development\usr\lib

assuming that you have built a version of the library with the traditional Unix layout, extracted it to %UserProfile%\SourceCache with the name of dependency-development.

This is the generic means for including a binary dependency for use in a SPM package and apply to any package really. The same mechanism is well documented and used in the build of the swift-driver.

[hwjeremy (Hugh Jeremy)]

I can't tell you how much I appreciate your responses @compnerd. As an aside to give colour to my quest - Last night, I thought "Oh this is just too hard, I'll go back to Unreal/C++". A few files deep in C++ and I was missing Swift's powers so much I closed that too.

A key issue to finding a solution, I think, is that this problem domain is just outside the bounds of my current technical competencies. So from your perspective, these things appear quite simple, but for me, they're quite challenging. I don't work in Windows often!

If we go back and forth a few times, I bet we can get practical solution, and start slinging some polygons on Windows.

@migueldeicaza is kindly providing guidance from the perspective of SwiftGodotKit in a GitHub issue, and he has also corrected my erroneous assumption that C++ interoperability is salient to this problem.

Perhaps we could frame the problem in terms of practical steps. I'll break your paragraph "The example that I have..." out into a sequenced list.

1. Select the location of the dependency - I read this as, "Find the location of the headers for the installed DLL".
   i. I'll need to research this more, but I'm further assuming that ultimately there will be a directory on the system, which you identify as %UserProfile%\SourceCache, containing the libgodot headers.
   ii. Compiling DLLs is probably my most critical weakness, with respect to the competencies required to make this work. In particular, I'm concerned at the note "assuming you have built a version of the library with the traditional Unix layout". Given libgodot is compiled by MSVC, I imagine there is a risk it is not using a "traditional Unix layout". I note that the output of compilation is a .dll, an .exp, and a .lib, I'll need to do more learning and research here. There don't appear to be many resources on the public internet about implanting modulemap files, so I really am at your mercy in this regard, and value any wisdom you can provide.
2. Identify the root of the include tree - I read this as identifying a particular directory in %UserProfile%\SourceCache created in (1).
3. Implant the module.modulemap file - I read this as, place the module.modulemap file, the creation of which for macOS Miguel provides an example for using gdextension_interface.h, at the root of the include tree.
4. Build with SPM - This step appears crystal clear to me, thank you for providing the example invocation.

If you could correct any mistakes I am making above, I would be terribly appreciative. This does appear to be crystallising into a practical operation that I can begin to experiment with.

I know you must have many pressures on your time, and I assure you I don't expect you to serve me answers on a silver platter. If we can iteratively bridge the gap between your knowledge and mine, I think we'll have a solution of tremendous interest to the wider game development community.

[compnerd (Saleem Abdulrasool)]

A little out of order, but a bit of non-linearity doesn't hurt right?

@hwjeremy

Identify the root of the include tree - I read this as identifying a particular directory in %UserProfile%\SourceCache created in (1).

Yes, this is the direct consequence of the first step.

@hwjeremy

Select the location of the dependency - I read this as, "Find the location of the headers for the installed DLL".
i. I'll need to research this more, but I'm further assuming that ultimately there will be a directory on the system, which you identify as %UserProfile%\SourceCache, containing the libgodot headers.

Right, ultimately there will be a file system location that needs to be identified. I chose a "random" location (I personally use a separate physical drive for development which is at S:, and put all the cloned sources into SourceCache).

The difference is that I expect this location to not only contain the headers (include), but the import library (lib) and the runtime component (bin).

@hwjeremy

ii. Compiling DLLs is probably my most critical weakness, with respect to the competencies required to make this work. In particular, I'm concerned at the note "assuming you have built a version of the library with the traditional Unix layout". Given libgodot is compiled by MSVC, I imagine there is a risk it is not using a "traditional Unix layout". I note that the output of compilation is a .dll, an .exp, and a .lib, I'll need to do more learning and research here. There don't appear to be many resources on the public internet about implanting modulemap files, so I really am at your mercy in this regard, and value any wisdom you can provide.

This is technically beyond the scope of this forum. The compilation of Godot is not a Swift issue at all, but a Godot development issue. If you build with MSVC, IIRC, the default output directory is $(SolutionDir)$(Configuration), so under Debug or Release there should be a bin directory with the .dll file which is the runtime component. There should be near it, a .pdb possibly (debug information), as well as the .lib which is the import library. The headers will need to be coalesced manually depending on how the project is setup. These four pieces should go into the appropriate location.

Again, this has absolutely nothing to do with Swift, but is a general packaging issue of how does one get a copy of Godot. This packaging is what I was referring to earlier about the nuget (which you can find an example of being built at Sentry Native SDK for a different package).

@hwjeremy

Implant the module.modulemap file - I read this as, place the module.modulemap file, the creation of which for macOS Miguel provides an example for using gdextension_interface.h, at the root of the include tree.

Yes, once the package is staged, you will need to add in the module.modulemap at the root of the include tree (include).

@hwjeremy

1. Build with SPM - This step appears crystal clear to me, thank you for providing the example invocation.

[hwjeremy (Hugh Jeremy)]

Thank you Saleem, this is wonderful guidance. I'll take the time to absorb it properly and have go at performing the operation. You're absolutely right to point out that some aspects are Godot-specific, I'll try to keep any follow up questions scoped to this forum.

[hwjeremy (Hugh Jeremy)]

Hi again everyone,

I've made some good progress, but I think I've hit a road block, the navigation of which may require some more guidance. Any input anyone could give would be much appreciated.

Following Saleem's advice above, in pursuit of building SwiftGodotKit on Windows, I constructed a library structure like so:

libgodot/
├── bin/
│   └── libgodot.dll
├── lib/
│   └── libgodot.lib
├── pdb/
│   └── libgodot.pdb
└── include/
    └── libgot/
        ├── gdextension_interface.h
        └── libgodot.h
    └── module.modulemap

... with modulemap:

module libgodot {
    header "libgodot/libgodot.h"
    export *
}

I provided the libgodot library to SwiftGodotKit package.swift as a .systemLibrary target:

.systemLibrary(
    name: "libgodot",
    path: "libgodot/include
)

And then invoked swift with arguments:

swift build -Xlinker -I"libgodot/include" -Xlinker -L"libgodot/lib"

Compilation proceeded as follows:

Building for debugging...
Build complete! (0.76s)
warning: unable to create symbolic link at C:\Users\Hugh\Desktop\SwiftGodotKit-main\.build\plugins\debug: Error Domain=NSCocoaErrorDomain Code=256 "(null)"
Building for debugging...
error: compile command failed due to exception 3 (use -v to see invocation)

With the following error output, the "output" body of which I've snipped due to its verbosity:

error: failed parsing the Swift compiler output: unexpected JSON message: {
  "exception" : 3,
  "kind" : "abnormal-exit",
  "name" : "compile",
  "output" : "\u001b[1m<unknown>:0: \u001b[0m\u001b[0;1;31merror: \u001b[0m\u001b[1merror opening 'C:\\Users\\Users\\Hugh\\Desktop\\SwiftGodotKit-main\\.build\\plugins\\outputs\\swiftgodot\\SwiftGodot\\CodeGeneratorPlugin\\GeneratedSources\\generated.swiftdeps' for output: no such file or directory\r\n\u001b[0mAssertion failed: loadedGraph.has_value() && \"Should be able to read the exported graph.\", file C:\\Users\\swift-ci\\jenkins\\workspace\\oss-swift-windows-toolchain\\swift\\lib\\AST\\FineGrainedDependencies.cpp, line 237\r\nPlease submit a bug report (https:\/\/swift.org\/contributing\/#reporting-bugs) and include the crash backtrace [ **snip** ],
  "pid" : 32024,
  "process" : {
    "real_pid" : 32024
  }
}: dataCorrupted(Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "kind", intValue: nil)], debugDescription: "invalid kind", underlyingError: nil))

My initial impression is that the warning regarding symbolic link creation failure is the proximate cause of the compilation failure, and I intend to investigate that. However, the manner in which swift is crashing here has lead me to post here, as I've found that the swift compiler generally crashes gracefully.

Does anyone have any thoughts about what's happening here, and how I could investigate further?

[Max_Desiatov]

This looks like an issue with invoking SwiftPM plugins, which I guess are used by SwiftGodotKit in some way? Would you be able to reproduce this with a small self-contained example project and create an issue with it on Issues · apple/swift-package-manager · GitHub?