Foundation on Windows (CI/Nightlies)

Here by popular demand, Foundation on Windows, built "continuously"!

I have finished improving the build rules for the Windows build so that the Azure hosts can have automated builds of Foundation for Windows. These builds are scheduled to run "nightly". Furthermore, assuming that the Windows build remains green, I have configured the builds to be triggered on the toolchain build (so, roughly once ~90m). This at least gets a starting point for the SDK.

Hopefully, I will get some time to setup MSIs for this build as well so that Foundation would also be available in a convenient MSI form.

The Windows SDK (which currently builds libdispatch, libxml2, libcurl, and Foundation) is available here: Azure DevOps Services | Sign In.

As usual, help to improve this is welcome.

CC: @millenomi @Michael_Gottesman @tkremenek

21 Likes

That's great, thanks, @compnerd!

such a great news! :star_struck:

I believe that the issue was reported by you and @pvieito that the import library for Foundation is not preserved. The install rules for Foundation now handles that properly. I've verified that the SDK image (not a MSI :slightly_frowning_face:) from the Windows SDK build contains the import library. The rest of the Foundation image has been tweaked to install the pieces to the right location for Windows as well (i.e. dlls in the binary directory). This should be ready for use barring bugs in Foundation itself, which I'm still trying to work through.

Additional hands on Foundation are welcome :wink:

2 Likes

Thank you for your great efforts. Currently, there does not seem to be an artifact of Foundation available on Azure DevOps Services | Sign In (at least if I do not scroll very far, and when scrolling to far, how can I know it is not to old to work with the current Swift version?). Also, how to get and install Foundation is not explained on swift-build/Windows.md at master · compnerd/swift-build · GitHub (that would be very helpful, I consider Foundation as an important part of Swift). So maybe you (or someone else) could regularly post the last successful builds of the toolchain, Foundation, and the Runtime in the „same“ version somewhere? (Others proposed swift.org to be a good place, even if marked as „experimental“ on that site, but of course this should be answered by the Swift core team (@tkremenek?).)

I am currently using this build.
https://dev.azure.com/compnerd/windows-swift/_build/results?buildId=5458&view=results
And I haven't had a problem with it yet.
I use a toolchain that was built on the same day.
https://dev.azure.com/compnerd/windows-swift/_build/results?buildId=5452&view=results

1 Like

Patches to improve the documentation are more than welcome :-). As to the builds for Foundation being out of date, that is due to a build issue - I've not had time to look into it. Basically, Foundation requires CURL built with zlib, and the builds currently do not link against zlib resulting in unresolved symbols. I can probably put in a quick workaround for this, though, the best solution is to help move everything to CMake 3.15+.

FYI, the x64 build has been repaired. The ARM64 builds are still a problem due to a clang issue.

Thank you @Daniel_Mullenborn for the links.

I tried to mix the content of sdk-windows-x64.zip into the content of toolchain.zip. Before doing this, I got the error "no such module 'Foundation'". Now, it get the following error (first line says the file 'Foundation.lib' cannot be opened):

LINK : fatal error LNK1104: Datei "Foundation.lib" kann nicht geöffnet werden.
clang: error: linker command failed with exit code 1104 (use -v to see invocation)
:0: error: link command failed with exit code 1104 (use -v to see invocation)

So how do I add the contents of sdk-windows-x64.zip to the content of toolchain.zip, i.e. how to I add Foundation and maybe other libraries to my already installed Swift toolchain?

By the way, what is the "windows-sdk.msi" installer doing, does it install Swift + Foundation, and does it add e.g. the file "module.modulemap" etc.? Should I use the installer instead?

@compnerd: The point for me is not that some builds go wrong, but that it would be nice to get the same successful builds (same version) for Swift, Foundation, and the Runtime easily even if they do not contain the very last changes.

Yup, it would be great to have a full toolchain installer or archive similar to the ones in swift.org. Currently you have to download multiple ones a mix and match some artifacts.

@pvieito: To be fair, that should not be a critique to @compnerd, who is doing a great job; mixing those directories on a regular basis could be done by someone else (if this could not somehow be integrated into the process). Currently, I just would like to have a clear explanation of how to mix them (the obvious way I tried did not work, or I made some stupid mistakes).

Yeah, of course, we should be very thankful for the great job @compnerd has done!

Now that the port is very advanced I also think we should start asking about official Windows support (including proper GitHub PR CI support and official toolchains in Swift.org) to Swift Project Lead (ie. Apple and @tkremenek).

2 Likes

Thanks for that feedback. I was hoping that the Windows instructions would be sufficient for the explanation of what was going on. The split of the components is intentional as I believe that it provides better building blocks (the pieces are setup to be composed into a final image). The desired break down is as follows:

  • toolchain: the actual tools needed to compile C/C++/Swift code and inspect binaries (e.g. clang, swiftc, lld)
  • sdk: the Swift supplemental content needed to build software for a platform (e.g. Windows, Android, Linux)
  • developer tools: tools needed to write software (e.g. llbuild, s-p-m, SourceKit-LSP)

Currently, the last component is not available, but, I hope to add that to the builds (although some part of the build does exist currently).

You would need at least the first two items for writing Swift code.

The first item - toolchain - needs to be for the machine that you are going to be working on (currently, Windows x64 or Linux x64; Windows arm64 is experimental).

You additionally need the SDK for the platform you are building for (currently one of Windows, Android, or Linux). There is some discussion going on at Swift Linux layout considerations (aka Linux is difficult, lets go shopping) about how to deal with multiple architectures for these platforms. With the patch which led to that thread, it is possible to have these SDKs be relocatable wherever you like - you just pass along some flags to the compiler to identify where they are stored.

Finally, I suspect is the piece that most people claim is absolutely required: the developer tools. This is currently under development still and includes items for the developer not the final product. These are tied to the machine that you are building on again (just like the toolchain). It would include tools like s-p-m and SourceKit-LSP.

Why devtools is split out

The reason to split out the developer tools like this is two fold. First of all, these are high level tools which are written in Swift, and as such need the SDK to be prepared for the host, before they are usable (that is, we need the previous component for Windows before we can build these tools for Windows). Second of all, they are not part of the core toolchain, but really part of the IDE experience.

My long term vision (plan?) is to actually get the developer tools stable and then try to construct a single VSCode build which has the toolchain and devtools pre-packaged and configured. You would just add the SDK for the platform that you wish to target and can start developing :-). It should be possible to actually have a plugin in VSCode which would download the tools (much like how Xcode downloads the simulator images or Android Studio downloads pieces).

Each one of these layers is built upon the previous and it takes time to ensure that the base is solid enough to support that. I have been slowly moving through the strata building up the infrastructure to make that possible. In the mean time, I have been simultaneously trying to make all the pieces entirely relocatable, which makes it easier for others to actually experiment with the artifacts that are being published.

If the current documentation on the getting started workflow is not sufficient, I would love to have some patches to improve them and make them clearer for others. I am trying to find a way to lower the bar for others to use Swift on the platform of their choice (both for writing and running). If there are pieces which I have overlooked in the short term, I would be open to trying to smooth those out in a way that makes sense for the longer term.

6 Likes

Great outlook.

Please do not forget that the environment for running the resulting program should also be available in a compact, reduced form, or maybe this would be resolved with static linking (which is not available yet?)? I am talking here about running on a "naked" Windows (without any Visual Studio or additional Windows Runtime installed), i.e. it is should be easy to build a compact, small package that is runnable as-is (without installation) on a freshly installed Windows.

Right now you only need Visual C++ Redistributable for Visual Studio 2015 to run Swift on Windows.

@Daniel_Mullenborn: It seems to be OK (for Microsoft) to just put the needed DLLs (that are also contained in this "Visual C++ Redistributable") into the according directory of the application see https://msdn.microsoft.com/en-us/data/dd293565(v=vs.85). But yes, you are right in principle, thanks. But what is still needed is in extraction of DLLs from Swift/Foundation or what else is needed, the whole Developer directory is just too much.

I think if you have these files in the folder it should work.

BlocksRuntime.dll
concrt140.dll
dispatch.dll
Foundation.dll
FoundationNetworking.dll
FoundationXML.dll
icudt64.dll
icuin64.dll
icuio64.dll
icutu64.dll
icuuc64.dll
msvcp140.dll
msvcp140_1.dll
msvcp140_2.dll
swiftCore.dll
swiftDemangle.dll
swiftMSVCRT.dll
swiftRemoteMirror.dll
swiftSwiftOnoneSupport.dll
swiftWinSDK.dll
vccorlib140.dll
vcruntime140.dll

All together about 50MB.

1 Like

@Daniel_Mullenborn: Thanks.

All that ? is this what's needed to run the compiled .exe program ? seems to be a lot.. are there plans to shrink that, by a lot, on same level as rust or even c++ ?