Swift Soars Ever Higher

compnerd · February 23, 2020, 7:41pm

A year ago, to the day, I posted A Swift Takes Flight. In it, there was the announcement that Foundation was possible to build on Windows. This was a pretty large milestone in supporting Windows as a platform. Foundation is a complex codebase that has very interesting cross-language and cross-object-model interactions (CoreFoundation bridging).

I am extremely excited to announce that as of today we have the full test suite for Foundation passing on Windows as well! Although there are a few tests which are testing Unix specific behaviors which do not port, this is largely on parity with the Linux test suite. There are a total of 1706 tests on Windows that are running with Linux currently having 1714.

I think that it should be pointed out that in the recent months this effort has required work in Foundation and CoreFoundation and that the output of the compiler has not even been a concern. To me, this is an indication that the Windows support in the compiler and standard library has largely matured to the point where it is reliable. This is a huge step forward for the idea of portable code written in Swift, and should hopefully make it easier to bring codebases to all the platforms easily and uniformly.

This work would not have been possible without the tremendous help of @millenomi. Also, thanks to the various people who have tried to improve the state of the Foundation test suite on Windows (like @gmittertreiner, @spevans, and @lxbndr amongst others).

And to keep the tradition of screenshots alive:

CC: @tkremenek

Spencer_Kohan · February 24, 2020, 7:50am

For everyone involved: bravo, and thank you so much for your contributions on this. I have several use cases where I would be very happy to use Swift on Windows, and it is extremely heartening to see progress in this direaction.

Is there any way to follow the roadmap and progress on this effort? I would be very happy to understand, for example, the state of SPM support on Windows, and if/when we might be able to build a swift package which would work on Linux. Windows and Apple platforms.

BastianInuk · February 24, 2020, 9:57am

This is exciting news! Does individuals have to compile Swift and Foundation themselves or are there readily available packages conveniently available?

I also remember not having the neat swift build syntax, does this improve that in any way?

As always, thank you guys for this amazing achievement!

realdawei · February 24, 2020, 2:43pm

This is a fantastic milestone. Congratulations to everyone involved

compnerd · February 24, 2020, 4:15pm

There are prebuilt binaries and SDK support for Foundation on Windows in the Azure builds, so you do not need to build it yourself in order to use it.

No, swift build is part of swift-package-manager, which is not yet in a state that it can be easily ported to Windows. However, a number of projects have adopted CMake based builds which enable building with ninja.

compnerd · February 24, 2020, 4:21pm

Currently, the best approach for this is to monitor the commits that are going into master. There is no explicit roadmap since there are not many other contributors who are continuously active on the Windows port and there is a lot of ground to cover. Unless there needs to be coordination because there are sufficiently many other contributors, its not a very effective use of time to add all the additional planning overhead.

swift-package-manager is not currently in a state which makes it easy to port to Windows, and so it has been something which I have attempted a few times but never made sufficient progress to be able to complete a port. Others may have a better experience than I.

You should be able to do that now. Using CMake, you could have a single build system that can target all three platforms. You should be able to use the standard library, libdispatch, Foundation, and XCTest for writing packages which work on all three platforms.

Spencer_Kohan · February 24, 2020, 7:22pm

Thanks for the information. I might try to do a trivial example this weekend.

Torust · February 24, 2020, 8:49pm

Would you be able to give a quick overview of what sorts of problems you were running into? It looked like you got pretty far in Swift-package-manager on Windows? Sure Why Not! - as far as you know, are things still working to that degree, and how many of those pending patches made it in? Judging from Pull requests · apple/swift-package-manager · GitHub, it seems like a lot of the patches were approved and just never merged.

Aciid · February 24, 2020, 10:50pm

There have been many steps in the direction of making SwiftPM more portable:

SwiftPM can build cleanly with CMake.
Most of SwiftPM's FS abstraction goes through Foundation now. Last I checked, there's just two remaining method that use low-level FS APIs.
SwiftPM now compiles package manifest instead of interpreting them.
The old POSIX compatibility layer is gone.
SwiftPM's Process delegates to Foundation on Windows.

It would be great if we can start collecting the remaining tasks for completing the SwiftPM port under this JIRA.

Torust · February 25, 2020, 7:37am

To answer some of my own questions: LLBuild, ToolsSupportCore, and SwiftPM all build fine under CMake on Windows. The first problem I ran into was that AbsolutePath doesn't seem to be aware of Windows path formats; Path: use `isAbsolutePath` instead of re-implenting by compnerd · Pull Request #2294 · apple/swift-package-manager · GitHub goes some of the way towards solving this. I'll update the JIRA and/or submit PRs as I run into more problems.

compnerd · February 25, 2020, 5:32pm

@Torust, yes, the packages build, however, getting from building to working and stable is another journey :).

The file system representation in swift-package-manager is designed around the Unix semantics and those do not carry over very well for Windows. You can have NT paths, device specific paths, UNC paths, drive relative paths, non-canonical path separators in paths, DOS 8.3 style paths, extended paths, etc all pointing to the same file but are treated entirely different. As an example:

\??\S:\BinaryCache\Release\Windows-x86_64\swift-package-manager\bin\swift-build.exe
\\UNC\S:\BinaryCache\Release\Windows-x86_64\swift-package-manager\bin\swift-build.exe
\BinaryCache\Release\Windows-x86_64\swift-package-manager\bin\swift-build.exe
S:/BinaryCache/Release\Windows-x86_64\swift-package-manager/bin/swift-build.exe
\\?\S:\BinaryCache\Release\Windows-x86_64\swift-package-manager\bin\swift-build.exe
\\.\Devices\HardDisk1\BinaryCache\Release\Windows-x86_64\swift-package-manager\bin\swift-build.exe
\\UNC\localhost\S:\BinaryCache\Release\Windows-x86_64\swift-package-manager\bin\swift-build.exe
S:\Binary~1\Release\Window~1\swift-~1\bin\swift-~1.exe
S:\Binary~1\Release\Window~1\swift-~4\bin\swift-~2.exe

are probably all the same file path. However, the way that they are handled in tools-support-core does not account for that.

There are other file system issues like realpath, dirname, basename, etc do not currently function (e.g. while path.isRoot { path = path.parentDirectory } currently infinitely loops!)

There are other assumptions in swift-package-manager, such as the ability to use which (there is no which, where is similar, but a builtin shell function, which means you cannot exec it, you can do a cmd /c "where ..." but, that has its own set of problems - e.g. command line limits, environments, permissions, etc).

There are hardcoded tools in swift-package-manager (e.g. swift and clang) which cannot be used, as the .exe suffix is important, especially when there is an ambiguity due to directory and executable being named the same modulo the suffix. Furthermore, clang on Windows is a rather scary idea, and should instead be using clang-cl to ensure that INCLUDE and LIB are honoured correctly and that the ABI is conformed to properly.

Static linking is prevalent in swift-package-manager, but is not currently supported on Windows.

Although this something you can work around, llbuild assumes that the archiver is named ar rather than lib, or preferably llvm-ar. Fortunately, modern Unix has eschewed the librarian in favour of the archiver, so ranlib is not needed (and similar on windows, lib handles both tasks).

I doubt that there is anything insurmountable, but they add up very quickly.

CC: @Aciid

Torust · February 25, 2020, 9:10pm

Thanks - that should be plenty to go off. One other quick question: how do you control the -libc parameter via CMake (for debug builds of SwiftPM/TSC with the release Swift toolchain)? -DCMAKE_Swift_FLAGS="-libc MD" adds it to the command-line, but it gets overridden by the -libc MDd inserted elsewhere.

compnerd · February 26, 2020, 12:42am

You can control it explicitly if you like via -D CMAKE_MSVC_RUNTIME_LIBRARY=MultiThreaded[Debug][DLL]. The standard CMake documentation covers the values (they map as expected - /MT, /MTd, /MD, /MDd).

flowtoolz · February 28, 2020, 8:52am

I couldn't even find the repo. Can someone post a link? Thanks

compnerd · February 28, 2020, 4:59pm

swift: GitHub - apple/swift: The Swift Programming Language
swift-corelibs-foundation: GitHub - apple/swift-corelibs-foundation: The Foundation Project, providing core utilities, internationalization, and OS independence

tkremenek · March 2, 2020, 7:58pm

I had meant to comment earlier when this thread showed up, but I just wanted to echo the acclaim here on this thread that this is a fantastic step forward for the project.

zienag · March 3, 2020, 1:42pm

SwiftPM's Process delegates to Foundation on Windows.

I had a problem with using Process from Foundation API, briefly described in foundation - Getting stdout from Process (NSTask) sometimes fails - Stack Overflow. TLDR, Foundation API doesn't have a way to retrieve all output from command synchronously. I used SwiftPM Utility module to solve it. Have you found a way to overcome this? (It is not specific to OS, but API itself don't have a way to describe the intent)

spevans · March 3, 2020, 1:58pm

The tests for Process in swift-corelibs-foundation use this code: swift-corelibs-foundation/TestProcess.swift at 3db863830c6244aed78ee065e8d4d8661ccc6df6 · apple/swift-corelibs-foundation · GitHub which you may find useful.

zienag · March 3, 2020, 4:09pm

Hmm, it seems like appending remaining data to output does the trick. Thanks!

pvieito · March 21, 2020, 4:13pm

@compnerd Hi! I was trying the new Swift for TensorFlow Windows installer (great improvement on user experience ) and it seems that currently you have to define the libraries, headers and resources location in each invocation of swiftc:

> swiftc Z:\Documents\Developer\Other\SwiftBase\SwiftBase\main.swift ^
    -resource-dir C:\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk\usr\lib\swift ^
    -L C:\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk\usr\lib\swift\windows ^
    -I C:\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk\usr\include ^
    -o main.exe && main.exe

Is there any environment variable to set these paths as the defaults? Also I would rather use PowerShell instead of the cmd do you know if it is posible to set the configuration of the x64 Native Tools Command Prompt into PowerShell?

Finally, is there any way to run the Swift REPL on Windows? It seems to be posible to run it, but it crashes each time you send any command with the following error:

Assertion failed: false && "called into swift language runtime stub", file S:\14\s\toolchain\lldb\source\Target\SwiftLanguageRuntime.cpp, line 296