Windows Nightlies

windows

(Saleem Abdulrasool) #1

Hello swift developers [and users],

I've been working on the Windows port of swift for a while now. A little bit late for Chistmas, but, they do say better late than never :stuck_out_tongue:.

I've setup a build for swift on Windows (x64) on Azure (pipelines). It is pretty crude, but a starting point none-the-less. I have it configured to build once a night at 0h00 PST (UTC-8), so we should have some attempt of nightlies for Windows! This has sufficient bits built and packaged to be semi-usable (I would like to add lld and possibly integrate VS Code integration bits too). It should have most of the tools as built by llvm (e.g. llvm-mt) including clang! I have the pipelines configuration available on GitHub if someone else wants to replicate it.

I am currently using the free tier, so the build is pretty slow and adding the test run might exceed the time permitted. I hope that the nightlies are useful to people who wish to try to use this as people had expressed interest in this as well as the concern of the barrier to entry that building posed.

Help is certainly welcome and appreciated. There is a lot to be done still, but this at least gets a stake in the ground!

As a status update, I am waiting for the tree to be unlocked for draining a large queue of patches for improving windows support and fixing various bugs.

My local test runs show ~220 failures still (~3400 passing tests). There are 6 failures in the standard library, 6 in the interpreter, and the rest are scattered throughout. I've been trying to get the test suite fully passing, but it is a lot of work for a single person, and I am trying to work across the entire project.

I do have libdispatch ported to Windows, along with the SDK overlay. A hello world libdispatch program works on Windows. There are no external dependencies now for libdispatch (everything comes from either the Windows SDK or LLVM, clang, or swift).

I have CoreFoundation ported to Windows (barring two patches). Similar to libdispatch, it is clear of dependencies (everything is part of the Windows SDK, LLVM, clang, swift, and libdispatch). I have a work in progress patch that has a snapshot of what my work to port Foundation to Windows.

Happy holidays!

CC @Torust @pvieito @tkremenek @Chris_Lattner3 @Michael_Gottesman @bgogul @Karl @blangmuir @akyrtzi


(Jeffrey Macko) #2

This is incredible Thanks !


(Pedro José Pereira Vieito) #3

Awesome, thanks @compnerd!


(Pedro José Pereira Vieito) #4

@compnerd It seems that the "https://dev.azure.com/compnerd/windows-swift/_packaging" section is not accessible.

After trying to log in with a Microsoft account it returns the following error: “401: NOT AUTHORIZED – Sorry, but XXX <xxx@xxx.com> (Microsoft account) is not authorized to access this page.”


(Saleem Abdulrasool) #5

Sorry, I'm still trying to figure out the whole pipelines stuff. I've adjusted it as of last night so that you can dig a bit and get the artifacts.

The REST API for build artifacts is useful.

https://dev.azure.com/compnerd/windows-swift/_apis/build/builds/<id>/artifacts?api-version=5.0-preview.5 gives you the blob that has the info on the artifacts. The downloadUrl should be accessible, e.g. for last night's build. I'm not sure how to tie that to the latest build id yet.


(Saleem Abdulrasool) #6

Amongst the things that I have just learnt: the new UI on azure devops has an artifacts button that will allow you to download the artifacts for a particular build so you don't have to go through the REST API. Just select the latest build that is complete/successful and the artifacts button should let you download it!


(发抖喵小咪) #7

Happy holidays!


(Pedro José Pereira Vieito) #8

Oh, great, thank you and happy new year! :slight_smile:


(Dave Yost) #9

I assume your Swift port participates in CLR? Have you considered doing your development within the open source mono CLR environment, so it's not Windows-only?


(Saleem Abdulrasool) #10

@daveyost, I'm not sure I follow. Why would you expect this to interact with the CLR at all? This is native target code, not MSIL, as it doesn't execute atop the CLR, but as native binaries. This work is geared towards supporting Windows as a target host.


(Dave Yost) #11

@compnerd, Did I mean CLI? In other words, Swift would be most useful on Windows if it could make a dll that can communicate with .NET Core libraries.


(Pedro José Pereira Vieito) #12

I was trying the nightly toolchain and I have the following problem:

PS C:\Users\pvieito\Downloads> cat main.swift
let s = "Hello World!"
print(s)
PS C:\Users\pvieito\Downloads> swift .\main.swift
<unknown>:0: error: could not load the swift standard library
PS C:\Users\pvieito\Downloads> swiftc .\main.swift
LINK : fatal error LNK1104: no se puede abrir el archivo 'swiftSwiftOnoneSupport.lib'
clang++.exe: error: linker command failed with exit code 1104 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1104 (use -v to see invocation)
PS C:\Users\pvieito\Downloads> swiftc .\main.swift -O
LINK : fatal error LNK1104: no se puede abrir el archivo 'swiftCore.lib'
clang++.exe: error: linker command failed with exit code 1104 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1104 (use -v to see invocation)

Is the Standard Library ready to use? Why are the static libraries missing at usr\lib\swift_static?


(Saleem Abdulrasool) #13

@daveyost, oh, I see what you are trying to do. I'm not particularly interested in that. Why would interop with .NET be useful at all? You have interop with C, which is far more valuable. If you really want, you can write a layer to P/Invoke bridge from C to .NET and expose that to Swift.


(Saleem Abdulrasool) #14

@pvieito the standard library is quite ready for use. The static libraries should be missing, the static library is not ready for use, you must use the shared standard library. The import libraries should be in usr\lib\swift\windows and the runtime bits in usr\bin. But it seems to be currently broken on the Windows build (that is, building for Windows on Windows rather than cross-compiling from Linux to Windows). I had a patch for that, but I think it has may have been lost in the pending 30 patches for when the tree is unlocked. I'll try to get a temporary patch into the build so that the next run has the import libraries.

I think that there is enough stuff now in place that others should be able to contribute to getting the windows hosted build working well :-).


(Pedro José Pereira Vieito) #15

So, after some digging I got it to "work":

  • The expected swiftCore.lib file is not the static library, but an import library which uses the same extension (hence my confusion + my zero Windows development background :slight_smile:). I got to generate the swiftCore.lib and swiftSwiftOnoneSupport.lib import libraries from their .dll counterparts and copied them to usr\lib\swift\windows (as the current build does not add them to the toolchain).
  • Move the libraries (DLL and LIB) from usr\lib\swift\windows to usr\lib\swift\windows\x86_64.
  • With that, swiftc .\main.swift -o main.exe works.
  • To run the generated main.exe I had to copy the .dll libraries from usr\lib\swift\windows to the executable directory (I suppose this could be solved with something like @rpath?).
  • I also downloaded the ICU libraries (they seem to be also missing from the toolchain).
  • After that, the executable runs but it does not print anything nor it exits, it seems to hang indefinitely:
PS C:\Users\pvieito\Downloads> cat .\main.swift
print(1)
PS C:\Users\pvieito\Downloads> swiftc .\main.swift -o main.exe
PS C:\Users\pvieito\Downloads> .\main.exe
# ... hangs indefinitely :(

(Saleem Abdulrasool) #16

I am hoping to get the import library stuff sorted out soon now that some of my queue can be drained. Glad that you have something to unblock you for the next couple of days.

Excellent, so at the very least, the compilation portion works.

There is no rpath equivalent on Windows. The current directory is searched for DLLs, as well as PATH. I think that adjusting the PATH so that the libraries are in the PATH is the nicest solution and is what I have been using.

Yes, the ICU is not repackaged ... I'm using the binaries from the ICU project so that they can easily be "replicated".

Interesting, the last time that I saw that behaviour it was due to a bad incremental build. Would you be able to attach windbg to it and figure out where it is hanging? I've had even libdispatch hello world working, so something is definitely different here and we should figure out why that is different.


(Pedro José Pereira Vieito) #17

Also, I think that the libraries (both DLL and LIB) should be in usr\lib\swift\windows\x86_64 instead of usr\lib\swift\windows as clang++ only adds that path to the linker step (-libpath:C:\Users\pvieito\Download s\toolchain\toolchain\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\lib\swift\windows/x86_64). In fact, after cleaning the toolchain only adding the LIBs to usr\lib\swift\windows does not work, I had to move all the libraries to usr\lib\swift\windows\x86_64.

In the main thread it seems to be waiting in an NtWaitForKeyedEvent call (from swift_stdlib_random):

And in the other two threads it is in NtDelayExecution calls (from the debugger?):


(Saleem Abdulrasool) #18

The dll is not needed at all until the runtime, so there is no need for the dll to be in the resource dir. In fact, the DLL should not be available at all, it should just be part of a redistributable for clients only.

The top of the stack isn't interesting ... the middle of the stack is. It seems that the initialization path is causing a call to random which is hanging? The heap object being constructed is the interesting thing to look into I think.


(Thomas Roughton) #19

Did you manage to get that sample code snippet I posted (https://gist.github.com/troughton/9fdecf4dc09b0406c444c28f40f80d43) working? Last I tried on Windows it had similar behaviour - hanging mid-execution, with the behaviour differing depending on slight changes in the standard library or whether the executable was built in debug or release mode.

It might be worth noting that this worked back on Swift 4.2 with static linking – something might be off in how dynamic linking is implemented.

It might also be the same memory corruption issues you mention in https://github.com/apple/swift/pull/21566. When I tried to debug it I found that the corruption occurred in a call to Builtin.allocWithTailElems_1 with a generic parameter, but I didn’t manage to track down the root cause.


(Saleem Abdulrasool) #20

Sorry, I haven't had a chance. Been trying to get the test suite passing (and fixing all the compiler, runtime, tests, etc), porting CoreFoundation and Foundation, and setting up the nightlies. Please file a bug on JIRA for it.