CI for Packages on Windows & Android?

Using Swift
, ,
32

Thanks for exploring this and writing up the findings.

The reason that "XCTest is off on its lonesome" is because it is not part of the host SDK, but rather it is part of the build. That is to say, if the SDK was swapped out for Android, the XCTest binaries would remain Windows x86_64. I believe that copying XCTest.dll to /Library/Swift/Current/bin is a bad precedent to set. This fine is not meant to be distributed to users. This is part of the development tools. Adding that to PATH during testing is the preferable solution here IMO. The content of /Library/Swift/Current/bin is what is part of the runtime redistributable - that is, the pieces which the user will have to install, and we should strive to keep it as lean as possible.

The duplication in compile and link flags is unnecessary. The reason that you can collapse them into one is that the compile and link operation occurs as a single phase unlike C/C++ where the compile and the link are separate. The Swift compilation model is not easily broken into the compile and link phases since the driver decides whether it needs to recompile source files as it builds, and it seems fragile to duplicate that analysis into build systems when the driver is capable of doing this.

I will be adopting the / as the path separator in the rest of this post. This is valid on Windows, but requires that your current drive is the same as the drive where it is installed. These flags can be used as is on Windows (and Linux and macOS). This is the reason for the layout and the odd choice of location for all the build artifacts.

Breaking down the flags into the components we find:

Compile Flags

-resource-dir /Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk/usr/lib/swift
-I /Library/Developer/Platforms/Windows.platformm/Developer/Library/XCTest-development/usr/lib/swift/windows/x86_64

Link Flags

-L /Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk/usr/lib/swift/windows
-L /Library/Developer/Platforms/Windows.platform/Developer/Library/XCTest-development/usr/lib/swift/windows

Breaking this down further, the second entry for both the compile and link is related to XCTest. Now, the ideal solution to this is to:

  • Improve CMake's XCTest support to permit the Swift XCTest
  • Improve swift-package-manager to do similarly

This would allow CMake projects to elide both those options in favor of:

find_package(XCTest REQUIRED)
xctest_add_bundle(LibraryTests Library
  LibraryTests/Test.swift)
xctest_add_test(XCTest.Library LibraryTests)

This would both setup the PATH when running the tests, as well as handle the include path and the library search path. It would also setup the tests so that you can use the standard test target to run the test suite and get dashboards for the status of the tests.

This effectively leaves the first two flags. This actually relates to Swift Linux layout considerations (aka Linux is difficult, lets go shopping). My vision for this is to add a single variable for CMake (and swift-package-manager) which is CMAKE_Swift_SDK. This would take the path /Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk. At that point, it would actually synthesize the necessary -resource-dir and -I flags that currently require to be added manually. This effectively would then change the CMake invocations to something of the following sort:

cmake -G Ninja^
  -B /BinaryCache/Project^
  -D BUILD_TESTING=YES^
  -D CMAKE_BUILD_TYPE=Debug^
  -D CMAKE_Swift_COMPILER=/Library/Developer/Toolchains/unknown-Asserts-2020.2.xctoolchain/usr/bin/swiftc.exe^
  -D CMAKE_Swift_SDK=/Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk^
  -S /SourceCache/Project
cmake --build /BinaryCache/Project
cmake --test /BinaryCache/Project

Now, I realize that we are not near this yet and perhaps I am letting perfection get in the way of progress. But, I hope that this explains where I have been trying to steer the experience to and why the state is as it stands currently. I feel as though they are the correct pieces from which we can compose the experience that I was describing. Note that the idea is to have both the toolchain and the SDKs be relocatable, which is the reason for the paths needing to be passed to the build tool.

We should definitely update the documentation on the swift-build repository to reflect the required flags to make ti easier for others. I suppose that I never really even thought about it, since I am so used to just building up the command line as I am working.

The fantastic work that you have done will certainly make things better for those that are trying to setup Swift projects and CI for them on GitHub. I do have a few thoughts on the results though.

  1. Why is it not possible to use cmd as the shell? Are the machines that are available with GitHub actions not similar to the Azure ones? Some of the variable handling stuff could be simpler with that.
  2. Is it possible to convert this into smaller steps and package it up into a reusable action?
  3. Why not use the ICU build from Azure? They are the ones that we are building the runtime against anyways.
  4. Why is the llbuild that is being built on Azure insufficient and you need to clone it again?
  5. Would using MSIs make the setup easier?

Lessons Learnt:

  1. We need to start considering doing some additional work to package up the module map files as part of the distribution.
  2. We need to package up ICU as part of the distribution.