CI for Packages on Windows & Android?

Using Swift
, ,
21

Still some clean‐up to do, but I just successfully built a hello world executable in a GitHub action on Windows! :tada:

2 Likes
22

Awesome! That’s great to hear. If you have some time, I think that it would be great to actually document the process so that others can replicate it more easily.

23

Yes, I was planning on posting back when I’m done. At that point it should be possible to either:

  • from macOS or Linux, clone my tool and run one or two commands to automatically generate both a workflow and CMake instructions for any package.
  • from Windows, copy and paste the workflow source and then set up CMake manually.
24

Is it possible to import XCTest?

25

I’m missing something to do with linking.

For reference, on macOS and with imports and target_link_libraries, I get the following unless I twiddle with LD_LIBRARY_PATH:

dyld: Library not loaded: @rpath/libExampleLibrary.dylib
  Referenced from: [/output/directory]/bin/ExampleTests
  Reason: image not found

I can “fix” that on macOS with this:

set_property(TEST ExampleTests PROPERTY ENVIRONMENT "LD_LIBRARY_PATH=${CMAKE_BINARY_DIR}/lib")

I’ve stripped the example down to the point where there are no imports or target_link_libraries so that macOS can run it without LD_LIBRARY_PATH, but Windows still cannot execute it:

    Start 1: ExampleTests
1/1 Test #1: ExampleTests ...................Exit code 0xc0000135
***Exception:   0.01 sec

That exit code is STATUS_DLL_NOT_FOUND and the only thing I can imagine it still needing to load would be the standard library.

But how do I tell Windows where to find it? I tried adding the following to PATH before calling ctest, but it didn’t seem to make any difference:

[...]/Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk/usr/bin
26

Yes, XCTest does work on Windows.

Adjusting PATH to contain that is the right thing. However, you also need all the dependent libraries too (ICU, swiftCore, Foundation, dispatch). In fact, if you look at the CI, you will find that we have to do this adjustment for running the Foundation test suite which uses XCTest. I would recommend looking at the CI configuration in the swift-build repository.

27

The issue had something to do with the setting of PATH itself. I don’t think the executable was picking up the change.

In any case, when I gave up on PATH and just copied all the DLLs (from both the SDK and ICU) into the same directory as the executable, it ran just fine.

28

Interesting; it could simply the quoting - the shell quoting is different, and I know that I've gotten it wrong in the CI configuration previously.

29

That’s a possibility. In the end I think copying the DLLs is probably better anyway though.

  • If the archive layout changes, PATH adjustments will fail silently and lead to head‐scratching, but the copy operations will immediately fail with a helpful description of the problem.
  • If someone happens to use the CI script as a starting point for actually installing or packaging for distribution, then it will be more obvious that those external pieces are necessary.
1 Like
30

Note that the VS2019 builds have had the MSI packaging revived. The windows-runtime-amd64.msi artifact actually contains a MSI for the redistributable component of the runtime (~5MiB) rather than the ~70MiB package for development.

https://dev.azure.com/compnerd/swift-build/_build/results?buildId=24685&view=artifacts&type=publishedArtifacts

1 Like
31

Well, at the end of all my own experimentation, the following is what I landed on, which very closely follows your instructions and .wxs files:

The Windows directory beside it contains the generated CMake manifest and XCTest executable, which together are basically equivalent to $ swift test with SwiftPM.

I already know it works, but are there any parts of it that you think ought not to have been necessary or that would have been better some other way? Since it will be generating the set‐up for others, and serving as an example, I want to be a good citizen.

There were at least two things that weren’t mentioned in your documentation that I had to figure out on my own:

  • XCTest is off on its lonesome in /Library/Developer/Platforms/Windows.platform/Developer/Library/XCTest-development/usr/bin, and it needs to be copied to /Library/Swift/Current/bin in addition to the other SDK binaries.

  • You did mention -DCMAKE_Swift_FLAGS='--resource-dir [...]' here in the thread, but I don’t think it was in the documentation anywhere. There is actually a lot you have to specify each time, since CMake cannot do anything useful without being told each time where many of the pieces are:

    • -DCMAKE_Swift_FLAGS
      • -resource-dir C:\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk\usr\lib\swift
      • -L C:\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk\usr\lib\swift\windows
      • -I C:\Library\Developer\Platforms\Windows.platform\Developer\Library\XCTest-development\usr\lib\swift\windows\x86_64
      • -L C:\Library\Developer\Platforms\Windows.platform\Developer\Library\XCTest-development\usr\lib\swift\windows
    • -DCMAKE_Swift_LINK_FLAGS
      • -resource-dir C:\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk\usr\lib\swift
      • -L C:\Library\Developer\Platforms\Windows.platform\Developer\SDKs\Windows.sdk\usr\lib\swift\windows
      • -I C:\Library\Developer\Platforms\Windows.platform\Developer\Library\XCTest-development\usr\lib\swift\windows\x86_64
      • -L C:\Library\Developer\Platforms\Windows.platform\Developer\Library\XCTest-development\usr\lib\swift\windows

    (It seems odd to me that the compiler needed linker flags and the linker needed compiler flags, so maybe I was doing something wrong?)

(Also, Ninja cannot find any Unicode file names on Windows (:cry:), but that is not directly related to Swift and will be solved automatically when SwiftPM can take its place.)

2 Likes
32

Thanks for exploring this and writing up the findings.

The reason that "XCTest is off on its lonesome" is because it is not part of the host SDK, but rather it is part of the build. That is to say, if the SDK was swapped out for Android, the XCTest binaries would remain Windows x86_64. I believe that copying XCTest.dll to /Library/Swift/Current/bin is a bad precedent to set. This fine is not meant to be distributed to users. This is part of the development tools. Adding that to PATH during testing is the preferable solution here IMO. The content of /Library/Swift/Current/bin is what is part of the runtime redistributable - that is, the pieces which the user will have to install, and we should strive to keep it as lean as possible.

The duplication in compile and link flags is unnecessary. The reason that you can collapse them into one is that the compile and link operation occurs as a single phase unlike C/C++ where the compile and the link are separate. The Swift compilation model is not easily broken into the compile and link phases since the driver decides whether it needs to recompile source files as it builds, and it seems fragile to duplicate that analysis into build systems when the driver is capable of doing this.

I will be adopting the / as the path separator in the rest of this post. This is valid on Windows, but requires that your current drive is the same as the drive where it is installed. These flags can be used as is on Windows (and Linux and macOS). This is the reason for the layout and the odd choice of location for all the build artifacts.

Breaking down the flags into the components we find:

Compile Flags

-resource-dir /Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk/usr/lib/swift
-I /Library/Developer/Platforms/Windows.platformm/Developer/Library/XCTest-development/usr/lib/swift/windows/x86_64

Link Flags

-L /Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk/usr/lib/swift/windows
-L /Library/Developer/Platforms/Windows.platform/Developer/Library/XCTest-development/usr/lib/swift/windows

Breaking this down further, the second entry for both the compile and link is related to XCTest. Now, the ideal solution to this is to:

  • Improve CMake's XCTest support to permit the Swift XCTest
  • Improve swift-package-manager to do similarly

This would allow CMake projects to elide both those options in favor of:

find_package(XCTest REQUIRED)
xctest_add_bundle(LibraryTests Library
  LibraryTests/Test.swift)
xctest_add_test(XCTest.Library LibraryTests)

This would both setup the PATH when running the tests, as well as handle the include path and the library search path. It would also setup the tests so that you can use the standard test target to run the test suite and get dashboards for the status of the tests.

This effectively leaves the first two flags. This actually relates to Swift Linux layout considerations (aka Linux is difficult, lets go shopping). My vision for this is to add a single variable for CMake (and swift-package-manager) which is CMAKE_Swift_SDK. This would take the path /Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk. At that point, it would actually synthesize the necessary -resource-dir and -I flags that currently require to be added manually. This effectively would then change the CMake invocations to something of the following sort:

cmake -G Ninja^
  -B /BinaryCache/Project^
  -D BUILD_TESTING=YES^
  -D CMAKE_BUILD_TYPE=Debug^
  -D CMAKE_Swift_COMPILER=/Library/Developer/Toolchains/unknown-Asserts-2020.2.xctoolchain/usr/bin/swiftc.exe^
  -D CMAKE_Swift_SDK=/Library/Developer/Platforms/Windows.platform/Developer/SDKs/Windows.sdk^
  -S /SourceCache/Project
cmake --build /BinaryCache/Project
cmake --test /BinaryCache/Project

Now, I realize that we are not near this yet and perhaps I am letting perfection get in the way of progress. But, I hope that this explains where I have been trying to steer the experience to and why the state is as it stands currently. I feel as though they are the correct pieces from which we can compose the experience that I was describing. Note that the idea is to have both the toolchain and the SDKs be relocatable, which is the reason for the paths needing to be passed to the build tool.

We should definitely update the documentation on the swift-build repository to reflect the required flags to make ti easier for others. I suppose that I never really even thought about it, since I am so used to just building up the command line as I am working.

The fantastic work that you have done will certainly make things better for those that are trying to setup Swift projects and CI for them on GitHub. I do have a few thoughts on the results though.

  1. Why is it not possible to use cmd as the shell? Are the machines that are available with GitHub actions not similar to the Azure ones? Some of the variable handling stuff could be simpler with that.
  2. Is it possible to convert this into smaller steps and package it up into a reusable action?
  3. Why not use the ICU build from Azure? They are the ones that we are building the runtime against anyways.
  4. Why is the llbuild that is being built on Azure insufficient and you need to clone it again?
  5. Would using MSIs make the setup easier?

Lessons Learnt:

  1. We need to start considering doing some additional work to package up the module map files as part of the distribution.
  2. We need to package up ICU as part of the distribution.
33

Makes sense. I will switch it. I never really had the distribution side of things on my mind. I was only trying to get tests to work, so I mimicked the process for the other core libraries without thinking.

That may just be my own ineptness with the native shell. My first attempts were all with PowerShell, which is GitHub’s default for a Windows job. But I eventually gave up and switched to the more familiar bash. I think the main issues had to do with exports not propagating to parent processes, but I may just have been bungling the differences in quoting, variable substitution and path notation. Perhaps I discounted something as nonviable that would have worked but for a typo I didn’t notice or understand. If someone were to submit a working native translation, I would be happy to switch over.

Yes and no. Pieces of it probably could, but not the whole thing:

  1. Environments are lost from one action to another—and even from one run entry to the next—, so anything that needs the Visual Studio environment needs to be in the same script.
  2. Some sections need to be generated according to the details of the particular package, such as the cloning of dependencies. That generation needs to link against SwiftPM’s package library, which cannot run on Windows within the action. And passing a loaded manifest as an argument to the action is more complication than it is worth.

Ultimately, if someone needs to work around something, they can manually adjust a script after generation, but an action would just be broken until the next release. So I at least want to wait until the set‐up is more battle tested.

Well, I was following your instructions :wink:. I did look into trying to get it from Azure, but its Web UI seems broken right now. It just loads indefinitely. (This is where I’m looking.)

That is only in there because it happens to be in that particular package’s dependency graph. It is unrelated to the general set‐up.

I don’t know, maybe? Although if stuff doesn’t land in a standard path, it might suffer from the same issues with the environment not persisting. I wouldn’t really know until I tried it, and I suspect other things are more worthy of your time. By the way, congratulations on your acceptance to the core team!

34

Working on Android now...

Is the following supposed to work, or am I doing it all wrong?

{
  "version": 1,
  "sdk": "/Library/Developer/Platforms/Android.platform/Developer/SDKs/Android.sdk",
  "toolchain-bin-dir": "/Library/Developer/Toolchains/unknown-Asserts-development.xctoolchain/usr/bin",
  "target": "aarch64-unknown-linux-android",
  "dynamic-library-extension": "so",
  "extra-cc-flags": [],
  "extra-swiftc-flags": [],
  "extra-cpp-flags": []
}

This is the error message:

/Library/Developer/Platforms/Android.platform/Developer/SDKs/Android.sdk/usr/lib/swift/android/aarch64/Swift.swiftinterface:2934:1: error: unknown attribute '_implicitly_synthesizes_nested_requirement'
207
@_implicitly_synthesizes_nested_requirement("CodingKeys") public protocol Encodable {
208
^

That is using the compnerd/swift docker container (on Ubuntu), and then installing the official 5.1.3 toolchain separately in order to use its SwiftPM.

5.2‐dev and master error too, but are less specific:

:0: error: unable to load standard library for target 'aarch64-unknown-linux-android'

@Geordie_J, you seem to be an expert on using Android from the Azure builds. Do you have any advice?

35

I can also now confirm that to be the case, and I have removed the entire redundant LINK set.

To any readers who actually want to use it, the most up‐to‐date version will always be here. You can copy and paste the workflow and adjust it to point at your own CMake files, or you can install the tool on a UNIX platform and from there generate the whole thing, including the CMake stuff:

$ workspace refresh continuous‐integration
36

Hi there! I am using the toolchain with CMake, for examples see the links above. I can’t help you with any other setup but I can give you my insight why:

I couldn’t get SPM working on Mac with any cross-compiling toolchain. The issue was that Mac SPM had some hard coded paths to the internal Mac-only toolchain that ignored any toolchain bin dir etc you gave it, because reasons. I can feel my blood pressure rising just thinking about it. Over the last 3 years I’ve spent months in the deepest imaginable frustration trying to deal with SPM’s idiosyncrasies and shortcomings here.

We had a somewhat non-hacky setup working with SPM 4.x for Android, although that involved manually adding a whole bunch of include paths and module maps to its command line invocation, because SPM itself wouldn’t. That evidently broke at some point on the transition to 5.x because of the issue above. While I’m sure making a patch to fix that issue wouldn’t be too difficult, I am certain that one further issue after another would come up if we did and frankly I’ve had enough of that stress.

With CMake you just tell it when you want built, adding subdirectories if needed for various library deps, and you’re done. the CMake implementation does have its own bugs regarding static linking right now, but they didn’t stop us putting our app into production within a week of moving over

/rant

2 Likes
37

Interesting.

Since posting my question, I have managed to get a simple “Hello, World!” to cross‐compile with a Swift 5.1 manifest using the indirect toolchain from flowkey. That is the one you are associated with, right? You mentioned in another thread that it is somehow derived from @compnerd’s Azure artifacts. Are you (or @michaelknoch) able to tell me what adjustments you make as you package it up? I am trying to figure out why one works and the other doesn’t. Also, when I download that toolchain, which official versions or snapshots does it roughly correspond to? How often do you update the download?

I am already invested in SwiftPM and have a lot of tooling built on it, so I would rather not use CMake where I don’t have to. (The fact that the underlying Ninja doesn’t care about Unicode is a real turn‐off.) However, I have made contributions to SwiftPM itself, and its lead developers have come to recognize me. I stepped in only just yesterday to catch a PR that almost removed the toolchain from --destination altogether, because no one knew what it was for. If I can acquire a good understanding of cross‐compilation, I have a reasonable chance of improving SwiftPM’s stability going forward.

1 Like
38

Great! Happy to hear you got it working.

Yes our toolchain is just repackaging artifacts from @compnerd’s CI. As far as I know we’re not really changing anything. The main thing is we get a toolchain and the Android SDK (ie stdlib, Foundation etc) that have been built from roughly the same commits.

We’re not pinned to any particular version because there are no official releases for Android, and because work is constantly going in to improve the Android toolchain’s reliability. Once that slows down it would make sense to pin Android releases to releases on other platforms. Since Swift is so source-compatible we haven’t seen this to be a problem in practice. To answer your question more directly those binaries were built a few weeks ago from master, so it’s somewhere between Swift 5.1 and 5.2.

@michaelknoch might have something to add about what else we’ve changed. From what I know it’s just a repackaging as mentioned above + some bits and pieces to make working with Android studio on Mac + Linux easier, with some docs.

Good luck and let me know if I can help (with anything but SwiftPM :slightly_smiling_face:)

39

Improving cross-compilation support is definitely one of SwiftPM's goal. @SDGGiesbrecht it would be great if you want to drive that effort! Feel free to make PRs or start conversations in the package manager category on how we can start improving the support.

2 Likes
40

By “toolchain”, do you mean the toolchain-linux-x64 artifact of the Ubuntu 18.04 (FlowKey) pipeline? Which pipeline does the Android SDK originate from? I see what looks like a conversion of Windows paths in setup.sh. Since the Azure page won’t load anymore, I’m stuck blindly requesting artifacts that may or may not actually exist until I get lucky and end up with something.