(Partial) Nightlies for android SDK

Development Core Libraries
, , ,
1

Hello android developers,

I often hear that android users tend to have a hard time with getting started with using swift on android. Part of this stems from people not wanting to build the full toolchain, part of it stems from the complexity of the builds, part of it stems from the android port not being kept as well as Windows (or Linux), although @drodriguez has been doing a bunch of work to keep android healthy as of late.

To that end, I've been working a bit on getting some more immediate term signal on the state of the android SDK. Note, that this is purely the SDK and runtime components for the android development and not the toolchain running on android.

I've been working on getting nightlies (well, these can be more frequent if desired) of the android SDK - the Swift standard library, libdispatch (and swift SDK overlay), as well as libxml2 and curl. Foundation is something which will take a bit more work, but is easily added (the supporting infrastructure is in place). There are some preliminary builds now at Azure DevOps Services | Sign In . These are not exactly perfect (would be nice to remove the libxml2 documentation, etc), but should get you fairly up-to-date builds of the Swift standard library and libdispatch for the moment. Foundation builds uncovered a bunch of additional work which needs to be done to repair the Foundation build on android.

All of these builds are being done on Windows, but the generated artifacts are consumable on other targets as well (i.e. you can use this with an up-to-date toolchain on Windows, Linux, or Darwin). They currently target android API level 21, though if anyone has a strong reason to support something lower, we could adjust that downwards as needed.

There is plenty of work to be done beyond just cleaning up the android build of Foundation. The CI rules could be cleaned up and simplified to support the multi-target builds (like in ICU). This work is using a custom built ICU from azure which is built for android armv7, aarch64 and windows x66, arm64. Getting the data generation would mean that everything is fully contained and available. Cleanup of the CI build rules would enable the builds of the SDK for ARMv7 and x86_64 (I know that @v.gorlov is interested in that and has been doing some where there as well). I would be happy to have some help with this.

CC: @tkremenek @millenomi @Michael_Gottesman

25 Likes
2

Thank you!

I am also trying to make Swift CI configuration to support fully automated macOS builds for Android armv7a, aarch64, x86, x86_64 architectures.

In fact it is already exists, but not yet official (without tests). There is approximately 8 pull requests left with patches for Swift, Dispatch and Foundation. After applying these patches there will be possible to adjust existing Shell and Python scripts and enable Android macOS build job on Swift CI server with tests like it done for Linux.

4 Likes
3

You guys are real wizards. :man_mage: Looking forward to see some fixes for Raspberry PI side of the project someday.

2 Likes
4

That sounds pretty cool. Are you working on build-script or are you using the YAML configuration which Azure/CircleCI/etc use these days?

The interesting thing is that exploring the cross compilation on windows seems to indicate that there may have been some leaks of the build into the host. I put up a number of patches for foundation that have the potential to break the android build due to uncovering latent bugs. But, I think that the android target should be better for that. There definitely were points where there was no way that Foundation couldā€™ve been building for android :confused: Iā€™m looking forward to what you have found.

5

@DevAndArtist hmm, is there a SDK (sysroot) for that in Visual Studio? I know that the previous versions supported ā€œLinuxā€ (this is prior to modern Linux support which supports desktop as well) and Windows IOT which covered rPi. If so, it might be possible to setup a job that runs for rPi as well.

Really, itā€™s a matter of getting machines to run custom images as the downloads tend to be flaky, which is why it can be a challenge. But, the SDK builds are relatively light - they will take about 40m or so, so with a bit of help I think that it should be possible to enable those as well.

6

I donā€˜t think I can answer that question, but maybe @Helge_Hess1 can or at least he might know whom we could ping regarding Swift on rPI. I personally have never seen the rPI badge showing anything else than failing.

7

I have a plan to start updating build-script and Python scripts after achieving patch-less cross-compile build (and passing tests) of Swift and core libraries.

The Foundation and Dispatch patches I am using on macOS can be found here: swift-everywhere-toolchain/Patches at master Ā· vgorloff/swift-everywhere-toolchain Ā· GitHub

Note, that patches needs maintenance after massive merges (like one few days ago :0 ). Some Foundation patches are rather workaround. I can apply proper fix later, after achieving stable builds and tests of Swift and core libraries.

8

too much text to follow up, but Swift on RPi works very well nowadays. we got cross compilers, remote debuggers and last wwdc someone even got xcode debugging working.
there is a swift-arm slack if anyone wants to join.

3 Likes
9

@v.gorlov - I think that it may be valuable to actually unite on this instead:

The only thing missing there is the generation of the full data file. That allows an easy cross-compilation to all the targets and is how I am building ICU for android armv7 & arm64 as well as Windows x64 & arm64. Honestly, the biggest hurdle to adding more targets is really figuring out how to extract the steps into a template so that we aren't duplicating the CI configuration to make it easier to adjust them all at once.

I'm now setting up XML2 and CURL builds for android armv7 & arm64 as well as Windows x64 & arm64. These three are re-used in the rest of the pipelines to build the SDKs.

Looking at your changes for libdispatch, I think that swift-build/android-aarch64-swift-flags.cmake at master Ā· compnerd/swift-build Ā· GitHub is not particularly elegant, but is sufficient for the time being. I think that this requires a bit more experimentation with build: port to new Swift support by compnerd Ā· Pull Request #488 Ā· apple/swift-corelibs-libdispatch Ā· GitHub . I would like that to JustWork (TM) for the cross-compilation.

Looking at the changes for CoreFounation, I think that I hit the same set of issues and have some patches up which should address those. The one problem that remains that you clearly realized is a hack is the CFString ABI change. I think that if we can get the ability to override the compiler with the Android NDK in CMake, that would be the ideal solution.

Looking at the changes for Foundation, seems that most of those have already been identified and are being addressed.

Seems like we should be pretty close to having this all working properly :-)

1 Like
10

Super! Having Cmake setup for ICU, xml, ssl and curl would be ideal .)

At the moment I still not fully understand when we have to use clang from NDK, system-supplied (comes with Xcode) and when to use one which we manually built :0

11

In my ideal world:

  • system provided clang for building the toolchain
  • our clang for the SDK

Currently, the setup is that you need the just built clang for just Foundation and the standard library as libdispatch does not exercise much in terms of recent clang changes and XCTest is fully Swift.

This requires one additional piece to be further developed:
-DCMAKE_SYSTEM_NAME=Android -DCMAKE_NDK_PATH=ā€¦ to permit -DCMAKE_C_COMPILER= and -DCMAKE_CXX_COMPILER to override the version that is part of the NDK.

I am working on some clean up, and hope to work on adding lld to the toolchain that I am building. At that point, the only thing that the NDK will be needed for is the sysroot as we will have the complete toolchain.

I am going one step further than just CMake setup for the dependencies (well, except for SSL - I'm putting that off for now, hopefully someone else will be willing to help set that up). I am setting up CI which will allow us to rev and re-use the builds of the dependent packages. The full CI setup would be able to deliver a MSI that contains a full [Swift] SDK that can be easily used with a preexisting NDK.

1 Like
12

How you going to deal with Android-specific dependencies on CI?

Will you checkout git repos for ICU, xml, curl, ssl using swift/utils/update-checkout?

Will NDK be configured manually or it will be downloaded every build on CI?

13

I'm not using update-checkout nor build-script. Everything is automated to just use straight git and cmake. The NDK is provided by Visual Studio. I am building mostly from the official upstream repositories (libxml2 is the one exception where I have a fork pending a MR in GNOME's GitLab).

What android specific dependencies are you thinking of? Everything needed for the build is just the NDK.

14

Under "android specific dependencies" I mean only ICU, xml, ssl, curl.

15

Those are handled exactly the same as on Windows: local builds that are being cached. I don't see a strong reason to bump those more than once a day - though, it is possible to trigger them more frequently if needed for some reason.

16

Iā€™m so stoked about this! Canā€™t praise the work you are doing enough!

Because you mention it: it would be fantastic (albeit with diminishing returns) to get API 19 support. We have a build of the Swift 2 compiler somewhere that supports API 16 with some stubs, so itā€™s not impossible. And, incredibly, we still have about 500 regular users running that code in production.

I do remember there being some licensing issues about upstreaming some of the missing libbionic functions though (BSD licence was compatible but Apple didnā€™t want to dual-licence the project, which makes sense), so Iā€™m not sure how to go about that.

1 Like
17

Thanks!

Right now, I am targeting API 21 (Lollipop). Once we have everything building in CI, we could look at reducing the API level.

Just to make sure that others understand what this really accounts to: API Level 19 (KitKat) will get an additional 6.9% of the total android deployment. Dropping to API Level 16 (Jellybean) will get an additional 3.2% deployment (10.1% additional from the current level). API Level 15 (Ice Cream Sandwich) support was removed from the NDK in r18b (September 2018). The current minimum NDK requirement (as I just learnt) is r16b. However, as of August 2019, all android images will be required have a ARM64 slice. At that point, you may need to update the NDK anyways.

Well, upstreaming bionic functionality is less helpful as that bionic has shipped and to back deploy to that version you need to provide the implementation anyways. So, I think that may be less valuable. Although, we should continue to file requests against them to improve bionic so that we can remove the workarounds in the future version.

18

An update for those following the thread: we now have ARM64 target builds active on Azure. You can download the binaries from the artifacts at the latest build at Azure DevOps Services | Sign In

It currently says that the build failed but that is because I am working on getting the ARMv7 builds enabled as well as part of that build. I hope that once those are working, we should have something building this nightly if not more frequently.

1 Like
19

Itā€™s worth mentioning (for anyone following along) that an additional 6.9% doesnā€™t sound like much, but it amounts to support for about 200,000,000 more devices.

Re: libbionic- with ā€œupstreamingā€ I was referring to adding the code of the functions missing from Bionic in API Level 16 to the Swift repo (basic things like like log2). I used the wrong term for it but I canā€™t think of a better one. Just so weā€™re on the same page :slight_smile:

1 Like
20

Hmmm, cool, but will there also be a macOS -> Android ARM64 build? :thinking::thinking::thinking: