The current state and future of Android testing on the CI

Platform Android
,
1

As the CI contact on the Android workgroup, I thought I'd lay out where we are at with bringing up Android on the official CI and where we are going, so that anyone interested knows the plan and can pitch in.

Currently, we have a linux CI job that builds the Swift compiler and cross-compiles the Android SDK bundle, running most of the compiler validation suite on each of the three architectures we ship: AArch64, x86_64, and 32-bit armv7 (this is one of only two public OSS Swift CI that runs most of the Swift compiler validation suite for a 32-bit arch). The Windows trunk snapshot toolchain CI also spends a significant amount of time cross-compiling and running most of the compiler validation suite for the Android SDKs on every run, in addition to its native Windows SDKs (this is the other CI that runs the 32-bit compiler validation suite, for Android armv7 and i686).

I say "most of" because the 5-7% of tests that must be run as executables on the cross-compilation target platform from the compiler validation suite obviously cannot be run directly on linux or Windows and must be run in the Android emulator, but the official Swift CI runners don't seem to support nested virtualization to run the Android emulator well, as we've found before that running in full emulation for native code is way too slow. I have a request out to the CI team to see if any of the runners already have nested virt or if it can be enabled. We will run the remaining 5-7% of "executable" tests once that is set up.

The official CI is running well for the trunk 6.4 branch and the 6.3 release branch, generating the Android SDK snapshots you can download and use. I have been running the full "executable" compiler validation suite and the corelibs tests natively on my Android phone in the Termux app for the last 7 years on a monthly basis, by building the full Swift toolchain natively on Android, which shows less than 1% of tests failing or disabled. Once the Android emulator can be run, we plan to get those remaining executable and corelibs tests running on the official Android CI too.

Another step is to get mandatory pull request testing- must be run, but doesn't have to pass yet- going on the compiler and other relevant repos, as eg swift-testing and swift-foundation have set up to cross-compile their non-test code with the latest Android SDK snapshots.

We are looking for volunteer contributors to help with any of this, whether keeping the Android CI green or bringing more of this testing online, similar to the great work @gabriele has been doing to get Swift debugging working with the Android SDK and target runtime. This would be a good way to learn how Swift tests its latest commits on the official CI, which would be useful for any other toolchain work you do going forward, or to learn more about how Swift works on Android.

Please DM me or the @android-workgroup if you are interested, looking forward to getting much more of the Swift test suite set up for Android.

3 Likes
2

I neglected to mention that in addition to this SDK CI that we set up and are expanding, @marcprux has added Android cross-compilation and testing in the Android emulator to the official Github Actions workflows for swiftlang, which can be used to easily cross-compile and test your Swift packages for Android. That is what the swift-testing and swift-foundation workflows I linked above are using to cross-compile each of their commits.

Adding Android testing to your Swift package on GitHub is as simple as adding these 7-8 lines I submitted for Swift Configuration, enabling the official swiftlang testing workflow for Android while disabling the default-enabled linux and Windows builds.

Another way to contribute to this Android effort is by enabling Android CI on more Swift packages, such as my recent pulls for Numerics and Algorithms.

Of course, not all packages will cross-compile to Android right away, so read Marc's tutorial on common changes that must be made. We hope to get those up on swift.org soon, including for other platforms like Wasm and Windows.

3

Out of curiosity, how does the run time of the Android CI compare to the run times of other CI jobs used by the toolchain, such as macOS/Linux/Windows smoke tests? One of my biggest pain points for making compiler PRs is the slowness of those jobs (with Windows being the worst bottleneck, and flakiness makes it worse), so I'm hoping that Android is closer to macOS/Linux than Windows in that regard.

4

Android is by far the fastest, as you can see on this pull I recently manually ran the Android CI on, by commenting @swift-ci test android.

It is rare for the Android CI job to take more than 3 hours, generally clocking in just under, whereas the other required smoke test jobs generally take 4-5 hours, more if running the full validation testing with @swift-ci test.

It's not a fair comparison though, as we only cross-compile the stdlib and corelibs for Android, albeit for all three architectures, whereas those other platform CI must also build most of the toolchain, including SwiftPM, which has become a huge chunk of the build time on CI jobs lately.

1 Like
5

It's still sort of fair in the sense that I would be concerned if the Android job did take as long or longer than the other jobs that were building a full toolchain! But those numbers are right where I'd expect them to be.

1 Like