There are now Swift 5.4 cross-compilation SDKs for AArch64, armv7, and x86_64 available, which you can use with a prebuilt Swift toolchain to cross-compile packages like swift-crypto or swift-system for Android with the Swift package manager.
Instructions on building and testing those packages on an Android device or emulator are in the README. Building with these SDKs is primarily tested on linux, but using them on Mac or Windows should be possible too.
I have set up a daily CI on github Actions that builds latest trunk of five Swift packages for all three architectures, then runs their tests for Android x86_64 in an emulator.
I plan to set up the same for the 5.5 branch snapshots and trunk, which will build the latest Swift SDK from source before testing it. I will also add some Swift scripts to automate the configuration of the SDK and building it.
Any suggestions of Swift packages to add to this Android CI are welcome, and you could use the CI script as an example to build or test packages yourself. Right now, the largest package tested on the CI is Swift NIO, which is about 60k lines of Swift code, about half tests.
I have also put out a Swift 5.4 toolchain that runs on Android in the Termux app, which can be used on Chromebooks that support running Android apps too.