fwiw, I have begun to support the Android platform due to your wonderful Swift Android SDK, so things are moving in that direction with this initial commit on that effort.
One thing I'm finding exceedingly more annoying are the preprocessors inside the SwiftPM package, that are dependent on the Host platform doing the compiling (so its fully impossible to exclude specific source on Android in SwiftPM, have to resort to modifying the C++ itself with #if !defined(__ANDROID__) gunk), so it would probably be much easier to do this from a Linux machine; however, I have been doing the cross-compilation for Android from macOS because, well, that's just more fun .
Not sure what you mean: you want to exclude certain C++ files from being cross-compiled to Android, but SwiftPM is not allowing this to be configured in the package manifest? You could try looking for or filing an issue for that with the SwiftPM devs.