Linux or macOS?
If you are iterating with small source changes and then want to run the tests on a different platform architecture, I would think pushing small source diffs to that central git would be faster than pushing a cross-compiled test binary. If you simply want to make sure the source still builds okay for the other arch, then local builds may be faster.
It has been simple for awhile now, as I laid out late last year. 1. And 3. there are very easy to download from swift.org, with only 2. depending on the distro and your level of expertise with it.
Then, you simply configure all those together with a SwiftPM target destination config file, as I showed with my Android example.
Alternately, you could try the new Swift SDK generator, which currently generates linux SDK bundles from macOS, that you could then use on linux also for cross-arch compilation.
@Max_Desiatov or @ktoso, are there any plans to start distributing official linux SDK bundles on swift.org, for @taylorswift's use case of building on desktop macOS/linux and deploying to linux servers? I think this would help those using Swift on the server.