Not a pipe dream, but a reality! 
You can already target Darwin/PowerPC today, see x.com (and many other systems, see x.com), by cross-compiling to WebAssembly, and then further compiling to native code, via e.g. w2c2, or wasm2c.
This requires no changes to the Swift compiler or LLVM.
This approach comes with support for WASI (essentially a POSIX-like/subset API), but you can also import and call "host" C functions. I'm currently working on GitHub - turbolent/W2C2BridgeGenerator: Generate bindings for w2c2 from a bridgesupport file, a tool that automatically generates bindings for C and Objective-C code, so it's possible to use all Mac OS X frameworks, e.g. AppKit (via GitHub - turbolent/BridgeSupport-Tiger: Modern BridgeSupport (LLVM/Clang-based), adjusted for Mac OS X Tiger (10.4) and Leopard (10.5)).
Even small Swift binaries are quite large, as they include the whole Standard Library. Those binaries can be translated to C very quickly with w2c2, but the final compilation with gcc takes quite a while, especially on era-appropriate hardware.
The new Swift Embedded feature greatly improves the size and performance of WebAssembly binaries, see Some feedback from my short experience with SwiftWasm - #5 by Max_Desiatov. However, this feature is still experimental and seems to have some bugs (Bug in Embedded/ARC/WebAssembly combination?).