Thanks for this pitch, it seems like an interesting idea! I do have some concerns about the pitch as it relates to non-Apple platforms, though.
Firstly, and I appreciate that this is a problem SwiftPM already has: Linux is not a platform that one can meaningfully target. Linux is not a homogeneous collection of environments that differ only in GUI toolkits, but a number of wildly different environments that share a common kernel. The only code that truly targets Linux is kernel extensions or anything making raw syscalls (libc, Go). All other code actually targets a complex matrix of dependency versions.
We can get away with this vagueness about the "Linux" platform when distributing source, as we can resolve the dependencies using compile time macros and package managers, but when distributing binaries this is no longer true. This is because, unlike Apple's platforms and Windows, there is simply no such thing as "the Linux ABI" (again, excluding making literal raw syscalls). A given Linux distribution may define a stable ABI within a single release (e.g. Ubuntu 18.04), but even there it is rare for that ABI to be stable when moving forward to a new release: only when staying within that specific release is there any hope that the ABI of the system will remain stable.
Additionally, these ABIs are not consistent with one another. As a general rule, binaries compiled on a single Linux distribution are not portable to another, as the set of libraries required for the binary to run likely were compiled with different options, and often have a non-overlapping set of versions available.
As an additional note, not only is "Linux" not an appropriate platform specifier, but "ARM" is not an appropriate architecture specifier either. In fact, neither is "x86"! At the very least we should note that Swift does support 32-bit environments, and so at the very least we need a 32bit/64bit split (there are no fat binaries on Linux). But even that isn't sufficient: see Swift Linux layout considerations (aka Linux is difficult, lets go shopping) for a more comprehensive list of what it means to have an actual description of an ARM ABI.
For the moment, then, I would suggest that we forbid distributing binary packages on Linux until we come up with a more holistic solution for how we plan to do that. I've spun off a separate thread for that discussion: Binary dependencies on Linux.