One thing I'm missing from the proposal is more elaboration on how dynamic binaries (dylib/so) are going to be handled. AFAIK, SPM does not output bundles of binaries to be distributed into environments that do not build the binaries (e.g. uploading a binary to be run into a Linux server).
I can see swift run working since the dylib and binary can be placed into the same .build location directory, but the moment you need to move the binary to another location, it becomes non-trivial with dynamic dependencies. It'd be interesting to see SPM set/conform to a standard for bundled distribution that works well on Linux and other platforms (does such a standard already exist in Linux? I'm not aware of one)
On the Apple side, iOS will be fine since Xcode will take care of the bundling for you, and with XCFrameworks it will only bundle the required architectures (as @harlanhaskins mentions), but if you want to build macOS Swift CLI tools outside of Xcode, the same concerns as Linux apply.
If the binary dependencies are static only, most of these concerns go away, since the code would be linked into the binaries themselves, but since the proposal mentions dynamic as well, I think this needs to be further elaborated.
EDIT: Just found out about the bundle support thread which might alleviate some of these concerns.