This is a great idea: do we have any more detail on it? I can see some issues with integrating with SwiftPM, most notably that as long as we rely on git clone to distribute source-available libraries then "distribute" is not really a discrete step. You end up maintaining parallel branches, one for each Swift release you support, which you'll presumably have to automate cross-merges to. I'm not sure this idea is totally straightforward.
While I'm here I'll also say that while this idea is a good one, that tool does not exist yet, so in the meantime the pain remains.