I understand that this might work for some non-system frameworks, but that is not what this feature was designed for. Packages intentionally follow a model where dependencies are explicitly declared.
There is a pitch thread for support for binary dependencies in SwiftPM here: [PITCH] Support for binary dependencies