"Proxy/Local" binary frameworks are a potent idea that could be applied to many repos. I'm not familiar with SPM's repo but you could probably use an old version of Xcode 13 to generate a .xcodeproj from the Package.swift and it's largely RTFM from there. I singled out swift-syntax as it seemed to be causing huge discomfort in the macro community and I was looking at it recently for a PR. The static linking etc I had to get up to was only because macro plugins are sandboxed. It'd be far easier for other repos.
My take on why SPM coupling comes up specifically is these issues where people are having to have a dependency on the sourcekit-lsp repo (which depends on SPM) when all they are after is the Codeable data model for the LSP protocol so they can write clients. That really should be separated out into its own package.
Edit: I used .xcodeproj path largely because I had experience with it.