Binary-only Swift frameworks are expected to use library evolution mode, not because they'll be replaced at runtime like OS-level frameworks, but because there might be other binary-only frameworks that depend on them. (We touched on this briefly when I was still at Apple near the end of the Binary Frameworks in Swift talk at WWDC 2019.) Minimizing dependencies for the end user isn't considered interesting for source packages because you still have to build the source package itself, so this mostly applies to (1) copying around build products for source packages to save on build times, and (2) future improvements to SwiftPM and such wherein package libraries that weren't explicitly exported wouldn't be importable by clients (or something like that). I've also personally run into (3) "my C dependency needs certain macros predefined, and it stinks forcing my clients to define them too". All of these will be great to have, but I don't think they're blocking anything today.
EDIT: the "future improvements to SwiftPM" I poorly described here are mostly "fix the issues mentioned by @mmfl in Swift packages and module dependencies". Definitely worth doing, but also examples of things that work when they shouldn't, which is not quite as bad that things that should work but don't.