If I am understanding you correctly, I think your suggestion is to re-use the cache from one build of the final project to the next?
The main problem with that is that SwiftPM's cache management isn't completely reliable (see also this thread on server-side pain points); I think we've all experienced having to occasionally delete the build folder in order to resolve a build problem. We're reluctant to introduce that element of unpredictability to our CI pipelines.
Since you mentioned using this technique, have you run into this sort of problem? If so, how did you address it?