Xcode's DX issues are, of course, but many of the underlying behaviors are fundamental SPM issues, like overall performance. For instance, the issues outlined by @pyrtsa in this thread. Notably, neither of the PRs he linked at the end have been merged in any form, so there is still quite a lot of performance left on the table. Improvements to SPM in those areas would immediately benefit Xcode without needing Xcode to do anything.
There are also various functional SPM limitations which prevent it from fully replacing other package managers (or just being used at all), like the inability to manage build configurations, and the complete block on custom build flags, just to name two. And the issue of private dependencies has nothing to do with Xcode at all. So there's really quite a lot we could do here.