I'm sorry, but this doesn't really seem like an acceptable change for a minor release. Not only do you vastly underestimate the ease with which users can switch from CocoaPods to SPM, there are also specific use cases where SPM just doesn't work. Not to mention the sub-par Xcode UX and frequent bugs. And in general, refusing security updates mid release cycle just seems bad for the community.
This just isn't true. I can't think of any library which doesn't support CocoaPods that did it based off some statistic. Instead, it's usually just a matter of policy, like the various swift-x libraries refusal to support CocoaPods from day one. Additionally, since many of these frameworks are foundational, it essentially forces any library which adopts them not to support CocoaPods, whether they want to or not. So the fact that some libraries only support SPM isn't evidence of anything by itself. It was only in the last two years that SPM surpassed CocoaPods for personal usage and CocoaPods still leads for corporate usage (likely due to SPM's missing features and corporate inertia).
So obviously I think this is a bad idea. However, if you insist on dropping support, can you at least archive all of the podspecs into a single place so people can fork and maintain them on their own?