Exactly this. Thanks to ABI stability and module stability, source stability is no longer as important in the teams I've worked with (codebases big and small). Especially, with support for binaries in SwiftPM, some of the most critical dependencies no longer require us to recompile everything from scratch with every new Swift version and to complain about breakages. That dependency can be stored as a binary as long as possible, while every other module (even when not ABI-stable) can evolve at their own pace in terms of source compatibility.
Even on the platforms that don't have ABI stability (basically every non-Apple platform at this point), the time to make Swift clean and consistent is now, before those platforms get more significant adoption. At a later time changes would be more painful. And again, ability to specify Swift version per package in package manifests means it's not as painful, just a minor annoyance if anything in most cases.