Still happening as of Xcode 14.3 when you depend on branch or tag using git URL.
This has nothing to do with the ssh keys. It is simply a bug in either Xcode or SPM since adding the package works initially, and it also works after Reset Package Cache is invoked from Xcode.