Using SPM build of a package with whatever version of Xcode you have now

Is there a way to point Xcode to use some particular version of SPM without Building whole toolchain but just SPM package? It will be nice to have an ability to define version of SPM package to use instead of complete toolchain, and untangle releases of SPM from toolchain and Xcode.