Looking into this: the problem seems to be because SPM runs a git clone -c core.symlinks=true --mirror which doesn't pull in LFS references.
If you add a git lfs fetch --all after creating the mirror, that pulls in the LFS references, and then the rest of the chain works. Unsure what the best way to orchestrate this would be or if it's in the roadmap, I'm not sure if there is a way to do this such that the lfs fetch only happens for repos that need it. The issue is with the "--mirror" parameter, from what I can tell.