Hello @jml5qh
So, I came across the same issue and it's partially solved on my side.
1- The first thing to do, is to make sure Xcode's git has access to git-lfs. Which, on a local machine require the symlink command unless you specified which git to use...somehow I couldn't find that option in Xcode anymore, I always used Xcode's git anyways. So I needed the symlink.
2- It seemed to work for a while, I had the impression Xcode's Swift Package Manager was pulling git-lfs files normally until I faced your issue when I tried manually in the DerivedData checkouts directory, to perform git lfs pull. And in fact, the issue seemed to be authentication with the lfs server. I updated my git repo to add a .lfsconfig files that adds the url and then git lfs pull worked properly.
3- In the end, doing so manually afterwards allowed me to work with the Swift Package properly in Xcode.
So now I am trying to figure out how to have that done automatically when SPM clones the said package. Or at last I'll try to get a script-step that knows where to perform the git-lfs pull on my CI