Are there any known issues with using Git LFS with SwiftPM in Xcode? I did some testing and found the following:
I first tried tracking a very small file in a Swift Package via Git LFS. Xcode successfully adds this package as a dependency in a new Xcode project. However, the actual file isn't downloaded / included. When I clone the repo via the command line, the file is downloaded correctly. I created a test repo with a very small file tracked via Git LFS: GitHub - jml5qh/swiftpm-lfs.
I also tried tracking a very large file in a Swift Package via Git LFS. I am unable to add this package as a dependency in another Xcode project and get this error:
Since Xcode uses its own tools to handle swifft packages, git-lfs might not be picked up if it is installed via Homebrew. Run the following command to create a symbolic link if Xcode fails to fetch the package:
Thanks for sharing this repo! Unfortunately, still running into the same issue. I can add the package to Xcode, but the actual asset tracked with git lfs isn't downloaded. For FluentDarkModeKit, it doesn't actually matter since the png tracked by git lfs is just for the README and isn't used in the real package.
So, I came across the same issue and it's partially solved on my side.
1- The first thing to do, is to make sure Xcode's git has access to git-lfs. Which, on a local machine require the symlink command unless you specified which git to use...somehow I couldn't find that option in Xcode anymore, I always used Xcode's git anyways. So I needed the symlink.
2- It seemed to work for a while, I had the impression Xcode's Swift Package Manager was pulling git-lfs files normally until I faced your issue when I tried manually in the DerivedData checkouts directory, to perform git lfs pull. And in fact, the issue seemed to be authentication with the lfs server. I updated my git repo to add a .lfsconfig files that adds the url and then git lfs pull worked properly.
3- In the end, doing so manually afterwards allowed me to work with the Swift Package properly in Xcode.
So now I am trying to figure out how to have that done automatically when SPM clones the said package. Or at last I'll try to get a script-step that knows where to perform the git-lfs pull on my CI
Note: I'm far from a git / SwiftPM expert so I may have some of these items incorrect, but I think I figured out the issue.
It looks like SwiftPM uses a local mirror when cloning / checking out the git repo. So there will be a mirror in DerivedData/[Project]/SourcePackages/repositories. When resolving a branch / tag, SwiftPM will checkout from that local mirror into DerivedData/[Project]/SourcePackages/checkouts which is where the actual package files are stored. However, that local mirror doesn't actually have the lfs objects (sounds like this is expected when working with mirrors). This means git isn't able to resolve the files from git lfs in your checkout directory. It does look like there is a possible solution - SwiftPM could call git lfs fetch --all origin [revision] which will grab / store the lfs files into the repositories folder (info taken from How to properly mirror remote with LFS · Issue #1338 · git-lfs/git-lfs · GitHub)
Looking into this: the problem seems to be because SPM runs a git clone -c core.symlinks=true --mirror which doesn't pull in LFS references.
If you add a git lfs fetch --all after creating the mirror, that pulls in the LFS references, and then the rest of the chain works. Unsure what the best way to orchestrate this would be or if it's in the roadmap, I'm not sure if there is a way to do this such that the lfs fetch only happens for repos that need it. The issue is with the "--mirror" parameter, from what I can tell.
Here is what happens when SPM tries to download a git repo with LFS configured.
SPM tries to clone the repository with a local mirror(DerivedData/Runner/SourcePackages/repositories/[Project]-hash) and checks out into a local folder(DerivedData/Runner/SourcePackages/checkouts/[Project]). This Mirror folder acts a remote git server located in Local/LAN. You can verify this by running git config -l | grep remote.origin.url in /checkouts/[Project] folder.
The /repositories/[Project]-hash folder contains files related to git file system and /checkouts/[Project] folder contains all the package files.
During the checkout process GIT LFS Pull(fetch + checkout) is invoked if git encounters any LFS pointers. But because of the mirror setup LFS endpoint in /checkouts/[Project] points to /repositories/[Project]-hash folder. You can verify this by running git lfs env | grep -E "Endpoint|SSH" in /checkouts/[Project] folder.
This is the reason behind remote object missing error as /checkouts/[Project] folder is pointing to /repositories/[Project]-hash for LFS objects and this folder doesn't have the LFS objects, because git lfs fetch isn't invoked from this directory. Also if you run git lfs env | grep -E "Endpoint|SSH" in /repositories/[Project]-hash folder we can see the correct Endpoint/SSH Urls pointing to remote server.
Checkout Folder -> LFS endpoint Pointing to Mirror Clone -> Mirror Clone Folder -> LFS endpoint pointing to remote server
I guess the root cause of this issue is because of the mirror setup. If we were to run git lfs pull in repositories/[Project]-hash folder, LFS objects would be fetched and everything would work fine.
. SPM makes a mirror clone
. Checkout out the master branch to checkouts/[Project]-hash folder
. Git tries to replace LFS pointers during the checkout process
. Git tries to fetch LFS objects from the local remote which is repositories/[Project]-hash folder
. repositories/[Project]-hash folder doesn't have the LFS objects resulting in remote missing object error
I think the XCBUtil.BinaryReaderError happens when git lfs isn't installed or setup properly(missing softlink or binary in $(xcode-select -p)/usr/bin folder)
At this point of time there is no direct fix for this issue as it is caused by SPM and the way it handles GIT Clones which isn't playing well with LFS objects. One workaround could be to manually clone/setup the LFS configured repo and adding it as a local package
Is anyone able to provide insight as to the reason for the repo mirror in SourcePackages/checkouts? I'm wondering if there was a reason this was added.
It sounds like if .../checkouts were not used and the local repo in SourcePackages/repositories/[Project]-hash were to have its remote point to the actual remote then Git LFS would "just work". What problems would be re-introduced if this .../checkouts were removed?
I tried adding a .lfsconfig file to my repo, and this seems to have resolved the issue when using swift build, but it still doesn't work in Xcode. I've already tried manually running git lfs pull inside the DerivedData/.../SourcePackages/checkouts/... dir, and deleting ~/Library/Caches/org.swift.swiftpm. Is there anything else you did that made it work in Xcode?
@jtbandes I guess my answer, which is quite old now, hasn't evolved. I don't actually remember which commands I executed and in which order but I was able at the time to resolve the git lfs references to actually be downloaded. However this was never working without additional commands, therefore not sustainable.
In my experience, Xcode & SPM are still not able to support package that would use git lfs. And I am deeply sorry about that.
I found a way around this for people who want to use LFS and SPM together, without making their package consumers add steps for installing/linking LFS. Note: this only works if the files you have in LFS are optional, and not needed as part of your package build. In my case we used it for snapshot test PNGs (only needed to run tests, not needed to build/use the package).
Basically you make LFS optional, and only configure it for package developers, but not package consumers.
Remove the lfs config lines from your .gitattributes file and check in this removal. Make sure this change makes it into main/master as SPM checks that our first, and will fail with LFS config.
Move the lines you removed into your .git/info/attributes file -- this is local to only you and isn't checked in. This allows you as the package dev to keep using LFS. I made a gitattributes_package_dev_template file with instructions on how to setup LFS locally for other package devs, and checked those instructions in.
run swift package purge-cache to remove old cache
Try a fresh package clone in another dir. You should see the files pulled down as pointers (~150 bytes), not giant binaries. Nice because package install is fast as we are skipping the large files.
Try installing your package in a fresh Xcode project. It should now work!
Optional: add git lfs install && git lfs pull --include="*" --exclude="" to a script people run when they do local package development. This will install LFS hooks, and convert the pointers into binaries.
Setup a new release if you already have releases. Prior ones will be stuck with LFS issues.