SwiftPM with Git LFS

Using Swift
13

Adding to what @jml5qh and @davidharris had mentioned here is my understanding of the issue

Here is what happens when SPM tries to download a git repo with LFS configured.

  1. SPM tries to clone the repository with a local mirror(DerivedData/Runner/SourcePackages/repositories/[Project]-hash) and checks out into a local folder(DerivedData/Runner/SourcePackages/checkouts/[Project]). This Mirror folder acts a remote git server located in Local/LAN. You can verify this by running git config -l | grep remote.origin.url in /checkouts/[Project] folder.
  2. The /repositories/[Project]-hash folder contains files related to git file system and /checkouts/[Project] folder contains all the package files.
  3. During the checkout process GIT LFS Pull(fetch + checkout) is invoked if git encounters any LFS pointers. But because of the mirror setup LFS endpoint in /checkouts/[Project] points to /repositories/[Project]-hash folder. You can verify this by running git lfs env | grep -E "Endpoint|SSH" in /checkouts/[Project] folder.

This is the reason behind remote object missing error as /checkouts/[Project] folder is pointing to /repositories/[Project]-hash for LFS objects and this folder doesn't have the LFS objects, because git lfs fetch isn't invoked from this directory. Also if you run git lfs env | grep -E "Endpoint|SSH" in /repositories/[Project]-hash folder we can see the correct Endpoint/SSH Urls pointing to remote server.

Checkout Folder -> LFS endpoint Pointing to Mirror Clone -> Mirror Clone Folder -> LFS endpoint pointing to remote server

I guess the root cause of this issue is because of the mirror setup. If we were to run git lfs pull in repositories/[Project]-hash folder, LFS objects would be fetched and everything would work fine.

. SPM makes a mirror clone
. Checkout out the master branch to checkouts/[Project]-hash folder
. Git tries to replace LFS pointers during the checkout process
. Git tries to fetch LFS objects from the local remote which is repositories/[Project]-hash folder
. repositories/[Project]-hash folder doesn't have the LFS objects resulting in remote missing object error

I think the XCBUtil.BinaryReaderError happens when git lfs isn't installed or setup properly(missing softlink or binary in $(xcode-select -p)/usr/bin folder)

At this point of time there is no direct fix for this issue as it is caused by SPM and the way it handles GIT Clones which isn't playing well with LFS objects. One workaround could be to manually clone/setup the LFS configured repo and adding it as a local package

Hope this helps!!

5 Likes