SwiftPM with Git LFS

Are there any known issues with using Git LFS with SwiftPM in Xcode? I did some testing and found the following:

  1. I first tried tracking a very small file in a Swift Package via Git LFS. Xcode successfully adds this package as a dependency in a new Xcode project. However, the actual file isn't downloaded / included. When I clone the repo via the command line, the file is downloaded correctly. I created a test repo with a very small file tracked via Git LFS: https://github.com/jml5qh/swiftpm-lfs.

  2. I also tried tracking a very large file in a Swift Package via Git LFS. I am unable to add this package as a dependency in another Xcode project and get this error:

Error downloading object: Sources/SwiftPM-LFS/GStreamer (b857590): Smudge error: Error downloading Sources/SwiftPM-LFS/GStreamer (b8575902c6de4d8b4d7bd6dbf9a5b7c91af57a2e3770d1a9074860aea985f87e): error transferring "b8575902c6de4d8b4d7bd6dbf9a5b7c91af57a2e3770d1a9074860aea985f87e": [0] remote missing object b8575902c6de4d8b4d7bd6dbf9a5b7c91af57a2e3770d1a9074860aea985f87e

I'm able to clone this repo / checkout the branch via the command line. If you want to reproduce, you can use the lfs-big branch in https://github.com/jml5qh/swiftpm-lfs.

6 Likes

This might help:

Since Xcode uses its own tools to handle swifft packages, git-lfs might not be picked up if it is installed via Homebrew. Run the following command to create a symbolic link if Xcode fails to fetch the package:

ln -s /usr/local/bin/git-lfs $(xcode-select -p)/usr/bin/git-lfs

Quote from

3 Likes

Thanks for sharing this repo! Unfortunately, still running into the same issue. I can add the package to Xcode, but the actual asset tracked with git lfs isn't downloaded. For FluentDarkModeKit, it doesn't actually matter since the png tracked by git lfs is just for the README and isn't used in the real package.

Hello @jml5qh

So, I came across the same issue and it's partially solved on my side.

1- The first thing to do, is to make sure Xcode's git has access to git-lfs. Which, on a local machine require the symlink command unless you specified which git to use...somehow I couldn't find that option in Xcode anymore, I always used Xcode's git anyways. So I needed the symlink.

2- It seemed to work for a while, I had the impression Xcode's Swift Package Manager was pulling git-lfs files normally until I faced your issue when I tried manually in the DerivedData checkouts directory, to perform git lfs pull. And in fact, the issue seemed to be authentication with the lfs server. I updated my git repo to add a .lfsconfig files that adds the url and then git lfs pull worked properly.

3- In the end, doing so manually afterwards allowed me to work with the Swift Package properly in Xcode.

So now I am trying to figure out how to have that done automatically when SPM clones the said package. Or at last I'll try to get a script-step that knows where to perform the git-lfs pull on my CI

Note: I'm far from a git / SwiftPM expert so I may have some of these items incorrect, but I think I figured out the issue.

It looks like SwiftPM uses a local mirror when cloning / checking out the git repo. So there will be a mirror in DerivedData/[Project]/SourcePackages/repositories. When resolving a branch / tag, SwiftPM will checkout from that local mirror into DerivedData/[Project]/SourcePackages/checkouts which is where the actual package files are stored. However, that local mirror doesn't actually have the lfs objects (sounds like this is expected when working with mirrors). This means git isn't able to resolve the files from git lfs in your checkout directory. It does look like there is a possible solution - SwiftPM could call git lfs fetch --all origin [revision] which will grab / store the lfs files into the repositories folder (info taken from https://github.com/git-lfs/git-lfs/issues/1338)

@jml5qh did you get anywhere with this or still experiencing the same problem?

I wish! I uploaded a sample project and potential solution here, in a Feedback, and on the SwiftPM slack but have not received a response.

2 Likes

I am getting this error after SPM is added - The operation couldn’t be completed. (XCBUtil.BinaryReaderError error 2.)
No other explanation or message is given. I had created another forum - SwiftPM with Git LFS
This is the spm link - GitHub - abhisheksaralaya/CCProCalls

Seeing this error as well. Trying to vendor a framework using git-lfs, and we get something like:

MyApp/.build/checkouts/MyPackage: error: Couldn’t check out revision ‘12cc5f11cace7d9eec30dfcdab774775c48dc965’:
    Downloading vendor/MyFile (1.0 KB)
    Error downloading object: vendor/MyFile (6efa24a): Smudge error: Error downloading vendor/MyFilet (6efa24acbfce2557b76adefe2907d8718152ce9ed4b63f2d37ad13d8629d74e3): error transferring "6efa24acbfce2557b76adefe2907d8718152ce9ed4b63f2d37ad13d8629d74e3": [0] remote missing object 6efa24acbfce2557b76adefe2907d8718152ce9ed4b63f2d37ad13d8629d74e3
    Errors logged to MyApp/.build/checkouts/MyPackage/.git/lfs/logs/20211109T214122.512067.log
    Use `git lfs logs last` to view the log.
    error: external filter 'git-lfs filter-process' failed
    fatal: vendor/NMAKit.xcframework/Info.plist: smudge filter lfs failed

Looking into this: the problem seems to be because SPM runs a git clone -c core.symlinks=true --mirror which doesn't pull in LFS references.

If you add a git lfs fetch --all after creating the mirror, that pulls in the LFS references, and then the rest of the chain works. Unsure what the best way to orchestrate this would be or if it's in the roadmap, I'm not sure if there is a way to do this such that the lfs fetch only happens for repos that need it. The issue is with the "--mirror" parameter, from what I can tell.

1 Like

We've now run across this issue with the same conclusions as others on this post.

It will block us from releasing a SPM supported framework.

1 Like

Is there any update on a solution to this issue, or is a workaround posible?

We are experiencing the same issue and it is preventing our org from releasing our SDK as a Swift Binary Package.

Adding to what @jml5qh and @davidharris had mentioned here is my understanding of the issue

Here is what happens when SPM tries to download a git repo with LFS configured.

  1. SPM tries to clone the repository with a local mirror(DerivedData/Runner/SourcePackages/repositories/[Project]-hash) and checks out into a local folder(DerivedData/Runner/SourcePackages/checkouts/[Project]). This Mirror folder acts a remote git server located in Local/LAN. You can verify this by running git config -l | grep remote.origin.url in /checkouts/[Project] folder.
  2. The /repositories/[Project]-hash folder contains files related to git file system and /checkouts/[Project] folder contains all the package files.
  3. During the checkout process GIT LFS Pull(fetch + checkout) is invoked if git encounters any LFS pointers. But because of the mirror setup LFS endpoint in /checkouts/[Project] points to /repositories/[Project]-hash folder. You can verify this by running git lfs env | grep -E "Endpoint|SSH" in /checkouts/[Project] folder.

This is the reason behind remote object missing error as /checkouts/[Project] folder is pointing to /repositories/[Project]-hash for LFS objects and this folder doesn't have the LFS objects, because git lfs fetch isn't invoked from this directory. Also if you run git lfs env | grep -E "Endpoint|SSH" in /repositories/[Project]-hash folder we can see the correct Endpoint/SSH Urls pointing to remote server.

Checkout Folder -> LFS endpoint Pointing to Mirror Clone -> Mirror Clone Folder -> LFS endpoint pointing to remote server

I guess the root cause of this issue is because of the mirror setup. If we were to run git lfs pull in repositories/[Project]-hash folder, LFS objects would be fetched and everything would work fine.

. SPM makes a mirror clone
. Checkout out the master branch to checkouts/[Project]-hash folder
. Git tries to replace LFS pointers during the checkout process
. Git tries to fetch LFS objects from the local remote which is repositories/[Project]-hash folder
. repositories/[Project]-hash folder doesn't have the LFS objects resulting in remote missing object error

I think the XCBUtil.BinaryReaderError happens when git lfs isn't installed or setup properly(missing softlink or binary in $(xcode-select -p)/usr/bin folder)

At this point of time there is no direct fix for this issue as it is caused by SPM and the way it handles GIT Clones which isn't playing well with LFS objects. One workaround could be to manually clone/setup the LFS configured repo and adding it as a local package

Hope this helps!!