SwiftPM with Git LFS

Using Swift
1

Are there any known issues with using Git LFS with SwiftPM in Xcode? I did some testing and found the following:

  1. I first tried tracking a very small file in a Swift Package via Git LFS. Xcode successfully adds this package as a dependency in a new Xcode project. However, the actual file isn't downloaded / included. When I clone the repo via the command line, the file is downloaded correctly. I created a test repo with a very small file tracked via Git LFS: GitHub - jml5qh/swiftpm-lfs.

  2. I also tried tracking a very large file in a Swift Package via Git LFS. I am unable to add this package as a dependency in another Xcode project and get this error:

Error downloading object: Sources/SwiftPM-LFS/GStreamer (b857590): Smudge error: Error downloading Sources/SwiftPM-LFS/GStreamer (b8575902c6de4d8b4d7bd6dbf9a5b7c91af57a2e3770d1a9074860aea985f87e): error transferring "b8575902c6de4d8b4d7bd6dbf9a5b7c91af57a2e3770d1a9074860aea985f87e": [0] remote missing object b8575902c6de4d8b4d7bd6dbf9a5b7c91af57a2e3770d1a9074860aea985f87e

I'm able to clone this repo / checkout the branch via the command line. If you want to reproduce, you can use the lfs-big branch in GitHub - jml5qh/swiftpm-lfs.

8 Likes
2

This might help:

Since Xcode uses its own tools to handle swifft packages, git-lfs might not be picked up if it is installed via Homebrew. Run the following command to create a symbolic link if Xcode fails to fetch the package:

ln -s /usr/local/bin/git-lfs $(xcode-select -p)/usr/bin/git-lfs

Quote from

4 Likes
3

Thanks for sharing this repo! Unfortunately, still running into the same issue. I can add the package to Xcode, but the actual asset tracked with git lfs isn't downloaded. For FluentDarkModeKit, it doesn't actually matter since the png tracked by git lfs is just for the README and isn't used in the real package.

4

Hello @jml5qh

So, I came across the same issue and it's partially solved on my side.

1- The first thing to do, is to make sure Xcode's git has access to git-lfs. Which, on a local machine require the symlink command unless you specified which git to use...somehow I couldn't find that option in Xcode anymore, I always used Xcode's git anyways. So I needed the symlink.

2- It seemed to work for a while, I had the impression Xcode's Swift Package Manager was pulling git-lfs files normally until I faced your issue when I tried manually in the DerivedData checkouts directory, to perform git lfs pull. And in fact, the issue seemed to be authentication with the lfs server. I updated my git repo to add a .lfsconfig files that adds the url and then git lfs pull worked properly.

3- In the end, doing so manually afterwards allowed me to work with the Swift Package properly in Xcode.

So now I am trying to figure out how to have that done automatically when SPM clones the said package. Or at last I'll try to get a script-step that knows where to perform the git-lfs pull on my CI

5

Note: I'm far from a git / SwiftPM expert so I may have some of these items incorrect, but I think I figured out the issue.

It looks like SwiftPM uses a local mirror when cloning / checking out the git repo. So there will be a mirror in DerivedData/[Project]/SourcePackages/repositories. When resolving a branch / tag, SwiftPM will checkout from that local mirror into DerivedData/[Project]/SourcePackages/checkouts which is where the actual package files are stored. However, that local mirror doesn't actually have the lfs objects (sounds like this is expected when working with mirrors). This means git isn't able to resolve the files from git lfs in your checkout directory. It does look like there is a possible solution - SwiftPM could call git lfs fetch --all origin [revision] which will grab / store the lfs files into the repositories folder (info taken from How to properly mirror remote with LFS · Issue #1338 · git-lfs/git-lfs · GitHub)

1 Like
6

@jml5qh did you get anywhere with this or still experiencing the same problem?

7

I wish! I uploaded a sample project and potential solution here, in a Feedback, and on the SwiftPM slack but have not received a response.

3 Likes
8

I am getting this error after SPM is added - The operation couldn’t be completed. (XCBUtil.BinaryReaderError error 2.)
No other explanation or message is given. I had created another forum - SwiftPM with Git LFS
This is the spm link - GitHub - abhisheksaralaya/CCProCalls

9

Seeing this error as well. Trying to vendor a framework using git-lfs, and we get something like:

MyApp/.build/checkouts/MyPackage: error: Couldn’t check out revision ‘12cc5f11cace7d9eec30dfcdab774775c48dc965’:
    Downloading vendor/MyFile (1.0 KB)
    Error downloading object: vendor/MyFile (6efa24a): Smudge error: Error downloading vendor/MyFilet (6efa24acbfce2557b76adefe2907d8718152ce9ed4b63f2d37ad13d8629d74e3): error transferring "6efa24acbfce2557b76adefe2907d8718152ce9ed4b63f2d37ad13d8629d74e3": [0] remote missing object 6efa24acbfce2557b76adefe2907d8718152ce9ed4b63f2d37ad13d8629d74e3
    Errors logged to MyApp/.build/checkouts/MyPackage/.git/lfs/logs/20211109T214122.512067.log
    Use `git lfs logs last` to view the log.
    error: external filter 'git-lfs filter-process' failed
    fatal: vendor/NMAKit.xcframework/Info.plist: smudge filter lfs failed
10

Looking into this: the problem seems to be because SPM runs a git clone -c core.symlinks=true --mirror which doesn't pull in LFS references.

If you add a git lfs fetch --all after creating the mirror, that pulls in the LFS references, and then the rest of the chain works. Unsure what the best way to orchestrate this would be or if it's in the roadmap, I'm not sure if there is a way to do this such that the lfs fetch only happens for repos that need it. The issue is with the "--mirror" parameter, from what I can tell.

11

We've now run across this issue with the same conclusions as others on this post.

It will block us from releasing a SPM supported framework.

1 Like
12

Is there any update on a solution to this issue, or is a workaround posible?

We are experiencing the same issue and it is preventing our org from releasing our SDK as a Swift Binary Package.

13

Adding to what @jml5qh and @davidharris had mentioned here is my understanding of the issue

Here is what happens when SPM tries to download a git repo with LFS configured.

  1. SPM tries to clone the repository with a local mirror(DerivedData/Runner/SourcePackages/repositories/[Project]-hash) and checks out into a local folder(DerivedData/Runner/SourcePackages/checkouts/[Project]). This Mirror folder acts a remote git server located in Local/LAN. You can verify this by running git config -l | grep remote.origin.url in /checkouts/[Project] folder.
  2. The /repositories/[Project]-hash folder contains files related to git file system and /checkouts/[Project] folder contains all the package files.
  3. During the checkout process GIT LFS Pull(fetch + checkout) is invoked if git encounters any LFS pointers. But because of the mirror setup LFS endpoint in /checkouts/[Project] points to /repositories/[Project]-hash folder. You can verify this by running git lfs env | grep -E "Endpoint|SSH" in /checkouts/[Project] folder.

This is the reason behind remote object missing error as /checkouts/[Project] folder is pointing to /repositories/[Project]-hash for LFS objects and this folder doesn't have the LFS objects, because git lfs fetch isn't invoked from this directory. Also if you run git lfs env | grep -E "Endpoint|SSH" in /repositories/[Project]-hash folder we can see the correct Endpoint/SSH Urls pointing to remote server.

Checkout Folder -> LFS endpoint Pointing to Mirror Clone -> Mirror Clone Folder -> LFS endpoint pointing to remote server

I guess the root cause of this issue is because of the mirror setup. If we were to run git lfs pull in repositories/[Project]-hash folder, LFS objects would be fetched and everything would work fine.

. SPM makes a mirror clone
. Checkout out the master branch to checkouts/[Project]-hash folder
. Git tries to replace LFS pointers during the checkout process
. Git tries to fetch LFS objects from the local remote which is repositories/[Project]-hash folder
. repositories/[Project]-hash folder doesn't have the LFS objects resulting in remote missing object error

I think the XCBUtil.BinaryReaderError happens when git lfs isn't installed or setup properly(missing softlink or binary in $(xcode-select -p)/usr/bin folder)

At this point of time there is no direct fix for this issue as it is caused by SPM and the way it handles GIT Clones which isn't playing well with LFS objects. One workaround could be to manually clone/setup the LFS configured repo and adding it as a local package

Hope this helps!!

5 Likes
14

Stil today very annoying that this is not supported. cocoapods supports this and frankly spm should too. As a workaround we removed git lfs but would like to use it back in the future.

2 Likes
15

Great sleuthing, @hemachandsai!

Is anyone able to provide insight as to the reason for the repo mirror in SourcePackages/checkouts? I'm wondering if there was a reason this was added.

It sounds like if .../checkouts were not used and the local repo in SourcePackages/repositories/[Project]-hash were to have its remote point to the actual remote then Git LFS would "just work". What problems would be re-introduced if this .../checkouts were removed?

16

I'm running into this issue as well with this repo. It seems like there is already an open ticket about it:

17

I tried adding a .lfsconfig file to my repo, and this seems to have resolved the issue when using swift build, but it still doesn't work in Xcode. I've already tried manually running git lfs pull inside the DerivedData/.../SourcePackages/checkouts/... dir, and deleting ~/Library/Caches/org.swift.swiftpm. Is there anything else you did that made it work in Xcode?

18

@jtbandes I guess my answer, which is quite old now, hasn't evolved. I don't actually remember which commands I executed and in which order but I was able at the time to resolve the git lfs references to actually be downloaded. However this was never working without additional commands, therefore not sustainable.

In my experience, Xcode & SPM are still not able to support package that would use git lfs. And I am deeply sorry about that.

19

(...1 year later...) Seems like a lot of folks are still bumping into this limitation (see github issue) – @Aciid @NeoNacho any chance there has been some progress made? :slight_smile:

2 Likes
20

I found a way around this for people who want to use LFS and SPM together, without making their package consumers add steps for installing/linking LFS. Note: this only works if the files you have in LFS are optional, and not needed as part of your package build. In my case we used it for snapshot test PNGs (only needed to run tests, not needed to build/use the package).

Basically you make LFS optional, and only configure it for package developers, but not package consumers.

Steps:

  • Remove the lfs config lines from your .gitattributes file and check in this removal. Make sure this change makes it into main/master as SPM checks that our first, and will fail with LFS config.
  • Move the lines you removed into your .git/info/attributes file -- this is local to only you and isn't checked in. This allows you as the package dev to keep using LFS. I made a gitattributes_package_dev_template file with instructions on how to setup LFS locally for other package devs, and checked those instructions in.
  • run swift package purge-cache to remove old cache
  • Try a fresh package clone in another dir. You should see the files pulled down as pointers (~150 bytes), not giant binaries. Nice because package install is fast as we are skipping the large files.
  • Try installing your package in a fresh Xcode project. It should now work!
  • Optional: add git lfs install && git lfs pull --include="*" --exclude="" to a script people run when they do local package development. This will install LFS hooks, and convert the pointers into binaries.
  • Setup a new release if you already have releases. Prior ones will be stuck with LFS issues.
2 Likes