SPM Binary Frameworks - "The data couldn’t be read because it isn’t in the correct format."

I'm trying to integrate the Agora SDK into my iOS app which uses SPM for all of my packages and dependencies.

Unfortunately, they don't provide an SPM package - they just provide either a zip of five xcframeworks which you can drag into your project, or integrate via Cocoapods.

I created an open-source SPM package for them here:
https://github.com/AttilaTheFun/AgoraAudioiOS
It just has the frameworks and Package.swift file.

I was able to get Xcode to accept the SPM package and I can see that it checked out the frameworks correctly + let me link them.

However, when I try to build I get an error "The data couldn’t be read because it isn’t in the correct format."

Do you have any ideas what could be causing this issue?

NOTE

I had to use Git Large File Storage because one of the frameworks is a little over 100MB.

If you would like to clone the project to reproduce the issue, you can install git lfs with homebrew:
brew install git-lfs

And then simlink this so Xcode can use it:
sudo ln /usr/local/bin/git-lfs /Applications/Xcode.app/Contents/Developer/usr/bin/git-lfs

I found a detailed explanation of a similar issue on StackOverflow, it may or may not be relevant: ios - Admob Framework class not detected by Xcode interface builder - Stack Overflow

Having the exact same issue in a test project @Logan_Shire where we have agora in an xcframework in git lfs, have you discovered anything on this issue?

Did you manage to find a solution? I also have the same exact issue