If you can share a small sample, that may be helpful. We've had challenges around xcframeworks that import swift packages but were able to work around them. It's hard to say what the issue is as there are several things that can go wrong.
Finally, I converted all my xcframework projects to Swift packages except the iOS App project. This solves all dependency issues, but it creates another issue that I can not build a Swift package to a binary xcframework. Eventually, I fixed this too. Now, it works perfectly.