So after a ton of trial and error I was able to get it sorted out, but this feels like a bug. The issue appears to be the headers are not correctly copied or the path is incorrect from the XCFramework during the archiving process. (Maybe xcframework cli tool is buggy?) Although Swift PM normally recognize the OpenCV headers from XCFramework, builds correctly, and runs correctly.
I had to manually copy the opencv headers for the target while retaining the correct directory structure and include them. But the exact same headers and structure exists inside XCFramework. So this feels like a big bug to me.
I included the working package.
I didn't take a look at the slack thread yet, but if I get time I will.
let package = Package(
targets: ["MyCVPackage", "opencv2"]),
], targets: [
], cxxLanguageStandard: .cxx11