After trying all the suggestions I could find, I came to the conclusion that .binaryTarget is not suitable for use with local frameworks in Xcode 12.3 due to the bug described above (link), unless there's something I'm missing here.
We've tried many different approaches, yet every time, we get intermittent failures where the build system can't find the XCFramework from the binaryTarget. Definitely feels like dependency-resolution bug in Xcode, but I can't rule out some bizarre project-file corruption in our app or similar.
On another approach, I came up with a half-working solution here using .unsafeFlags and .linkedFramework arguments: Binary Frameworks with SwiftPM - #19 by 1oo7 (in case it's helpful to anyone.)