Reading further, it seems there have been a lot of problems with using .binaryTarget in SPM, such as Xcode not signing them, or Xcode not copying them into the build folder before trying to compile the code in the module, etc.
It has started to occur to me—what we need is a way to list an XCFramework in the "dependencies" section of a Swift package. "Target" should only be used for when the binary is published as part of your package.
It seems .binaryTarget was created as a way to let someone wrap an XCFramework inside of a Swift Package so they can publish a closed-source package, like how Google now publishes Firebase.
But what we users need, is a way to link an XCFramework (that doesn't have a Swift package) into a locally-declared Swift package (that's not published anywhere and which we're basically just using as a way to avoid using .xcproj files to organize code in our app).