Swift package in project & unused xcframeworks

I have built a package that bundles several binary targets (xcframeworks). I've used this package in an app, which doesn't actually require those binaries, since it doesn't use the functionality provided there. Other than split up my Swift package, is there any way I can make sure these xcframeworks don't get bundled in my app?