Distributing several xcframeworks in a single Swift Package

I currently have a set of xcframeworks which have dependencies between each other, and one has external dependencies (to a public SwiftPM package).

I've tried putting them all into a single Package, which declares several .library(), each attached to a .binaryTarget(), but when trying to use this package in a sample code, I get a lot of errors which seem to mostly come from the fact that the external dependencies are not "seen". They are properly specified in the package, the package builds on its own, but when used in an app, it's a if there are not there. Has anyone every tried this ? I don't think I have a particularly exotic use-case.

Thanks for any help