@ylorn I know this is an old thread, but I have tried achieving this and it appears to be possible with Swift Packages.
The only difference to your setup is that rather than the TestLibrary being a Cocoa Touch Static library, it's a Swift Package that produces a static library. I'm not sure what's different between the two that allows this. TestLibrary still appears in the Import of TestFramework, but the App Target does not need to explicitly depend on TestLibrary. As @jrose says, there's additional work that needs to be implemented with @_implementationOnly or something like an internal import to prevent any leakage of TestLibrary, but this seems like an OK solution until then.