As far as I understand it, this is not possible with SwiftPM today.
Your workaround is probably your best bet until a later proposal to create dynamic libraries is supported, and [PITCH] Support for binary dependencies - #89 by jakepetroules is resolved.