If you are trying to set this up as a Swift package that others depend on for their products, they will end up installing the LLVM libraries, including libc++, which shouldn't be an issue anymore.
Are you sure you don't have conflicting pkg-config files, one with Homebrew, one coming from SwiftPM as part of its dependency loading? That could cause confusion as well.