swift-5.1 SwiftSyntax only needs the parser library, not a full toolchain. There's the build-parser-lib script which is designed to only build the parser library and nothing else. It's WIP to get it to build the parser library for iOS, and any contributions would be very welcomed.
Once you have the parser library then you only need to build the SwiftSyntax side for iOS by referencing its source files in an Xcode project that was setup for this.