The TensorFlow library is a SwiftPM package. It depends almost exclusively on the differentiable programming feature, which we plan to push through Swift Evolution this year.
Assuming the differentiable programming feature had landed in master, the only other thing you need to compile TensorFlow package is to set up a CTensorFlow module map, which you can do with SwiftPM.
So in short, a fork of the compiler toolchain is not where we are headed in the long term. Instead, we try to push technologies to become included in the official Swift.