I think that my experiences packaging and porting software for Gentoo and Exherbo have given me a different perspective. There are subtle differences, there is no stable ABI for Linux. Each distribution is built differently with a different version of the libraries, and different compiler.
The C++ runtime is not ABI stable, the libstdc++ version must match exactly. If this is not done, you can find odd crashes at various points.
Similarly, the C library has subtle configuration differences and can result in similar failures as well.
The file system layout is "standard", so clearly, everyone has their own. Those differences matter because the package is going to be just completely ignored by the distro maintainers unless it conforms to the distribution's policies.
Integrating into the package manager is generally a good idea, not doing so means that any changes to the system configuration can accidentally render your tools broken. By integrating into the package manager, you ensure that the changes to the system are aware and can flag the problems in the first place.
These may seem like nice to have little details and a way to stop Swift from being available on Linux distributions, but, at the scale of distributing the package to distribution users, these become extremely important. More so, making it broadly available with a poor experience with the toolchain just not working the first time around is likely to be more detrimental to the project than not having it packaged.
I'm happy to go into the litany of the nuances associated with wider distribution of Swift on Linux, but, I don't think that it would be complete nor useful.