For some reason we've never shipped clang (& lld) with the Swift toolchains and that already leads to a multitude of issues:
the blocks runtime ABI between swift-clang and the distribution's clang needs to match, otherwise we'd be in trouble with any compiled C code that's using -fblocks
clang learned a trick: -fcf-runtime-abi=swift* flag which improves CoreFoundation interoperability in cases where there's no ObjC runtime (such as on Linux)
there can be issues for SwiftPM packages when the C bits are compiled with a very outdated clang, example here
pretty sure it would be easier for SwiftPM if it had a more consistent view of the world
...
Summing up I was always puzzled by why we didn't ship a matching clang with the Swift toolchains but we seem to have gotten away so far but it gets more and more brittle. I propose to fix this and start shipping clang . And while we're at it, let's do the same with lld.
This has always seemed weird to me: I've never quite understood why we went to all the effort of building a clang only to not ship it.
Regardless, this would be very useful if we go ahead with RFC: Moving SwiftNIO SSL to BoringSSL, as it would guarantee the use of a recent clang compiler when building BoringSSL. This reduces the risk that BoringSSL ages out of some of the quite ancient clang versions that are floating around in the Swift ecosystem.
I'm not sure how I feel about this. This would have the same problem as the lldb we're already bundling: With the current release scheduling, the bundled version would be behind the upstream version, often by months. For example, Swift 4.2 (on 2018-09-17) updated the bundled lldb from 5.0.0 (released 2017-09-07) to 6.0.0 (released 2018-03-08), just two days before 7.0.0 was released.
On Linux distros which package new releases quickly (e.g. Arch Linux), this makes packaging hard and makes it hard for swift to coexist with the normal and up-to-date clang/lldb/whatever.
I see the reasons to do this, but could we in the process either prefix all the names or stash them away in a swift-specific directory that's not in the default PATH?
It's a good point. I think it's also worth taking a step back and asking who would be the primary consumer if this bundled clang. Would it be SwiftPM, building C/C++ code in a package? If so, I suspect the bundled clang could be buried somewhere or named differently so it doesn't conflict with the system's clang, and SwiftPM knows where to look for it.
On the other hand, bundling clang in this way would ensure there is a consistent toolchain for Swift across all platforms for a given release. There are definite tradeoffs here.
Personally, I've been building a complete bundle with a clang, swift, lld, lldb package all in one. I can use that to build C/C++ and swift code and debug for the targets that I care about. It works rather nicely and alleviates all the issues of the mixing and matching of versions.
This is correct. SwiftPM finds most tools relative to itself in the toolchain's bin directory. As long as something is contained in the toolchain, SwiftPM can easily locate and use it for the build process. Somewhat related: SwiftPM also allows the standard CC env variable for overriding the C compiler.
@johannesweiss if we are going to do this I wonder if we should also include libc++. We right now symlink in the system c++ stdlib and IIRC that requires some weird code to work around that when using it to build the runtime. We should just build it and move on with our lives.
In truth, I wish we just shipped a full llvm toolchain + swift. It would make the whole toolchain concept awesome on linux.
Shipping clang in the Swift toolchain will solve a long outstanding request for adding support for sanitizers in the toolchain. (Swift and clang use the same sanitizer library, which is version locked with the compilers.)
Thanks everybody for the input, this is really valuable and shows that there's a common root to some of the issues we're experiencing.
Yes please, working sanitisers on Linux would be really important for us . And shipping compiler-rt and libc++ seem like no-brainers to me too, this will really solve issues. @compnerd seems to have made good experiences with this already and we've been running things in a controlled environment where all the compiler and runtime libraries truly fit together too and again it started to make things actually work by construction rather than accident. @kevints is the expert from our side here.
@ahti does obviously have a good point here that adding more binaries in 'standard locations' does cause issues but as @tkremenek says: we can put clang and friends in a place that SwiftPM can find but is also safe not to cause trouble. This also really makes sense: The reason someone would download the Swift toolchain is obviously to use Swift but using Swift does (maybe unknowingly) mean also using clang, the runtime libraries, and the sanitisers too. And it makes total sense to have the Swift binaries (swift, swiftc, swift-*) in a more prominent place than say clang even though they're all needed by almost everybody.
I have long agreed we should ship the Clang in the toolchain. I think it is fine to give it an alternate name and teach SwiftPM and other Swift focused things to use that name.
Another good reason to include clang with the toolchain is for Xcode, since currently when you have a Swift toolchain selected, C/ObjC/C++ IDE support significantly degrades until you switch back to the default Xcode toolchain.
Maybe have an equivalent to xcrun that would know how to execute and return paths to clang etc? This would give some redirection to the actual naming and locations, perhaps making those choices less fragile.
. And while we're at it, let's do the same with 
. And shipping