Right, and this is the problem I want to highlight: to get that "full Swift toolchain" to properly bootstrap would require bringing up Dispatch and Foundation. That is a lot of work relative to just bringing up Swift as it stands today.
The idea is that since these internal compiler packages would support both CMake and SPM, you only use CMake for that initial bootstrap, then only use SPM to build internal compiler packages once you have SPM working well for your new platform. As such, I don't think that SPM aspect will make a difference when bringing new platforms up.
Ok, maybe you can help me understand. So if SwiftPM can cross-compile, and SwiftPM is cross-platform itself, then why isn't the compiler a bunch of Swift Packages? I realize LLVM would still have to use its CMake build system but that's a separate thing.
This is great! Migrating compiler code to Swift would simplify a lot and make contributions easier.
Why not require the host compiler be the latest release, and build everything at step 2?
It seems like steps 3-7 add a lot of complexity in order to get the latest optimizations in the scenario where the host compiler is old.
But the most common scenario is that the host compiler will be the latest release.
Unless we require unreleased optimizations from the main compiler under development, the 7-stage process is optimizing for the worst case, and adding failure modes that could be hard to debug (particularly if semantics do depend on main compiler changes).
When the latest released compiler is not available on the host, but an older one is, we can build the latest released compiler (using the process & tooling that was required for it). In the degenerate case this may require building each release in turn, but it's a one-time operation.
While this might seem like more work for new platforms, it's likely to be easier since the steps between releases are clear and incremental, and likely to be easier to debug that a linking error.
This would put less strain on the build process. We're not requiring swiftpm to remember how to build much older versions, and a change to current swiftpm need not maintain compatibility or indeed be tested so far back. There's no need to re-run CI for released compilers.
CMake is portable and fairly good at cross-compilation. It's likely that a given host platform to which we want to port in the future will already have working CMake cross-compilation, and therefore require less work than porting SwiftPM.
SwiftPM is great for vast majority of folks. Open a SwiftPM package in VS Code or Xcode and you can just use it, or swift build from the command line, or whatever. Over the long term, I'd love to see us trend toward using packages more and CMake less... but today's reality is that SwiftPM needs work to make cross-compilation more fluid, and until we've done that work we cannot drop CMake.
Xcode is only relevant on macOS, the Swift compiler builds on number of other platforms (Linux, Windows, flavors of BSD, Android, iOS). And on macOS, folks working on the compiler choose various tools for their own reasons. Over a (long) time, I think we can get to a SwiftPM-centric build, but it's specifically not part of this plan because it is a long path. However, part of the goal here is that the SwiftPM packages for parts of the compiler that are in Swift will be easy to develop, because you can use the package and avoid the CMake. This is how we've developed the Swift Driver, and it's worked quite well.
We could make reducing dependencies part of the plan, although I'd be opposed to blocking any Swift code from the compiler based on those reduced dependencies. For example, I think we could replace uses of Dispatch in the compiler's stack with Swift Concurrency, which (as @Max_Desiatovnoted) does not require Dispatch. The single-threaded cooperative executor already exists, and one could bring up a thread pool for any other platform.
Yes. Until we've stabilized C++ interoperability, this will involve a bunch of very tedious work at the Swift/C++ boundary, but we can't leave ourselves in a position where we cannot build the compiler with anything but last week's compiler.
Doug
[EDIT: Dropped a paragraph that went with a separate reply]
"Latest release" could be a day old, and folks haven't even had a chance to adopt it yet, especially when "adopting" a new compiler might be a matter of updating an entire build fleet. Plus, out-of-tree ports to other platforms often work off the last released compiler, so they might need some additional time after the release. That's why we have a time-based "all releases in the last year", to make sure that folks have time to move up to a newer release before they fall off the path.
It's a one-time operation if you're doing it for yourself. It's a build-scripting nightmare if you're doing it in CI or any other automated build environment.
SwiftPM has to maintain compatibility with older package manifests. It doesn't need to maintain compatibility with older Swift compilers, because it ships alongside the Swift compiler. It's also not involved in the initial build of a Swift compiler in this scheme, because that's all done by CMake.
You have a valid point that the build process for the "optional" parts of the compiler is complicated. It's there to support rapid iteration on C++ interoperability, so we can make that feature great before it moves from the "experimental" stage to being a full-fledged feature that requires us to maintain source compatibility going forward. Once that happens, and we get past the 12-month window, the complexity from the "optional" parts just go away---and I suspect we wouldn't ever bring them back.
Well, to begin with, the Swift compiler is mostly written in C++ right now: the cloc code count utility reports 581 klocs of C++, 172 klocs of C/C++ headers, and only 5 klocs of Swift in the compiler directories include/, lib/, and SwiftCompilerSources/ from the latest Aug. 6 trunk source snapshot. Somebody has to first rewrite or translate all that C++ code to Swift.
Anyway, this is getting off-topic for this thread, so maybe open a new thread if you have more such questions.
The Zig programming language has made inroads in many companies not for the language itself, but for its build system, which fully supports C family languages and provides much more seamless "package" and cross compilation support than the native C frontends usually provide for themselves. Improving swiftpm to the point it can take over the majority of the Swift compiler's build system could have knock-on effects for the Swift ecosystem, if swiftpm could also find a niche as a "better C build system" in addition to being the Swift build system.
In case someone missed this: we are already doing that. It's the --bootstrapping=hosttools build-script option. But yes, currently (because evolving C++ interop) this only works when installing a recent snapshot toolchain.
Also note: It is possible to do the above with dependency-based build systems (such as Make or Ninja), but it’s tricky to get right. Note that a naive version would have the optional bits depend on both the stage 1 and stage 2 compilers.
The --bootstrapping=bootstrapping-with-hostlibs mode (which is the default on macOS) models the bootstrapping logic with cmake dependencies. When you type ninja swift-frontend everything is correctly rebuilt - including stage 1 and 2 compilers, which is nice (no need to use build-script for incremental compiler changes). And it's very efficient. The overall build time overhead compared to the hosttools mode is only ~1 minute.
I really like swift, cause now it is fully bootstrappable.
Allow Swift as a Swift compiler language:
make it very hard to bootstrap. And I am not talking about new architectures, I talking about some non-glibc linuxes, for example.
Swift compiler build time will be much longer, cause of new stages(stage0, stage1, etc).
I think that swift will never will be rewritten in swift only, so we will have TWO languages as dependencies, and not one, as of now. Regular swift compiler developer will need to know two languages, instead of one(I am not talking about swift system library, of course)
TL;DR - what we will get for those obvious complication?
Nothing in the proposal implies. The new stages only need to exist for optional components, and they are alreadty there; the mandatory components discussed in the proposal don't require them.
That doesn't matter. We can derive benefits elsewhere from those components that have been rewritten in Swift. It's not worth rewriting a compiler component in Swift just to have the language be Swift; there must be some additional benefit, such as usability in a library that can improve other tools. For example, the library-based architecture of the new Swift driver enables integration in build systems, and that integration has unlocked improvements in build-time performance.
Are you really trying to make the case that someone working on the Swift compiler shouldn't have to know Swift?
Better implementations of compiler components that are reusable throughout more of the Swift ecosystem and are more approachable for Swift developers. Plus, right now we're maintaining two implements of the Swift Driver, so moving to the Swift one (which is the better implementation) reduces maintenance overhead and technical debt.
I'm glad that @Finagolfin pointed me to this topic as it will help keep me motivated. I am packaging Swift 5.6.2 for Debian which at present does not have Swift available in the official apt repos. If I can get 5.6.2 or 5.7 into Debian before this switch to mandatory Swift is made then Debian and Ubuntu based distros will have an easier path forward for 5.8 and beyond.
The hardest part of packaging for Debian isn't getting the build to finish and get a working binary, it is doing so using the Debian build tools and having everything cleaned up and in accordance with Debian policies. Hopefully there isn't a policy that says I can't have swift as a build dependency for swift. I'll have to test that out soon.
Hopefully there isn't a policy that says I can't have swift as a build dependency for swift. I'll have to test that out soon.
I doubt that there is or you'd never be able to build any self-hosted compiler as a Debian package, like the ghc Haskell compiler package or as Swift is becoming.
Awesome! Will it be a monolithic package or will the runtime libraries be separated from the actual compiler? The latter would make it more convenient for 3rd party packages.
