Hey everyone. This post is going to explain how I am going to accomplish
splitting the toolchain and stdlib builds and most importantly how this will
affect you.
Splitting the Toolchain/Stdlib (Big Picture)
To perform this transition, I have written new code in build-script that can be
used to implement a multi-stage swift with a single implementation. It does this
by splitting arguments into stage dependent and stage independent
arguments. When it accesses stage dependent arguments, behind the scenes the
machinery actually dispatches to an option with an additional stage specific
postfix (e.x.: args.my_option
would return the value associated with
my-option-stage2
). This has enabled me to write a single basic implementation
for a swift build-script product that can be used to create arbitrary numbers of
stages and eventually use it to replace the stage 1 build-script-impl swift
build product. This would be a major achievement for our project since swift's
build-script-impl product is one of the worst parts of build-script-impl and
thus will advance us a significant way towards eliminating build-script-impl.
In order to test this out (before I eliminate the swift build-script-impl
product), I am using this to implement what I call a stage 2 stdlib. This is a
build-script product whose build logic inherits from stage generic swift, but
turns off building the toolchain and builds the stdlib against what the build
itself considers to just be a misc host compiler. Of course in the case of our
builds, that misc host compiler will be a very specific compiler, the just built
compiler. I am now with in striking distance of landing this, so I wanted to
send this post to provide details on how the transition will occur and what to
expect. I also describe below how this will affect local workflows and CI
workflows (after consultation with @mishal_shah and others).
Local Build Effects
The main effects that this will have on the build system are that:
-
If one passes in the flag
--swift-stdlib-stage2
(and always once we enable
this by default), one will see a new build product configure/built called
swiftstdlibstage2. It will have a separate build directory from swift itself
(e.x.:swiftstdlibstage2-macosx-x86_64
). One can control the stdlib build
using all of the normal options one is familiar with except one appends the
postfix-stdlib-stage2
to the option name (e.x.:-enable-my-stdlib-option
->enable-my-stdlib-option-stdlib-stage2
). -
The stage 1 swift compiler will only build the toolchain. The stdlib and unit tests will only be built by the stage2 compiler.
-
Tests will be built/run by the stage2 compiler. This matches what we already
do today: we do not test the toolchain before we use it to build the stdlib. I
already have the stage 2 split stdlib performing this action successfully on
mac/linux. This means that lit tests will no longer be run in the swift
product dir... instead they will be run from the swiftstdlibstage2 dir. This
may impact incremental workflows that build/test/loop in the build directory
since one will need to run ninja in one directory (swift-*
) and lit in a
different directory (swiftstdlibstage2-*
). I am going to change the stdlib
stage2 to generate a script at configure time that will let you run ninja in
the swift build directory in the short term so it isn't too annoying. With
time, we may be able to add a ninja command in stage 2 swift to do this if
people want, but I haven't looked into how I would do it.
The Bringup
In order to bring this up, I am going to be taking the following steps:
-
I am going to finish upstreaming the stage generic swift code. It will be
initially disabled. -
Then I am going to enable it on various portions of the CI (testing the
relevant jobs each time) that way we have a nice incremental run way and will
not be disrupted. My plan is to first start with the toolchain bot and then I
am going to enable it everywhere else. If one hits an error, to reproduce
this locally, one must just pass in --swift-stdlib-stage2 and one should be
able to reproduce. -
Once we have the CI working and this enabled everywhere, I am going to make
swift-stdlib-stage2 a default built product like llvm and swift. Then we can
get to the fun work of cleaning up our cmake! = ). -
Once I have flipped the switch, in consultation with @typesanitizer , I am going to
update the relevant getting started documentation so we can not have those
valuable resources atrophy.
As I do this work, I am going to be posting status updates in this forum thread.
Michael