[build-script] Splitting the toolchain and stdlib build (more detail)

Development Compiler
1

Hey everyone. This post is going to explain how I am going to accomplish
splitting the toolchain and stdlib builds and most importantly how this will
affect you.

Splitting the Toolchain/Stdlib (Big Picture)

To perform this transition, I have written new code in build-script that can be
used to implement a multi-stage swift with a single implementation. It does this
by splitting arguments into stage dependent and stage independent
arguments. When it accesses stage dependent arguments, behind the scenes the
machinery actually dispatches to an option with an additional stage specific
postfix (e.x.: args.my_option would return the value associated with
my-option-stage2). This has enabled me to write a single basic implementation
for a swift build-script product that can be used to create arbitrary numbers of
stages and eventually use it to replace the stage 1 build-script-impl swift
build product. This would be a major achievement for our project since swift's
build-script-impl product is one of the worst parts of build-script-impl and
thus will advance us a significant way towards eliminating build-script-impl.

In order to test this out (before I eliminate the swift build-script-impl
product), I am using this to implement what I call a stage 2 stdlib. This is a
build-script product whose build logic inherits from stage generic swift, but
turns off building the toolchain and builds the stdlib against what the build
itself considers to just be a misc host compiler. Of course in the case of our
builds, that misc host compiler will be a very specific compiler, the just built
compiler. I am now with in striking distance of landing this, so I wanted to
send this post to provide details on how the transition will occur and what to
expect. I also describe below how this will affect local workflows and CI
workflows (after consultation with @mishal_shah and others).

Local Build Effects

The main effects that this will have on the build system are that:

  • If one passes in the flag --swift-stdlib-stage2 (and always once we enable
    this by default), one will see a new build product configure/built called
    swiftstdlibstage2. It will have a separate build directory from swift itself
    (e.x.: swiftstdlibstage2-macosx-x86_64). One can control the stdlib build
    using all of the normal options one is familiar with except one appends the
    postfix -stdlib-stage2 to the option name (e.x.: -enable-my-stdlib-option
    -> enable-my-stdlib-option-stdlib-stage2).

  • The stage 1 swift compiler will only build the toolchain. The stdlib and unit tests will only be built by the stage2 compiler.

  • Tests will be built/run by the stage2 compiler. This matches what we already
    do today: we do not test the toolchain before we use it to build the stdlib. I
    already have the stage 2 split stdlib performing this action successfully on
    mac/linux. This means that lit tests will no longer be run in the swift
    product dir... instead they will be run from the swiftstdlibstage2 dir. This
    may impact incremental workflows that build/test/loop in the build directory
    since one will need to run ninja in one directory (swift-*) and lit in a
    different directory (swiftstdlibstage2-*). I am going to change the stdlib
    stage2 to generate a script at configure time that will let you run ninja in
    the swift build directory in the short term so it isn't too annoying. With
    time, we may be able to add a ninja command in stage 2 swift to do this if
    people want, but I haven't looked into how I would do it.

The Bringup

In order to bring this up, I am going to be taking the following steps:

  1. I am going to finish upstreaming the stage generic swift code. It will be
    initially disabled.

  2. Then I am going to enable it on various portions of the CI (testing the
    relevant jobs each time) that way we have a nice incremental run way and will
    not be disrupted. My plan is to first start with the toolchain bot and then I
    am going to enable it everywhere else. If one hits an error, to reproduce
    this locally, one must just pass in --swift-stdlib-stage2 and one should be
    able to reproduce.

  3. Once we have the CI working and this enabled everywhere, I am going to make
    swift-stdlib-stage2 a default built product like llvm and swift. Then we can
    get to the fun work of cleaning up our cmake! = ).

  4. Once I have flipped the switch, in consultation with @typesanitizer , I am going to
    update the relevant getting started documentation so we can not have those
    valuable resources atrophy.

As I do this work, I am going to be posting status updates in this forum thread.

Michael

7 Likes