The build-script: moving toward separate configuration and build steps for each product

Hi build-script people (@compnerd @Rostepher @Michael_Gottesman @bulbazord others?)

A lot of important build infrastructure work has been happening to add features for platform support and build products. My hope is that everyone working on this shares these core principles rearchitecting the build: decoupling build products and decomposing the build process.

I don't think anyone will argue in the abstract, but let me explain how this relates to day-to-day development. Apparently it was surprising that the --skip-build option should be maintained, but this use case is exactly what we need to migrate toward to get the build-script out of developers' way.

The central function of the build script is to standardize the set of CMake flags currently used for each build product (or other configuration parameters for non-cmake builds). The build-script should only be responsible for mapping presets onto these flags. The actual build steps need to be fully separable and trivially composed on top of this functionality. (I literally never want the build-script to actually build anything when I'm developing on a branch).

It should be expected that developers mix-and-max products from different build-script invocations. There is literally never a reason for me to rebuild clang more than once no matter how many local branches (based on master), Swift build configurations, and library configurations I need to build. It's also critical to be able to switch between asserts and non-asserts stdlib without reconfiguring the Swift compiler. Similarly, I should be able to build llbuild, swiftpm, and other dependent libraries on top of any standard library configuration. This workflow is important to all swift developers whether they realize it yet or not. Not only is it important for build turnaround and debugging to have a many-to-one relationship between configurations, but it's also important for debugging to be able to independently rebuild dependent products without necessarilly rebuilding all the dependencies.

It's even more important to be able to incrementally build for new platforms without reconfiguring anything! A reasonable approach to development is to configure for everything that is available on your host, just once, and build only the targets and platforms that you need as you need them.

There is no reason to artificially couple the configuration of different build products. It sometimes looks to me like this is happening, and I desperately need the build-script to move in the opposite direction. Ideally, the only thing a build product needs to "export" to a dependent build product is a header path and binary install path. Otherwise, their configuration should be unrelated to each other. Naturally, when build products share C++ headers, they need to use the same options that affect those headers. It's reasonable for developers who decompose the build process to assume that responsibility.

I really hope this makes sense to everyone, because the build-script is still very much standing in the way of an effective workflow.

1 Like

Another thing to consider is that perhaps we need some sort of test for things like --skip-build (just saying).

1 Like

Just to elaborate, I am not saying that this is ok b/c we need testing/etc. Instead I am just trying to say that the current state of things make testing stuff like this impossible which is a general problem and shows a key issue with the current approach.

Another thing. I think that what Andy is really getting at here is that for productivity purposes he needs the ability to configure all of the build and then just build the subprojects that he needs.

skip-build provides that functionality

I'm sorry for the misleading title of this post. I'm not saying that we need to keep the existing options working the same way. Please break everything about the build-script! I'm saying that before anyone thinks about rearchitecting the build, let's clarify the major problems and critical use cases:

  • Separate configuration and build steps.

  • Completely separate product configuration (clang, swift, swift libs, swift tools).

  • Many-to-many relationship between product configurations
    (e.g. release clang with debug swift or release swift).

  • Ability to customize configuration of a product without rebuilding anything else.

  • Ability to build additional targets and platforms without reconfiguring clang or swift.