Llbuild2

Hi all,

Almost 4 1/2 years ago, we open sourced llbuild alongside Swift and the Swift package manager. At the time, llbuild was still fairly new, but it's design has largely stood the test of time, powering SwiftPM builds as well as other products like Swift playgrounds and Xcode's XCBuild build system. And in that time, llbuild has driven a lot of builds!

In the meantime, a lot of other things have changed in the world. Swift's ecosystem has grown considerably, including substantial packages like Swift NIO giving us far easier access to sophisticated network technology from Swift. The world outside of Swift has changed too, with Bazel's Remote Execution API becoming something of a de facto standard with many backend implementations (like buildbarn, buildfarm, BuildGrid, and scoot) and multiple frontends (like Bazel, BuildStream, and recc). We've been pondering on how best to make forward progress on all the other things
we'd like to do with llbuild, while also taking into account some of the existing layering challenges in llbuild.

About a month ago, @David_M_Bryson started playing around with a concept of doing a new implementation of llbuild's API, but sitting atop NIO's Futures. We've since written a number of experiments atop it which make us think this is now the direction we'd like to go for a more modern API. At the same time, we've been having a lot of discussions with @Aciid and @sergiocampama about how we should go about supporting Swift "explicit modules" in the build system. One thing that quickly emerged in those discussions is there is an opportunity to get much more alignment between llbuild, SwiftPM, swift-driver, and the Swift compiler.

With that in mind, we are going to try something new... llbuild2! This an experiment, we don't yet know exactly how it will pan out, but our intention is to try it for a bit and see where it takes us. If we like it, we will most likely fold it back into the primary llbuild repository as the future direction for its API. Here is what we are planning:

  • We've created a new swift-llbuild2 repository to hold this work. This will be a pure Swift package for the time being (no CMake, and not worrying about bootstrap concerns yet).
    • We will most likely use the existing llbuild as a dependency, and for sinking performance critical code into C++ (where necessary).
    • We've moved the cevobuild experiment out into llbuild2 , and will begin to build up more, better, higher-level build system layers using the lessons we've learned.
    • Unlike llbuild, which tried very hard to have minimal dependencies, we can be much more open on taking on Swift package dependencies to let us build up functionality faster or in a more maintainable way.
    • In particular, we'd like to adopt Bazel's Remote Execution API so we can begin supporting some forms of automatic distributed builds on top of llbuild2 . We also intend to adoptgRPC-Swiftfor RPC support, when useful inother contexts.
    • We will also be creating a swiftpm-on-llbuild2 repository (wow, this is a mouthful), which will extend the existing swift-package-manager to use llbuild2 for its build system.
    • Our intention is to use that repository to develop SwiftPM-specific build rules on top of the new llbuild2 model, including work to develop the explicit modules support in swift-driver to make that performant. (https://github.com/apple/swift/pull/28515)
      • When combined with llbuild2 support for Bazel's Remote Execution API, this should allow us to start exploring fully distributed/cached builds for SwiftPM packages (when a suitable backend is available).
      • As with llbuild2, if those experiments work out, we intend to fold that back into the primary repository.

We are pretty excited about this new direction... if you are interested in contributing, come join us in #llbuild-dev on the SwiftPM slack.

Cheers,

  • Daniel
57 Likes

looks amazing

1 Like

@ddunbar I'm not familiar with the term "explicit modules", can you define it for us? thanks

1 Like

The concept "explicit modules" is an idea to get off of the implicit module cache systems used by Clang & Swift, and instead do something based off of https://github.com/apple/swift/pull/28515 and related work, so that the build system could end up understanding how to actually build the modules.

I believe @Douglas_Gregor intends to write up a clearer proposal around what some of his thinking is there, so I'm going to :crossed_fingers: for that.

6 Likes

This sounds great! I really hope the experiment with llbuild2 works out and we get a build system that can support cached remote artifacts and distributed builds. Will you also move over to use file signatures instead of time stamps @ddunbar?

3 Likes

@FranzBusch we consider digest based approaches for change detection very much a core feature in modern build system infrastructure, so yes, we are thinking on the invalidation logic being centered around the contents of the sources, not on the timestamps

1 Like

Yes, we plan to introduce a Content Addressable Storage protocol and to have the keys become the DataIDs that identify the content. That way everything is modeled as a hash of its content or description.

5 Likes

This is super exciting, congratulations - I can't wait to see where this goes!

5 Likes

@Dave_Lee in case you missed it, Doug’s write up is here: Explicit Module Builds, the new Swift Driver, and SwiftPM

2 Likes

Are you going to handle jobs with different levels of parallelism?
Assume clang jobs are single threaded while lld jobs use 8 threads. I would assume that ninja is unaware of that problem and overloads the machine.

This is somewhat unrelated to llbuild2 as a specific project... it is a problem we are aware of and would like to eventually have better solutions for (in both llbuild & llbuild2).

Terms of Service

Privacy Policy

Cookie Policy