History of build-script, build-script-impl, and CMake?

Development Compiler
1

Hello all,

I'm hoping a long-time Swift contributor can help shed some light on
how the apple/swift repository ended up with a utils/build-script, a
utils/build-script-impl, and CMake files.

Based on the Git history, I've pieced together the following:

- July 17, 2010: Chris Lattner first adds a Makefile to apple/swift in
his initial checkin. Until its eventual removal in 2014, the Makefile
only ever built apple/swift, and it required users to have built LLVM
in advance. [1].
- July 14, 2011: Doug Gregor adds CMake files to build apple/swift [2].
- February 6, 2013: Joe Groff adds utils/buildbot-script.sh. This
appears very similar to the modern-day uitls/build-script-impl. It
configures and builds LLVM, Clang, and Swift [3].
- February 25, 2014: Ted Kremenek removes the original Makefile Chris
added in 2010. I believe only CMake-based builds became possible as of
this point [4].
- December 23, 2014: Dmitri Gribenko (now no longer working on the
project at Apple) rewrites a large portion of the CMake files, and
deletes utils/buildbot-script.sh [5].
- October 30, 2015: Ashley Gardland "adds back" a bunch of build
files, including utils/build-script and utils/build-script-impl [6].

Could someone familiar with the apple/swift project during the years
2010 until 2015 shed some light on how the project was built during
this time? Specifically:

- Did most developers use the Makefile? What replaced the Makefile?
Did developers configure and build LLVM and Clang themselves,
manually, without using a script?
- Was the utils/buildbot-script.sh the precursor for
utils/build-script and utils/build-script-impl?
- What were the files that were "added back" in [6]? Did they exist in
some form internally to Apple? Were they used by project developers at
the time, or were they written solely for the benefit of external,
open-source contributors?
- Why do both utils/build-script and utils/build-script-impl exist?
The fact that they were "added back" makes me think that they grew
over time, and so were not written for the first time as part of the
commit in [6].

Thanks in advance to anyone who can indulge my curiosity here! :)

- Brian Gesiak

[1] initial checkin, nothing much to see here. · apple/swift@afc81c1 · GitHub
[2] Introduce a CMake build system for Swift. · apple/swift@86ab76d · GitHub
[3] Add a buildbot script. · apple/swift@4828269 · GitHub
[4] Remove root Makefile of Makefile build. · apple/swift@f383eb9 · GitHub
[5] Rewrite the CMake build system · apple/swift@6670bb7 · GitHub
[6] Add back remaining files for building and testing in Open Source · apple/swift@4ac9c80 · GitHub

2

Heh. Let's see:

Hello all,

I'm hoping a long-time Swift contributor can help shed some light on
how the apple/swift repository ended up with a utils/build-script, a
utils/build-script-impl, and CMake files.

Based on the Git history, I've pieced together the following:

- July 17, 2010: Chris Lattner first adds a Makefile to apple/swift in
his initial checkin. Until its eventual removal in 2014, the Makefile
only ever built apple/swift, and it required users to have built LLVM
in advance. [1].

1. Makefiles are easy, but…

- July 14, 2011: Doug Gregor adds CMake files to build apple/swift [2].

2. …LLVM standardized on CMake. (And this gave us both Ninja and Xcode support.)

- February 6, 2013: Joe Groff adds utils/buildbot-script.sh. This
appears very similar to the modern-day uitls/build-script-impl. It
configures and builds LLVM, Clang, and Swift [3].

3. This one really was for buildbots to have a standard configuration, but it's possible people were using it locally as well. There were also a lot of homegrown scripts at this time, and then people like me just used CMake directly.

- February 25, 2014: Ted Kremenek removes the original Makefile Chris
added in 2010. I believe only CMake-based builds became possible as of
this point [4].

4. Yeah, nobody cared by this point.

- December 23, 2014: Dmitri Gribenko (now no longer working on the
project at Apple) rewrites a large portion of the CMake files, and
deletes utils/buildbot-script.sh [5].
- October 30, 2015: Ashley Gardland "adds back" a bunch of build
files, including utils/build-script and utils/build-script-impl [6].

5/6. This is actually a misdirection! [5] is when build-script appeared, as part of a big push to get everyone building using the same tool. All the buildbots had their presets recorded in the build-presets file, which was the biggest win: if your own configuration didn't reproduce something, you didn't have to crawl int Jenkins to figure out what was different. And the Python was, in theory, much more maintainable than the increasingly-esoteric shell code used in build-script-impl (formerly buildbot-script.sh). That's why we had the task to migrate everything into Python, even though we never quite managed it.

The trouble was that we also had a bunch of Apple-internal configuration mixed in here. Rather than try to separate out the Apple and non-Apple parts in the commit log, we just removed it from the SVN history, took out the Apple parts, and then stuck in back in wholesale. That's [6].

(also, "Garland" ;-) )

So your guesses are all pretty much on the nose. The other thing that's cemented build-script and build-script-impl, though, is that they handle building multiple projects, even those with non-LLVMish build systems. (There's long been an advantage for Xcode users to build LLVM/Clang and Swift separately, but for things like swiftpm and swift-corelibs-foundation we'd never have a chance. And even LLDB is only just recently building like the other LLVM projects, if I remember correctly.)

Jordan

···

On Sep 8, 2017, at 08:14, Brian Gesiak via swift-dev <swift-dev@swift.org> wrote:

Could someone familiar with the apple/swift project during the years
2010 until 2015 shed some light on how the project was built during
this time? Specifically:

- Did most developers use the Makefile? What replaced the Makefile?
Did developers configure and build LLVM and Clang themselves,
manually, without using a script?
- Was the utils/buildbot-script.sh the precursor for
utils/build-script and utils/build-script-impl?
- What were the files that were "added back" in [6]? Did they exist in
some form internally to Apple? Were they used by project developers at
the time, or were they written solely for the benefit of external,
open-source contributors?
- Why do both utils/build-script and utils/build-script-impl exist?
The fact that they were "added back" makes me think that they grew
over time, and so were not written for the first time as part of the
commit in [6].

Thanks in advance to anyone who can indulge my curiosity here! :)

- Brian Gesiak

[1] initial checkin, nothing much to see here. · apple/swift@afc81c1 · GitHub
[2] Introduce a CMake build system for Swift. · apple/swift@86ab76d · GitHub
[3] Add a buildbot script. · apple/swift@4828269 · GitHub
[4] Remove root Makefile of Makefile build. · apple/swift@f383eb9 · GitHub
[5] Rewrite the CMake build system · apple/swift@6670bb7 · GitHub
[6] Add back remaining files for building and testing in Open Source · apple/swift@4ac9c80 · GitHub
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

3

Oops, sent this off-list accidentally.

···

---------- Forwarded message ----------
From: Brian Gesiak <modocache@gmail.com>
Date: Fri, Sep 8, 2017 at 3:38 PM
Subject: Re: [swift-dev] History of build-script, build-script-impl, and CMake?
To: Jordan Rose <jordan_rose@apple.com>

Great, thanks for the detailed response, Jordan!

On Fri, Sep 8, 2017 at 12:58 PM, Jordan Rose <jordan_rose@apple.com> wrote:

5/6. This is actually a misdirection! [5] is when build-script appeared, as
part of a big push to get everyone building using the same tool. All the
buildbots had their presets recorded in the build-presets file, which was
the biggest win: if your own configuration didn't reproduce something, you
didn't have to crawl int Jenkins to figure out what was different. And the
Python was, in theory, much more maintainable than the increasingly-esoteric
shell code used in build-script-impl (formerly buildbot-script.sh). That's
why we had the task to migrate everything into Python, even though we never
quite managed it.

The trouble was that we also had a bunch of Apple-internal configuration
mixed in here. Rather than try to separate out the Apple and non-Apple parts
in the commit log, we just removed it from the SVN history, took out the
Apple parts, and then stuck in back in wholesale. That's [6].

Ah, I see! So that's why [5] included the utils/SwiftBuildSupport.py
Python helpers, but not the Python utils/build-script itself. Thanks
for the info!

(also, "Garland" ;-) )

Oops! Sorry, I knew this in my head, but my hands mistyped!

So your guesses are all pretty much on the nose. The other thing that's
cemented build-script and build-script-impl, though, is that they handle
building multiple projects, even those with non-LLVMish build systems.
(There's long been an advantage for Xcode users to build LLVM/Clang and
Swift separately, but for things like swiftpm and swift-corelibs-foundation
we'd never have a chance. And even LLDB is only just recently building like
the other LLVM projects, if I remember correctly.)

This is a good point, thanks. It sounds like the goals are:

1. To eliminate the shellscript utils/build-script-impl, since it
provides no benefit over 100% Python utils/build-script.
2. To migrate as many of the non-CMake builds over to CMake. Thanks to
CMake support having been added to corelibs-libdispatch [1], I guess
corelibs-foundation could be built with CMake as well...?

- Brian Gesiak

[1] https://github.com/apple/swift-corelibs-libdispatch/pull/196