Survey: How do you build Swift?

Development Standard Library
,
1

I'm preparing a talk for potential new contributors to Swift and I found myself wondering: What build configurations does everyone use? If you'd like to describe your workflow, post something like what I've put below the line in this post. Your answer might influence my talk!

(This post is in the Standard Library section because it has to be somewhere, but it really applies to everyone who builds with the apple/swift repo.)

I mostly work on the Swift compiler frontend and sometimes on the standard library. I typically run tests with a Ninja-RelWithDebInfo build and edit and debug with an Xcode-Debug build. In a fresh checkout, I'll usually run:

utils/build-script --debug --xcode -S    # Generate Xcode projects so I can start reading code
open ../build/Xcode-Debug/swift-macosx-x86_64/Swift.xcodeproj
utils/build-script --release-debuginfo --test
utils/build-script --debug --xcode

(I have shell aliases for most of these commands, so I only have to remember the -S and the --test.)

I don't often test performance, but when I do, I add this build to the list:

utils/build-script --release --no-assertions
8 Likes
2

I don't, because it seems too overwhelming :|

Then again, it's probably best for the project to have a barrier for entry that filters out juniors

2 Likes
3

We really don't want there to be a barrier to entry. There's a lot that almost anyone can do!

Is there anything in particular that makes it seem overwhelming?

5 Likes
4

I use build-script for new builds, or after doing a git-pull.

build-script -r -t  # most of the time
build-script        # something has gone terribly wrong

For incremental builds I am usually doing something more like

ninja swift sourcekitd swift-ide-test
lit.py -sv test-macosx-x86_64/{IDE,SourceKit}
1 Like
5

I almost always use build-script --release-debuginfo --debug-swift-stdlib even for incremental builds. Very occasionally I invoke ninja or lit directly when going through build-script gets too troublesome.

6

I use build-script for high level builds. I will iterate in the build directory while debugging though (i.e. I have a test file that is isolated and I can just use ninja in the build directory to tweak, test, recompile).

The build-script commands I use are:

  • For working on SILGen and the diagnostic passes:
--release-debuginfo --assertions --debug-swift --debug-swift-stdlib --force-optimized-typechecker
  • For working on the SILOptimizer:
--release-debuginfo --assertions --debug-swift
  • For benchmarking swift:
--release-debuginfo --assertions --no-swift-stdlib-assertions

I want the compiler to still have assertions when I am benchmarking, just not the stdlib.

7

Sorry I hit enter too quick.

I also when working on the SILOptimizer will sometimes compile:

--release-debuginfo --assertions --debug-swift

AND

--release-debuginfo --assertions

and then use the just built debug swift to debug failures in the non-debug swift. This makes it quicker to iterate.

8

I think that I may be the odd person. I work on pretty much every variant (macOS, Linux, and Windows). I also cut across the compiler, standard library, and the core libraries. I work with two different ways:

For Linux, I pretty much just do a debug everything build and for Windows a release everything build.

9

I primarily use build-script -d -t --skip-build-benchmarks with a healthy dose of manually running ninja check-swift or ninja swift in the Swift build directory. I used to maintain an LLVM-style checkout and build, but eventually gave up.

1 Like
10

@John_McCall - mind explaining how come you gave up on the LLVM style build? That is what I was describing as the unified build (and the way that my setup is). I'm wondering if there is something fundamentally bad about it or if something which is broken that I haven't noticed.

11

TL;DR: The very idea of pinning down a specific configuration and set of targets before you start working is wrong.

For any of you who care to see how convoluted my workflow is, here you go...

I try not to use build-script for anything except an initial CMake invocation:

build-script --release --llvm-assertions --swift-assertions --no-swift-stdlib-assertions --build-subdir=ra --skip-build -- --install-prefix=/usr --toolchain-prefix=/

It's very commmon for people to forget this flag: --no-swift-stdlib-assertions, and assume they meaningfully measure run time.

Initial build for basic testing:
$ ninja

Occasionally I need to build for iOS on top of the existing build:

$ ninja swift-stdlib-iphoneos-arm64

I often need to build the debug compiler on top of the existing build:

  • build-script --release --llvm-assertions --debug-swift --swift-assertions --no-swift-stdlib-assertions --build-subdir=d --skip-build

  • "Manually" copy the llvm release build directory into the debug llvm build dir.

  • Build the swift compiler:
    $ ninja swift sil-opt

  • "Manually" copy stdlib and overlays from the Swift release build into the debug build dir.

I often need to build benchmarks on top of the existing build:

$ cmake <swift-source>/benchmark -G Ninja -DSWIFT_EXEC=<swift-build>/bin/swiftc -DONLY_PLATFORMS=macosx
$ ninja swift-benchmark-macosx-x86_64
  • "Manually" copy swift dylibs into the benchmark directory

I sometimes need build other toolchain components. I usually can't get that to work without rerunning the build-script, so I reinvoke it, making sure the options are exactly the same, omitting '--skip-build', adding a ton of options:

$ build-script --release --llvm-assertions --swift-assertions --no-swift-stdlib-assertions --build-subdir=ra --ios --tvos --watchos --skip-build-ios-simulator --skip-build-tvos-simulator --skip-build-watchos-simulator --skip-build-benchmarks --llbuild --swiftpm --libcxx --compiler-vendor = apple -- --install-prefix=/usr --toolchain-prefix=/ --install-swift --install-llbuild --install-swiftpm --install-destdir=<dir> --swift-install-components=compiler;clang-resource-dir-symlink;stdlib;sdk-overlay;parser-lib;license;sourcekit-xpc-service;swift-remote-mirror;swift-remote-mirror-headers --llvm-install-components=llvm-cov;llvm-profdata;IndexStore;clang;clang-headers;compiler-rt;clangd <define a bunch of variables>

For some reason, I also need to add a bunch of "darwin-toolchain" variable definitions at the end of the build-script invocation for it to produce the toolchain.

It drives me crazy that our build script is a monolithic mess with no clean separation between targets, actions, and configuration. We should be able to incrementally build for additional targets/platforms and build products without starting over!

3 Likes
12

It was always broken in some way and I was tired of working around it or trying to fix it. For example, for some reason it can't build SourceKit correctly, and I could never figure out why because XPC services plus our build system plus lit is just way too much magic at once, so I spent months just ignoring 40-50 test failures every time I ran ninja check-swift. I think that might have eventually been the breaking point: I made a change that actually required fixes to one of the SourceKit tests, and the only way I was ever going to get that to work was to switch styles so that I could actually run it.

13

As an outsider, I don't have a lot experience with build-script tips and tricks, but I think I do a fairly basic build of swift following the README. utils/build-script --release-debuginfo to make the initial build folder and when I make changes I'm always inside the build folder and running ninja swift to build compiler changes that I make and ninja swift-stdlib if I make any stdlib changes. I made the environment variable ${SWIFT_BUILD} to cd into from the README, but for testing I also made variables called ${SWIFT_TEST} and ${SWIFT_VALIDATION_TEST} that I run from the home directory that invoke lit directly.
E.g. ~: ${SWIFT_TEST} --filter=SILGen/stored_property_default_arg.swift

14

step 1: see if i have it cloned already on my computer
step 2: i do,, i dont remember what i've done to it though, so better reclone just to be safe
step 3: look for a script called update-checkout and run it
step 4: --clone
step 5: do something else while llvm downloads
step 6: utils/build-script --preset=ubuntu1804
step 6 ½: cd swift
step 6 ¾: utils/build-script --preset=ubuntu1804
step 7: google “swift utils/build-script preset ubuntu 18.04”
step 8: utils/build-script --preset=buildbot_linux_1804
step 9: utils/build-script --preset=buildbot_linux_1804 --install_destdir=~/swift-install
step 10: utils/build-script -h
step 11: utils/build-script --preset=buildbot_linux_1804 install_destdir=~/swift-install
step 12: utils/build-script --preset=buildbot_linux_1804 install_destdir=~/swift-install installable_package=~/swift.tar.gz
step 13: do something else while swift builds
step 14: empty Trash and delete Downloads folder and swift build folder
step 15: utils/build-script --preset=buildbot_linux_1804 install_destdir=~/swift-install installable_package=~/swift.tar.gz
step 16: do something else while swift builds
step 17: look for laptop charger
step 18: do something else while swift builds
step 19: profit

7 Likes
15

Ah, I see. I think that the thing that is different in my case is that I am directly using CMake and Ninja rather than going through the build-script so I do not have the same comparison as you. But it is something which has worked stably for me for a while now.

16

I'm mostly hacking on Sema these days and I prefer "unified builds" with just the core projects extract more build time concurrency between the projects on my Linux workstations. When a regression occurs in something downstream (that isn't self diagnosing or self evident), I use build-script but that is uncommon.

17

Yup. Let's not also forget that build-script needlessly reimplements logic found in the CMakeLists.txt files, and not surprisingly, this duplicated code diverges in surprising ways if you don't use build-script. :-(

18

We are working slowly towards getting us out of this mess. I think that eventually we can get to a world where we have a top level build-script that generates cmake caches that are passed to an underlying top level cmake that drives the build.

The recent work to always build a toolchain and to then have a chain of other builds based off the toolchain leads us into that direction. But it is going to take time and we shouldn't let perfection be the enemy of the good.

2 Likes
19

Hi @Michael_Gottesman – Glad to learn that the build-script is being worked on. Will unified builds keep working?

20

I am not maintaining it. If someone else wants to maintain it, I don't have an issue with that.

1 Like