Are Swift CI bot and Swift Jenkins configuration open source? (Also, let's discuss Swift toolchain developer experience)

Development Compiler
1

Does anyone know if the source code for Swift CI bot (the one that responds to @swift-ci please test requests) and general Jenkins configuration available for anyone to deploy locally or within their own organization?

Now before you ask "why would anybody need that?" let me describe the situation that we're in here with the port of Swift for WebAssembly (and I hope that people from the community working on other ports could relate).

Swift toolchain/SDK builds are huge and take a lot of time and disk space and need powerful hardware. It's not something that makes working on it locally a thrilling experience. This is caused by multiple issues that I don't see highlighted frequently (if at all) or actively worked on. I hope many iOS developers are familiar with long build times. Couple of years ago I started working on a huge iOS project in a big team, and cold builds of 3-5 minutes and incremental builds of 30-60 seconds were quite annoying to put it mildly. I now understand, that kind of developer experience was actually good when you compare it to something else here

I wanted to avoid strong words, but I don't know what's the better way to bring people's attention to this. In one of the previous episodes of ATP Marco Arment described working on Apple Watch apps as "moving through maple syrup that occasionally catches fire and burns you as you're in it [...], it's time-consuming and morale-sucking" because of how terrible and flaky the developer experience is. In my experience working on the Swift toolchain and SDK is like moving through a concrete sludge that's constantly on fire. A cold build takes at least half an hour on the top of the line MacBook Pro, and incremental builds frequently fail due to obscure reasons, in those cases it's easier to nuke the build directory and run a cold build again, just to verify if the build failure is an actual failure not caused by incremental builds. Yes, you can use caching, but after spending so much time before I discovered that tip, I think it should be at the top of README.md, not hidden in the documentation file. A cold Linux build in Docker takes closer to an hour and setting it up so that the cache is not lost after exiting the container takes some time.

Don't get me wrong, I love the language, I love the community, but I feel like I'm contributing not because I enjoy the development process, but despite the way it's done. Just as a side note, by this point I think I had a dozen people email me asking how they can help with porting Swift to WebAssembly. I've sent back detailed build instructions with links to our repository and a list of known issues. Now I regret I didn't follow up with them, because the amount of hand-holding to make things right when working on the Swift toolchain is huge. Only two of them have actually contributed. And I realize I'm privileged by having a top of the line MacBook Pro, can you imagine people trying this on something like a MacBook Air? I'm not even talking about non-Apple hardware here.

Anyway, the main issue I've stumbled upon is setting CI up on GitHub Actions in our SwiftWasm fork. The problem is that GitHub runners are restricted in time (5 hours, but in reality I saw runners killed just after 3-4 hours) and disk space (18 gigs). An average build there takes about 2-4 hours without caching, and 1 hour for macOS, 2-3 hours on Linux with caching. We're pretty close to reaching the limits (even with release builds). We can't use self-hosted runnners because it's a public project, so I was thinking we could reuse some of the infrastructure from upstream? I wasn't able to find any documentation describing how Jenkins is set up and how it interacts with the Swift CI bot.

Given the overall development experience, I think documenting the CI setup is the least that could be done to make it easier for community contributors. Making the local development experience pleasant when developing locally is probably a whole another topic (and we had a few already here on Forums).

5 Likes
2

Describing the situation, I also think there should be something that we aspire to. Have a look at how the Rust people have it with a nicely formatted and searchable rustc guide.

I don't know if there's a causation, but consider that the rust toolchain has 2,911 contributors at the moment, where the top 100 of them have at least 123 commits. The main Swift repository has only 799 contributors with the top 100 having at least 34 commits. This is a totally unscientific comparison, we could say that Rust people prefer smaller more numerous commits, or Swift contributors do more contributions to Foundation/Dispatch/SwiftPM (although those have 300 contributors each at most), but I still think that the developer experience (and the fact that Swift toolchain is written in C++) doesn't help.

1 Like
3

I have been thinking about this a bit, since I've been building and rebuilding a lot while I'm stuck in the house. ci.swift.org is open, you can click through to it and see the build settings and, I think, the Jenkins configuration. The builds take a while on the CI as well.

The build settings on the CI use sccache. I have noticed locally that sccache does not match the c++ lines in the build for me right now--it reads them incorrectly as multiple output files. The sccache on the CI might handle that differently or I might need to do something to my sccache locally to fix that. I just started using ccache instead which worked OK.

But if you're iterating a lot on anything fundamental you still get big builds because you will touch a deep dependency. The cache does not really help much with that. I'm not sure how (practically) to reorganize things to address that. I have been thinking about it a bit in the background. A couple of thoughts that seem impractical:

  1. A cold build builds llvm and the embedded clang. If we could get around that by using the llvm and clang in the toolchain that would speed that up a lot. I assume this is not possible for a bunch of reasons (including that a lot of fixes involve changes to llvm and clang source so you need to edit and build them) but it is something to think about.

  2. Some refactoring that isolated the components from each other would cut down on the dependencies between modules. If the AST, say, were an isolated module that just exposed a stable API to the earlier and later stages, you could (supposedly) work in the AST without forcing a recompile of the rest of the project. But there are obvious dependencies, because everything depends on the AST (or any other module) and the performance in the field might be worse because of marshaling everything through some API.

  3. If we replaced the clang code generator with swift-specific stuff we might get faster builds because we'd have only code relevant to Swift. But it would also introduce a new module, etc. You could make a similar argument for a lot of other pieces of llvm that the project uses. You might add more code in one place in exchange for less code in another place, and not get build- or runtime improvements. And obviously we're using llvm and clang because they help and work.

I think all of 1,2,3 above are sort of obvious things to think about that don't lead to actions, but if other people have more focused experiences it might be good to discuss. Isolating dependencies sounds good but the components do actually depend on each other so can't be isolated that much, you know what I'm saying.

Hope that helps,

Dan

1 Like
4

Yes, using upstream LLVM/Clang would help a lot, because rebuilding those from scratch takes close to half of the total build time. I also don't think that Swift is as close (or moves in that direction at all) to this policy that Rust has:

Despite this being a "fork" it's more of just a place where we can easily tweak the LLVM source for Rust's own build. All patches should by default go upstream first, and then we can cherry-pick them onto our branches if necessary.

So they still pull llvm-project as a submodule, but I guess at a given moment, especially for older versions of the Rust compiler you're more likely to take upstream precompiled LLVM/Clang and build the Rust compiler successfully without a need to build a forked version. At the same time Swift's fork of llvm-project has diverged so much, is there a plan to make all those changes available in upstream LLVM at all?

5

Hey @Max_Desiatov,

This is something that I've personally given a lot of thought and consideration to. In my opinion, the better approach here is more modularity. This is in fact, how I have setup nightlies for android and was how I was approaching the WebAssembly work (but got distracted with a bunch of other work). The idea is that you do not build everything, but rather you cross-compile.

The toolchain can be built, cached, and reused for an extended period of time (days). The main thing to test for many ports is what I will call the "standard library" (yes, I know that it is more than just the standard library, but the directory is called stdlib). You can build just the standard library on a more reasonable host (Windows, Linux, macOS). This drastically speeds up the build (~1h on a 2 core machine with ~8G of RAM for the standard library, libdispatch, Foundation, XCTest).

I agree that this is not in the ideal state to enable developers to easily do this operation, but is is possible with some amount of contortion.

This directly relates to my recommendations that I've previously given to @kateinoigakukun - you should get involved in the work to clean up the build system. Relying on CMake's inherent cross-compilation model would allow us to ease this and allow for a much more pleasant experience.

1 Like
6

@compnerd Thank you so much for writing this up! This is something that I had in mind but wasn't able to approach in a good way due to the amount of other work required to make the WebAssembly port fully functional. It's very hard to strike a balance between "get something working ASAP" and "merge upstream to avoid maintaining forks". The latter is especially hard because it requires a lot of effort to address the review feedback. And the effort is fully justified, but balancing this is very hard wtih limited resources. On the other hand, it's obvious that any improvement to the developer experience will pay off tenfold, so I should focus on that much more.

Do you think that the LLVM/Clang toolchain can also be built separately? I've tried removing llvm-project sources after those are built to get some space cleaned during a CI run, but seems like a lot of stuff from the swift sources directory assumes that headers exist in the llvm-project sibling directory. I'd expect it to look for public headers in the install directory, or at least in the build directory like build/Ninja-ReleaseAssert/llvm-macosx-x86_64. Do you think this behaviour can/should be cleaned up?

With regards to cleaning up the buld system, @kateinoigakukun has spent a substantial amount of work on it as described in the "Separate CMake process into compiler and stdlib" post. It seems like that is blocked by some other pending work with no clear (or at least not public) timeline. How would you recommend to approach it in this situation?

7

No, I don't think that it is viable to build this separately. The entire toolchain needs to be built as a single unit (which actually is faster than using build-script in my experience). You can build llvm, clang, lld, lldb, and swiftc as a single unit. This is the entire toolchain.

I don't think it is a matter of "cleaning up" here. Swift uses LLVM, plain and simple. That means that you need the headers and static libraries since it is part of the build.

Alternatively, you could spend a significant amount of effort modularizing the build so that you build only LLVM, only clang, and only swiftc as individual steps. Keep in mind that this is going to be a much slower build in general, but it is should be possible - LLVM and clang can already be built individually, and build-script does build swift and lldb separately. Were you to do so, you would still need roughly the entire build tree of LLVM and clang in order to build swift.

I think that it might be more valuable to see what the majority of the time is going to and trying to see if we can do better here. For example, my experiments show a ~10% reduction in build times by using MSVC rather than clang for the build and ~10-15% speed up from doing the unified build.

Yes, @Michael_Gottesman had some initial work completed that got us closer to this world. I think that the current state of that is best explained by @Michael_Gottesman, so I wont speak to the current state. It might be useful to just sync up and see how you could help get that work completed and merged into the tree perhaps.

8

Thanks for the clarification. I've asked this before, but I don't think this was completely described anywhere: how do you do make a unified build? Is this something that upstream Swift supports out of the box? Or does this require a separate bunch of CMake files that aren't available in Apple's git repositories? Does your swift-build repository provide that? I've looked at swift-build multiple times and it seemed to be very Windows-specific, is there anything there that could help in producing a unified build, or maybe something that could be reused in the WebAssembly port?

9

The easiest way is to change the layout of the checkout so that you have swift located at llvm-project/tools instead of a peer of llvm-project, along with cmark. Then you just use the LLVM_ENABLE_PROJECTS and LLVM_EXTERNAL_PROJECTS to build it as a single unified build.

Yes, as long as by out of the box you don't use build-script or update-checkout.

Yes, all the configuration information and examples are available in that repository.

That seems odd - there is a bunch of configuration in for Linux and macOS builds as well as android builds. There are nightly builds of the android runtime that are configured in that repository.

Yes, the CI configuration it has is setup for the unified build

Yes, I would recommend that you take a look at the android SDK builds - that should be pretty much exactly what you want for the WebAssembly builds.

I nearly had the setup for that complete but got stuck because of the WASI port being unfriendly to cross-compiling on the environment that I had most accessible. If you can help get WebAssembly/wasi-libc#154 merged upstream, I could probably finish up the setup that I was working on for nighly builds of the standard library.

1 Like