What does Swift support in CMake mean for Swift's build?

Development Core Libraries
, , ,
1

On the heels of the recent announcement about Swift support in CMake, I would like to bring up a topic of discussion which I think may take a bit to resolve. I would like to bring this topic up and see if we can come to a decision on what is the best thing to do for the project. The crux of the question is: is it viable to update CMake at a more aggressive pace than the project traditionally has?

CMake gaining proper support for Swift as a language gives a strong motivation for bumping the minimum required CMake for the Swift project. It will enable a very large simplification in the build rules, allowing us to delete hundreds of lines of CMake code. Additionally, we can adopt the general best practices for CMake usage and more modern syntax and idioms. The proper language support enables simpler dependency tracking and also enables the breath of the knowledge on the Internet about CMake behaviours to be applicable to the Swift build as well. This should also enable others to more easily delve into the build system which has largely been a sore point and arguably one of the most complicated build systems. Additionally, this will enable us to create export targets for the core libraries (much like clang/LLVM) and use the libraries from the build trees with CMake handling all the flags and location as well as easing the re-use of the libraries across projects from their respective build trees.

I think that the biggest downside of the upgrade is the fact that we would diverge from clang/LLVM and require a newer CMake than the dependencies. This means that we have to consider what the different environments offer in terms of CMake. On Windows, it seems that the upgrade cycle is actually the easiest - Visual Studio has done an excellent job of keeping up with CMake, and is currently shipping 3.14. The story on macOS is much more interesting, where the package is not available as part of the system or system development tools. However, there are DMGs available from KitWare pretty quickly, so I think that it shouldn't be too bad on macOS either. The one platform where I think that this may be slightly more challenging. Some distributions have the ability to update quickly while others lag behind. It is always possible to build CMake, but that may be additional friction.

Overall, I think that the benefits do outweigh the drawbacks, and that we should upgrade, but that is one person's opinion. I have a couple of patches up as drafts to start exploring what such a world would look like for the core libraries:

These are both the simpler builds that we have that are using Swift, and in both cases, we managed to remove about 300 lines of CMake in each repository. I think that the biggest gain here will be actually in an attempt to port the standard library to this mechanism.

Note that even if we decide to go ahead and bump the CMake version, we should probably do so cautiously optimistically with the 3.15.1 release. But, we should figure out what the change will be so that we can address any issues that come up as we start pushing more complicated real world workloads on the new support.

CC: @Michael_Gottesman @Rostepher @mishal_shah @graskind @jrose

2

For macOS, homebrew (and macports?) is probably the right baseline. Luckily they update very quickly (currently 3.14.4).

I think the friction of building cmake from source or even installing binaries directly is too high. We might do this as a fallback for an older supported platform that we don't expect many developers to be using, but I don't think it should be the common case.

For Linux, we are testing on Ubuntu 16.04, which provides 3.5.1 and Ubuntu 18.04, which has 3.10.2 (I'm going to ignore 14.04). To minimize the barrier to entry, we could support one of those as our minimum cmake, depending on how many users we expect are coming from each platform.

Alternatively, I see that KitWare provides an apt repository that provides the latest cmake for both 16.04 and 18.04: https://apt.kitware.com. Speaking only for myself, adding new apt repositories is something I'm accustomed to doing for other software, so I would be fine with this. It does create a barrier to entry for a new developer or a new Ubuntu user, but it's a big improvement over installing the binaries directly.

One of the challenges is that the folks most likely to be negatively impacted by installing a non-distro cmake are potential/new/occasional developers who are least likely to be on this forum to provide feedback.

1 Like
3

Interesting, I didn't know that KitWare had an apt repository with the packages. That is certainly an interesting idea. The other option is to actually add CMake to the build-script for the set of people who do not wish to add the additional repository. We already do this for Ninja, so I don't think that it is too crazy to add CMake to the list as well.

I really don't like the idea of having two versions of the build system for backwards compatibility. That invariably leads to one or the other build to break as only one gets tested by the developer working on something.

4

Are you suggesting we require one particular version of cmake instead of setting a minimum?

5

Ninja's a lot smaller than CMake and also requires a lot less configuration. I'd be against building a local version of CMake as part of Swift's default build process.

1 Like
6

@jrose How long does it take to build cmake? Have we measured?

7

NOTE: I am not saying that we should do it. I am just saying that part of saying a lot smaller is that cmake is going to add a bunch of time to the build. If that is an issue, we should be explicit/specific in the trade-off.

8

@blangmuir - no, I am suggesting that we have an option to allow users to have it built as part of the build if the system does not have a new enough CMake available. This would be a temporary bandaid, not a long term solution.

2 Likes
9

@Michael_Gottesman - it depends on the configuration. On my quad-core i7 Ivy Bridge (yes, they are
discontinued due to being ancient) it takes about ~15m if you do a single thread build. It takes ~5m or so with -j4. These are timed on Windows, so they should be scaled down for environments with reasonable file systems.

10

I'm very much in favour of simplifying the build system. Less lines of configuration obscuring what is going on makes it easier for people to understand, which in turn makes it easier to improve and maintain.

Seems like it's all fine on OSX, and one extra step on Ubuntu 16/18. Ubuntu 14 is out of support anyway (when will we consider dropping it?). I don't see a problem TBH.

1 Like
11

Okay, so now we have the following PRs:

This is interesting because these set of PRs port the core libraries to the new Swift support. This is not entirely ready just yet - it needs additional testing to ensure that the files are being installed properly and into the correct location. Additionally, I think that a couple of adjustments may need to be made to the test invocations for minor layout changes.

While it would be interesting to try building the standard library with this support, I don't think that I want to jump into that right now. I expect that for the standard library the only thing which is beyond what has already been explored here is the GYB handling. The previous work that I did to extract the GYB_SOURCES parameter to add_target_swift_library serves as the separation point and should enable the transition.

This test at least demonstrates that this functionality is sufficiently complete to support the needs of real world projects. There were a couple of minor issues that came to light as a means of this experimentation with some minor workarounds that I had to employ (I've filed some issues in CMake's issue tracker for them).

1 Like
12

To be clear, it's not time I'm mainly worried about; it's complexity. Maybe I should pull back from that a little since, like Ninja, anyone could choose not to build it and instead use a local version if they need extra configuration. But still.

13

Yes, it would be optional (and recommended that you use a system version). As to complexity, it really is just invoking bootstrap (so that you can avoid the CMake dependency to build CMake).

14

I was thinking about this a little bit and I actually think that there are two additional motivating actions here we are not considering:

  1. We could use this to break the dependency of swift based tools (e.x. swift-syntax) that swift's cmake builds on building the stdlib.

  2. CMake for free will let us integrate swift code into swiftc itself trivially.

I go through both below since there are a few different implications of both.

Breaking the Dependency of Swift based tools on building the stdlib

By compiling these tools with the host swiftc, we break the dependency of tools on the just built stdlib. This will have a number of knock on effects:

  1. Build/test time will be reduce by allowing for these tools (and any large downstream work) to be compiled and tested without building the stdlib. This will massively reduce the time for building/testing such tools. Such timing/productivity improvements can result in the development of these tools to be more agile and save development time/money (insert xkcd sword fighting reference here).

  2. The last cmake dependency in between "tools" and the stdlib/runtime will be broken. This will finally let us split the stdlib cmake from the main swift cmake build and have swift's cmake invoke the stdlib cmake as a subproject. This would let us move swift's cmake to standard llvm cmake and leave the current crazy cmake badness just in the stdlib (for now).

One last thing to note is that with proper integration and ABI/Module stability we should be able to still install these tools into a snapshot and have them use the just built swift stdlib.

Using Swift Code in swiftc

The most exciting realization I had was that the newer cmake will make it easy for us to write parts of swift the compiler in swift:interrobang:. In fact, I was able to put together a proof of concept using the latest cmake nightly. If you want to try it out:

  1. Download the latest cmake nightly:
curl -LO https://github.com/Kitware/CMake/releases/download/v3.15.0-rc2/cmake-3.15.0-rc2-Darwin-x86_64.tar.gz
  1. Compile master with this branch from my fork: GitHub - gottesmm/swift at cmake-swift-support-proof-of-concept. You inject the cmake by passing it to build-script like so:
./swift/utils/build-script --release-debuginfo --assertions --distcc
--skip-build-benchmarks --build-subdir swift-cmake-asserts --cmake
$PATH_TO_TARBALL_CONTENTS/cmake-3.15.0-rc2-Darwin-x86_64/CMake.app/Contents/bin/cmake
--reconfigure

The branch injects into SILGen a small bit of swift code that every time swiftc compiles a function prints to stdout the representation of a local swift class and then additional prints to stdout "hello world".

One important caveat is that if we decide that we do not want to update to the newer cmake immediately, we can still write parts of swiftc in swift that would not be needed to bootstrap the compiler. As an elucidating example consider a SIL optimizer pass as a candidate:

  1. The interface by which a SIL optimizer pass exposes itself to the rest of the compiler is limited to a Transform class. We could wrap transform in a c struct that could be imported into swift. We can inject the transform back into swift by defining a swift @_cdecl public function with an appropriate name. Since all swift optimizer passes are defined in a .def file, we can be sure that the optimizer pass pipeline would be able to take in the unwrapped Transform class and inject it into the pass pipeline.

  2. SIL's instructions themselves are defined via a .def file. This means that we can generate swift c wrapper classes for the c++ instructions using the c preprocessor.

  3. Optimizer passes are optional, so if we are doing a bootstrap build of swiftc, we can just not enable those passes.

The only downside is that we have written a bunch of helpers for the optimizer in c++, so it may not make sense to do this before we have true c++ interop. But perhaps some simple passes/new subsystems of various passes, could be written in swift itself.

16 Likes
15

I'm not up to speed on cmake support and the con's of upgrading to a newer version, but this does seem like the right step to take at some point! I'd also love to see the use of swift form within the Swift compiler itself :-)

3 Likes
16

Can‘t wait for that day, I would love to dive into the compiler which would be written in Swift and finally contribute real implementation not just feedback on the forums.

1 Like
17

... Sorry hit enter too fast ...

1 Like
18

Sorry for the fat finger earlier (I was hoping to make this more of a surprise).

But yes, I have a proof of concept that has a SILOptimizer pass written in swift running in the swift compiler in the optimization pipeline. You can use it in sil-opt if you want just like a normal pass. It is available in this branch here:

My hope is that I was able to do the majority of the hard work in terms of the build-system/etc. Bridging new types (and adding attributes/etc/newtype) could be done by others who are not that familiar with c++ but are familiar with swift/objc interop. The nice thing about my design is adding a new SILOptimizer pass is trivial. This is the diff for adding the SILOptimizer pass:

commit d50d06f937d5c219af76e655d91f6cce84ca4eae
Author: Michael Gottesman <mgottesman@apple.com>
Date:   Wed Jun 26 17:58:34 2019 -0700

    [swift] Add the proof of concept ModuleDumper SILOptimizer pass.

diff --git a/include/swift/SILOptimizer/PassManager/Passes.def b/include/swift/SILOptimizer/PassManager/Passes.def
index e205aab571b..5622ca35ae6 100644
--- a/include/swift/SILOptimizer/PassManager/Passes.def
+++ b/include/swift/SILOptimizer/PassManager/Passes.def
@@ -318,6 +318,8 @@ PASS(SerializeSILPass, "serialize-sil",
      "Utility pass. Serializes the current SILModule")
 PASS(YieldOnceCheck, "yield-once-check",
     "Check correct usage of yields in yield-once coroutines")
+BRIDGED_PASS(ModuleDumper, "swift-module-dumper",
+    "Just a proof of concept pass that verifies and dumps a SILModule")
 PASS(OSLogOptimization, "os-log-optimization", "Optimize os log calls")
 PASS(BugReducerTester, "bug-reducer-tester",
      "sil-bug-reducer Tool Testing by Asserting on a Sentinel Function")
diff --git a/lib/SILOptimizer/PassManager/PassPipeline.cpp b/lib/SILOptimizer/PassManager/PassPipeline.cpp
index 60d0948d709..c80c2169ab2 100644
--- a/lib/SILOptimizer/PassManager/PassPipeline.cpp
+++ b/lib/SILOptimizer/PassManager/PassPipeline.cpp
@@ -405,6 +405,7 @@ static void addPerfEarlyModulePassPipeline(SILPassPipelinePlan &P) {
 
 static void addHighLevelEarlyLoopOptPipeline(SILPassPipelinePlan &P) {
   P.startPipeline("HighLevel+EarlyLoopOpt");
+  P.addModuleDumper();
   // FIXME: update this to be a function pass.
   P.addEagerSpecializer();
   addSSAPasses(P, OptimizationLevelKind::HighLevel);
diff --git a/lib/SILOptimizer/UtilityPasses/CMakeLists.txt b/lib/SILOptimizer/UtilityPasses/CMakeLists.txt
index 31887874dd4..7ca941e4cbe 100644
--- a/lib/SILOptimizer/UtilityPasses/CMakeLists.txt
+++ b/lib/SILOptimizer/UtilityPasses/CMakeLists.txt
@@ -30,3 +30,7 @@ silopt_register_sources(
   StripDebugInfo.cpp
   OwnershipDumper.cpp
 )
+
+silopt_register_swift_sources(
+  ModuleDumper.swift
+  )
diff --git a/lib/SILOptimizer/UtilityPasses/ModuleDumper.swift b/lib/SILOptimizer/UtilityPasses/ModuleDumper.swift
new file mode 100644
index 00000000000..858cd1a8a5d
--- /dev/null
+++ b/lib/SILOptimizer/UtilityPasses/ModuleDumper.swift
@@ -0,0 +1,12 @@
+
+import SwiftBridge
+
+@_cdecl("WrapperTransform_EntryPoint_ModuleDumper")
+public func main(module: swiftbridge_sil_SILModule) {
+#if DUMP_MODULE
+  swiftbridge_sil_SILModule_dump(module)
+#else
+  print(" I AM IN SWIFT AND I AM VERIFYING THE MODULE !")
+  swiftbridge_sil_SILModule_verify(module)
+#endif
+}

A few things about the design:

  1. I am taking advantage of the way Swift uses .def files/etc to try to hide all of the badness.

  2. The code is using a new library I wrote called SwiftBridge. This is just a way to use metaprogramming to bridge back and forth in between swift/c++ using c. Since it is in c, any swift program can use it. My hope is that other people can take what I have done here and use newtype, overlays, etc to clean this up. Adding new bridged types is very trivial:

Right now it only supports methods with 0 arguments, but I think the right pieces are there that it should be easy to expand.

  1. I am taking advantage of the way lifetimes are managed in the SILOptimizer (namely that optimizer passes do not manage /any/ IR memory directly) to enable the code to just use value types.

  2. Keep in mind that b/c this is autogenerated via cpp, the code isn't pretty, but with a small bit of work and attributes to rename things (as well as use newtype to make the c structs act like swift structs), it could look a lot prettier.

In terms of the requirements get this to work, you need the cmake release preview AND the most recent swift snapshot from swift.org. The reason why is that Compnerd/Jordan added support for emitting static libraries recently into Swift (not for exporting to users, AFAIKT, but rather for helping move around object files in a sane way) and I needed a swift that supported that. So if one installs that package and then uses a script like the following:

export env TOOLCHAINS=org.swift.50201906231a
./swift/utils/build-script \
    --release-debuginfo \
    --assertions \
    --skip-build-benchmarks \
    --build-subdir \
    swift-cmake-asserts\
    --cmake \
    $PATH_TO_CMAKE/cmake-3.15.0-rc2-Darwin-x86_64/CMake.app/Contents/bin/cmake

one will see something like:

[67/405] Compiling /Volumes/Files/gottesmm/work/solon/build/swift-cmake-asserts-no-distcc/swift-macosxx86_64/stdlib/public/SwiftOnoneSupport/TVOS/arm64/SwiftOnoneSupport.o
I AM IN SWIFT AND I AM VERIFYING THE MODULE !

This is all of the time I have to spend on this now, but I wanted to show that it is possible and (assuming this proof of concept is productized/cleaned up by someone), we could have it way before we have c++ interop.

Michael

(*) In fact, it isn't in the proof of concept now, but I was able to late last night create a Swift Package that linked against SwiftBridge and code in Xcode, get code completion, etc. The best part is that even though it fails to link, if the pass builds, you know that your pass will also build in Swift's cmake quickly, except that since we are just bringing in a module header, it takes no time at all to compile. That was the coolest thing I noticed while working on this.

16 Likes
19

It’s fantastic to see progress in this direction. Great work!

3 Likes