Using cmake to build swift instead of build-script

Hi, I get the feeling this question is going to be shot down in flames but anyway... from what I can see, the utils tool scripts in swift do some fairly obvious things and are there to make your life easier, but of course slightly hide what's happening "really".

Caveat: my source is still a bit old, swift 5.3.x

As I understand it, build-script is a python wrapper for build-script-impl, which is itself a shell script, that in turn basically copies some files around to set up an out of source build area then runs CMake, which does the main body of the work to make a buildable project. For example if you do utils/build-script -x it will invoke cmake -G Xcode.

I want to see if I can bypass the middleman and do these steps myself, rather than relying on scripts.

My reasons?

  1. familiarity, I've always used CMake with llvm and I know where I am
  2. reliability, these scripts make "normal" builds of swift, especially for newer developers, much more of a point-and-click affair... but for me, I find they're easier to break if I don't do the standard builds, if python2 has disappeared from my mac (for example) and so on... generally I don't like this feeling and I think it's a bit of a "black box".
  3. flexibility, in my use case today I wanted to set up a CI server to build my forks of swift/llvm but just parts of them (like the llvm libraries but not compiler-rt and the swift compiler but not the standard library and not run unit tests) and I'm left scratching my head what is being done by the 4000 lines of scripting and what is being done by standard tools like cmake and ninja.

Any pointers are greatly appreciated.

3 Likes

I would suggest taking a look at swift/build-windows-toolchain.bat at main · apple/swift (github.com) as that is a trace of all the cmake invocations needed to build a complete toolchain for Windows. There are a couple of cmake caches for other platforms as well if you are building for a different platform.

2 Likes

That’s absolutely perfect! Thanks so much! :slight_smile:

Actually... @compnerd ... a follow up question while I'm trying to read your batch file (I'm not very good at reading Windows batch files!)

:: Build Toolchain
cmake ^
  -B "%BuildRoot%\1" ^

  -C %SourceRoot%\swift\cmake\caches\Windows-x86_64.cmake ^

  -D CMAKE_BUILD_TYPE=%CMAKE_BUILD_TYPE% ^
  -D CMAKE_C_COMPILER=cl ^
  -D CMAKE_C_FLAGS="/GS- /Oy /Gw /Gy" ^
  -D CMAKE_CXX_COMPILER=cl ^
  -D CMAKE_CXX_FLAGS="/GS- /Oy /Gw /Gy" ^
  -D CMAKE_MT=mt ^
  -D CMAKE_EXE_LINKER_FLAGS="/INCREMENTAL:NO" ^
  -D CMAKE_SHARED_LINKER_FLAGS="/INCREMENTAL:NO" ^

  -D CMAKE_INSTALL_PREFIX="%InstallRoot%" ^

  -D LLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-windows-msvc ^

  -D PACKAGE_VENDOR="swift.org" ^
  -D CLANG_VENDOR="swift.org" ^
  -D CLANG_VENDOR_UTI="org.swift" ^
  -D LLVM_APPEND_VC_REV=NO ^
  -D LLVM_VERSION_SUFFIX="" ^

  -D SWIFT_ENABLE_EXPERIMENTAL_CONCURRENCY=YES ^
  -D SWIFT_ENABLE_EXPERIMENTAL_DISTRIBUTED=YES ^
  -D SWIFT_ENABLE_EXPERIMENTAL_DIFFERENTIABLE_PROGRAMMING=YES ^
  -D SWIFT_ENABLE_EXPERIMENTAL_STRING_PROCESSING=YES ^

  -D LLVM_EXTERNAL_SWIFT_SOURCE_DIR="%SourceRoot%\swift" ^
  -D LLVM_EXTERNAL_CMARK_SOURCE_DIR="%SourceRoot%\cmark" ^
  -D PYTHON_HOME=%PYTHON_HOME% ^
  -D PYTHON_EXECUTABLE=%PYTHON_HOME%\python.exe ^
  -D SWIFT_PATH_TO_LIBDISPATCH_SOURCE="%SourceRoot%\swift-corelibs-libdispatch" ^
  -D EXPERIMENTAL_STRING_PROCESSING_SOURCE_DIR=%SourceRoot%\swift-experimental-string-processing ^

  -G Ninja ^
  -S llvm-project\llvm || (exit /b)
cmake --build "%BuildRoot%\1" || (exit /b)
cmake --build "%BuildRoot%\1" --target install || (exit /b)

:: Build Swift Standard Library
cmake ^
  -B %BuildRoot%\2 ^

  -C %SourceRoot%\swift\cmake\caches\Runtime-Windows-x86_64.cmake ^

  -D CMAKE_BUILD_TYPE=%CMAKE_BUILD_TYPE% ^
  -D CMAKE_C_COMPILER=%BuildRoot%/1/bin/clang-cl.exe ^
  -D CMAKE_C_FLAGS="/GS- /Oy /Gw /Gy" ^
  -D CMAKE_CXX_COMPILER=%BuildRoot%/1/bin/clang-cl.exe ^
  -D CMAKE_CXX_FLAGS="/GS- /Oy /Gw /Gy" ^
  -D CMAKE_MT=mt ^
  -D CMAKE_EXE_LINKER_FLAGS="/INCREMENTAL:NO" ^
  -D CMAKE_SHARED_LINKER_FLAGS="/INCREMENTAL:NO" ^

  -D CMAKE_INSTALL_PREFIX=%SDKInstallRoot%\usr ^

  -D LLVM_DIR=%BuildRoot%\1\lib\cmake\llvm ^
  -D SWIFT_NATIVE_SWIFT_TOOLS_PATH=%BuildRoot%\1\bin ^
  -D SWIFT_PATH_TO_LIBDISPATCH_SOURCE=%SourceRoot%\swift-corelibs-libdispatch ^
  -D EXPERIMENTAL_STRING_PROCESSING_SOURCE_DIR=%SourceRoot%\swift-experimental-string-processing ^

  -D SWIFT_ENABLE_EXPERIMENTAL_CONCURRENCY=YES ^
  -D SWIFT_ENABLE_EXPERIMENTAL_DISTRIBUTED=YES ^
  -D SWIFT_ENABLE_EXPERIMENTAL_DIFFERENTIABLE_PROGRAMMING=YES ^
  -D SWIFT_ENABLE_EXPERIMENTAL_STRING_PROCESSING=YES ^

  -G Ninja ^
  -S %SourceRoot%\swift || (exit /b)
cmake --build %BuildRoot%\2 || (exit /b)
cmake --build %BuildRoot%\2 --target install || (exit /b)

... a couple of questions.

  1. This looks like it tries to create the out of tree build directory for the "toolchain" in (forgive me translating to unix so I can understand it)... ${BuildRoot}/1 and then runs cmake --build in that directory to actually build the product and finally cmake --build ... --target install to install the built product somewhere suitable.
  2. It then builds the "standard library" in ${BuildRoot}/2 and goes on from there (libdispatch, spm, etc. etc.) ... the subdirectories in these builds just have numbers because there's no real need to give these intermediates meaningful paths, right? Or is this just me not able to read Windows batch files?
  3. When it says build "toolchain" that looks to me like llvm/clang, right? And where the comment says it's building the standard library, that looks to me like it's building the swift compiler and then the standard library, is that correct? The comments maybe aren't 100% accurate there? Or am I missing something?

Thanks,
Carl

  1. Yes, each set of products are configured, built, installed for the most part.
  2. They are numbers to compress the path as the paths are incredibly long (260+ characters) which causes problems.
  3. No, the toolchain is the full toolchain: all compilers, linkers, debuggers, resources, and binary utilities. The swift compiler is part of the toolchain. The runtime only builds the swift standard library.

OK, understood. Thanks!

@carlos42421 My Buildroot and Yocto Swift layers invoke cmake directly, you could take a look at those for how to call it to build the stdlib only

1 Like