How to build the 5.7.1 toolchain in swift.org release-like archive layout on Ubuntu 20.04 or 22.04

mz2 · December 5, 2022, 5:06pm

Hi there!

I'm stuck trying to build the most recent release version of the Swift toolchain on Linux into an archive with a similar layout as the releases available at https://swift.org . I am in need of doing this as part of an ongoing effort of packaging the toolchain for Ubuntu, initially as a a ppa and ultimately in the universe package repository (I work for Canonical and have sought advice on how to accomplish this also internally, with there being initial support for it).

Thus far I've made a ppa available at swiftlang : Matias Piipari as a starting point essentially re-packaging the Swift.org tarballs (really a ropey proof of concept given the filesystem layout inside the archive etc), and also successfully elsewhere built some Snap packages of SwiftLint, SwiftFormat, the Vapor Toolbox etc using this deb as a staged component inside, mostly to check that package is OK. To actually do this in a way that will work across ARM64 and AMD64 microarchitectures and to address various lintian (Debian package linting errors with my ropey current package), building actually from source (with Ubuntu intended additional compiler flags added) is necessary.

My following attempt with utils/build-toolchain (should be reproducible in a clean VM such as using a 22.04 instance launched using multipass or any other VM host of choice) demonstrates the failure:

sudo snap install sccache --candidate --classic
sudo apt-get -y update
sudo apt-get -y install build-essential clang cmake git icu-devtools libcurl4-openssl-dev libedit-dev libicu-dev libncurses5-dev libpython3-dev libsqlite3-dev libxml2-dev ninja-build pkg-config python2 python2-dev python3-six python3-distutils rsync swig systemtap-sdt-dev tzdata unzip uuid-dev

mkdir source
cd source
git clone git@github.com:apple/swift.git ../../swiftlang-builder/swift
cd swift
utils/update-checkout --clone
utils/update-checkout --tag swift-5.7.1-RELEASE
./utils/build-toolchain teh_prefix

The build failure is as follows happens when buildin libdispatch_static:

+ /usr/bin/cmake --build /source/build/buildbot_linux/libdispatch_static-linux-x86_64 -- -j8 all
[73/82][ 89%][14.758s] Linking Swift static library src/swift/libswiftDispatch.a
FAILED: src/swift/libswiftDispatch.a src/swift/CMakeFiles/swiftDispatch.dir/Block.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Data.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Dispatch.swift.o src/swift/CMakeFiles/swiftDispatch.dir/IO.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Private.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Queue.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Source.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Time.swift.o src/swift/CMakeFiles/swiftDispatch.dir/Wrapper.swift.o src/swift/swift/Dispatch.swiftmodule 
: && /source/build/buildbot_linux/swift-linux-x86_64/bin/swiftc -output-file-map src/swift/CMakeFiles/swiftDispatch.dir/Release/output-file-map.json -incremental -j 8 -emit-library -static -o src/swift/libswiftDispatch.a -module-name Dispatch -module-link-name swiftDispatch -emit-module -emit-module-path src/swift/swift/Dispatch.swiftmodule -emit-dependencies  -module-cache-path "/source/build/buildbot_linux/libdispatch_static-linux-x86_64/module-cache" -O -Xcc -fblocks -Xcc -fmodule-map-file=/source/swift-corelibs-libdispatch/dispatch/module.modulemap -Xcc -I/source/swift-corelibs-libdispatch -Xcc -I/source/swift-corelibs-libdispatch/src/swift/shims -I /source/swift-corelibs-libdispatch/src/BlocksRuntime -I /source/build/buildbot_linux/libdispatch_static-linux-x86_64 -I /source/swift-corelibs-libdispatch -I /source/swift-corelibs-libdispatch/src -I /source/build/buildbot_linux/libdispatch_static-linux-x86_64/src /source/swift-corelibs-libdispatch/src/swift/Block.swift /source/swift-corelibs-libdispatch/src/swift/Data.swift /source/swift-corelibs-libdispatch/src/swift/Dispatch.swift /source/swift-corelibs-libdispatch/src/swift/IO.swift /source/swift-corelibs-libdispatch/src/swift/Private.swift /source/swift-corelibs-libdispatch/src/swift/Queue.swift /source/swift-corelibs-libdispatch/src/swift/Source.swift /source/swift-corelibs-libdispatch/src/swift/Time.swift /source/swift-corelibs-libdispatch/src/swift/Wrapper.swift    && :
<unknown>:0: error: could not build C module 'SwiftShims'
[80/82][ 97%][15.643s] Linking CXX executable dispatch_io_pipe
ninja: build stopped: subcommand failed.
ERROR: command terminated with a non-zero exit status 1, aborting

What am I doing wrong? The instructions over at swift/GettingStarted.md at main · apple/swift · GitHub
which suggest doing utils/build-script --release-debuginfo --skip-early-swift-driver \ --skip-early-swiftsyntax do work, but don't yield the full toolchain.

mishal_shah · December 5, 2022, 6:29pm

Hi @mz2,

To build the full toolchain, I would recommend following steps using Docker.

If you would like to avoid using Docker, you can installed the required packages from Dockerfile.

Clone:

git clone https://github.com/apple/swift-docker.git

Build Docker Image:

docker build -f swift-docker/swift-ci/master/ubuntu/22.04/ . -t swift-ci-ubuntu2204

Clone the repos:

git clone https://github.com/apple/swift.git
git -C swift checkout release/5.7
./swift/utils/update-checkout --scheme release/5.7 --clone
./swift/utils/update-checkout --tag swift-5.7.1-RELEASE

Build the toolchain:

docker run \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -v <swift source location>:/source \
  -v <some docker volume name>:/home/build-user \
  -w /home/build-user/ \
  <docker image tag from above> \
  /bin/bash -lc "cp -r /source/* /home/build-user/; ./swift/utils/build-script --preset buildbot_linux install_destdir=/home/build-user/swift-install installable_package=/home/build-user/swift-DEVELOPMENT-SNAPSHOT-$(date +'%F')-a.tar.gz"

Also, if you are interested in the Debs and RPM checkout swift-installer-scripts/platforms/Linux at main · apple/swift-installer-scripts · GitHub.

Let me know if you have any questions.

mz2 · December 6, 2022, 1:03pm

Awesome! Well that Ubuntu deb build looks exactly like the thing I would like to build for my purposes (and would be happy to help work towards inclusion in universe if that's of interest?). I didn't manage to build the deb with at least the state of the main branch without changes though, and have posted a quick draft PR here with some changes I thought would help towards getting it built with 5.7.1: Adjust Ubuntu related patches to build 5.7.1 by mz2 · Pull Request #149 · apple/swift-installer-scripts · GitHub.

The main branch looked to be in some inconsistent state, noting 5.6.2 in changelog but referring to 5.7.1-RELEASE elsewhere (the yams dependency version also didn't look like it was the one used with 5.7.1?). More importantly there were build failures with 5.7.1 that I had seen before that I could resolve (the --skip-early-swift-driver being necessary being an example of that), only to see some other build failures. Possibly I am not adjusting the build preset correctly since the benchmarks were still getting built even if I supposedly made them not?

mz2 · December 17, 2022, 6:29am

Ah, there was another toolchain-benchmarks some lines below in the same preset, so of course the skip-toolchain-benchmarks somewhere above wouldn't have effect. Patched that away, trying again now… (that does work, now going to try to make lintian happy, too)

@mishal_shah any thoughts on why that failure is happening when building the Focal deb? I continue to see an equivalent failure both when attempting to build (with docker-compose, as advised in the README) either 5.7.1. or 5.7.2 if I don't patch away the toolchain-benchmarks:

--- Building benchmarks ---
Traceback (most recent call last):
  File "swift/utils/build-script", line 720, in <module>
    sys.exit(main())
  File "swift/utils/build-script", line 715, in main
    return main_normal()
  File "swift/utils/build-script", line 671, in main_normal
    invocation.execute()
  File "/tmp/swift-deb-builder/swiftlang-5.7.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 672, in execute
    self._execute(pipeline, [self.args.host_target] +
  File "/tmp/swift-deb-builder/swiftlang-5.7.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 735, in _execute
    self.execute_product_build_steps(product_class, host_target)
  File "/tmp/swift-deb-builder/swiftlang-5.7.2/swift/utils/swift_build_support/swift_build_support/build_script_invocation.py", line 791, in execute_product_build_steps
    product.build(host_target)
  File "/tmp/swift-deb-builder/swiftlang-5.7.2/swift/utils/swift_build_support/swift_build_support/products/benchmarks.py", line 49, in build
    run_build_script_helper(host_target, self, self.args)
  File "/tmp/swift-deb-builder/swiftlang-5.7.2/swift/utils/swift_build_support/swift_build_support/products/benchmarks.py", line 123, in run_build_script_helper
    toolchain_path = _get_toolchain_path(host_target, product, args)
  File "/tmp/swift-deb-builder/swiftlang-5.7.2/swift/utils/swift_build_support/swift_build_support/products/benchmarks.py", line 107, in _get_toolchain_path
    if swiftpm.SwiftPM.has_cross_compile_hosts(args):
  File "/tmp/swift-deb-builder/swiftlang-5.7.2/swift/utils/swift_build_support/swift_build_support/products/product.py", line 248, in has_cross_compile_hosts
    return self.args.cross_compile_hosts
AttributeError: 'Namespace' object has no attribute 'args'
ERROR: command terminated with a non-zero exit status 1, aborting

mz2 · December 27, 2022, 10:57pm

Another Jammy specific build failure I am scratching my head with based on the Deb packaging scripts you linked to above is the following llvm (patched runtime build related?) failure that happens when building with an attempted Jammy specific debian package build configuration I made (is happening with a debian/control unmodified from the focal build, is not happening when building with the docker run steps you describe above, so I am guessing whatever is the focal/jammy difference boils down to something I am including / not including amongst package build dependencies in my Jammy specific debian/control which makes the ABI detection fail?):

build_1         | [3832/4434][ 86%][625.140s] cd /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime && /usr/bin/cmake -E echo_append && /usr/bin/cmake -E touch /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-stamps//compiler-rt-patch
build_1         | [3832/4434][ 86%][625.140s] cd /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins && /usr/bin/cmake -DCMAKE_C_COMPILER=/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -DCMAKE_CXX_COMPILER=/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang++ -DCMAKE_ASM_COMPILER=/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/ninja-build/ninja -DLLVM_CONFIG_PATH=/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/llvm-config "-DLLVM_LIT_ARGS=-v --time-tests -j 64" -DCOMPILER_RT_OUTPUT_DIR=/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./lib/clang/13.0.0 -DCOMPILER_RT_EXEC_OUTPUT_DIR=/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin -DCOMPILER_RT_INSTALL_PATH:PATH=lib/clang/13.0.0 -DCOMPILER_RT_INCLUDE_TESTS=ON -DCMAKE_INSTALL_PREFIX=/usr/lib/swiftlang/5.7.2 -DLLVM_LIBDIR_SUFFIX= -DLLVM_RUNTIME_OUTPUT_INTDIR=/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin -DCMAKE_OSX_DEPLOYMENT_TARGET= -DCMAKE_OSX_SYSROOT:PATH= -DCOMPILER_RT_INTERCEPT_LIBDISPATCH=ON -DCOMPILER_RT_PREFIX=/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/projects/compiler-rt -DCOMPILER_RT_SRC_ROOT=/tmp/swift-deb-builder/swiftlang-5.7.2/llvm-project/llvm/../compiler-rt -GNinja /tmp/swift-deb-builder/swiftlang-5.7.2/llvm-project/llvm/../compiler-rt && /usr/bin/cmake -E touch /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-stamps//compiler-rt-configure
build_1         | Re-run cmake no build system arguments
build_1         | -- The C compiler identification is Clang 13.0.0
build_1         | -- The CXX compiler identification is Clang 13.0.0
build_1         | -- The ASM compiler identification is Clang with GNU-like command-line
build_1         | -- Found assembler: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
build_1         | -- Detecting C compiler ABI info
build_1         | -- Detecting C compiler ABI info - failed
build_1         | -- Check for working C compiler: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang
build_1         | -- Check for working C compiler: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang - broken
build_1         | CMake Error at /usr/share/cmake-3.22/Modules/CMakeTestCCompiler.cmake:69 (message):
build_1         |   The C compiler
build_1         | 
build_1         |     "/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang"
build_1         | 
build_1         |   is not able to compile a simple test program.
build_1         | 
build_1         |   It fails with the following output:
build_1         | 
build_1         |     Change Dir: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeTmp
build_1         |     
build_1         |     Run Build Command(s):/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/ninja-build/ninja cmTC_f7244 && [1/2][ 50%][0.109s] Building C object CMakeFiles/cmTC_f7244.dir/testCCompiler.c.o
build_1         |     clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
build_1         |     clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
build_1         |     [2/2][100%][0.182s] Linking C executable cmTC_f7244
build_1         |     FAILED: cmTC_f7244 
build_1         |     : && /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -g -O2 -ffile-prefix-map=/tmp/swift-deb-builder/swiftlang-5.7.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro CMakeFiles/cmTC_f7244.dir/testCCompiler.c.o -o cmTC_f7244   && :
build_1         |     /usr/bin/ld: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: error loading plugin: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory
build_1         |     clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
build_1         |     ninja: build stopped: subcommand failed.
build_1         |     
build_1         |     
build_1         | 
build_1         |   
build_1         | 
build_1         |   CMake will not be able to correctly generate this project.
build_1         | Call Stack (most recent call first):
build_1         |   CMakeLists.txt:10 (project)
build_1         | 
build_1         | 
build_1         | -- Configuring incomplete, errors occurred!
build_1         | See also "/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeOutput.log".
build_1         | See also "/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeError.log".

My latest attempted build dependencies in the attempted debian/control (happens also with an unmodified debian/control from the focal package as noted above):

Build-Depends: debhelper-compat (= 12), devscripts,
  build-essential,
  clang, cmake, git, icu-devtools, ninja-build, pkg-config,
  libcurl4-openssl-dev, libedit-dev, libicu-dev, libncurses5-dev,
  libpython3-dev, libsqlite3-dev, libxml2-dev,
  python2, python2-dev, python-six, python3-six, python3-distutils, python3-psutil,
  systemtap-sdt-dev, tzdata, unzip, uuid-dev,
  libgcc-9-dev, libstdc++-9-dev, libxml2-dev, zlib1g-dev, binutils, pkg-config

What might I be missing?

mz2 · December 28, 2022, 12:40pm

Amongst the relevant CMake output, I could pick out the following things which I wonder if they are meaningful:

ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/11/../../../../x86_64-linux-gnu/include"
ignoring nonexistent directory "/include"
/usr/bin/ld: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: error loading plugin: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory

There exists /usr/lib/gcc/x86_64-linux-gnu/11/include/ (some combo of libstdc++-11-dev, libgcc-11-dev, libobjc-11-dev install files there) but indeed /usr/x86_64-linux-gnu/11/include/ (which /usr/lib/gcc/x86_64-linux-gnu/11/../../../../x86_64-linux-gnu/include evaluates to, nor /include does not exist).

The full CMakeOutput.log looks like:

The system is: Linux - 5.19.0-26-generic - x86_64
Compiling the C compiler identification source file "CMakeCCompilerId.c" succeeded.
Compiler: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang 
Build flags: -g;-O2;-ffile-prefix-map=/tmp/swift-deb-builder/swiftlang-5.7.2=.;-flto=auto;-ffat-lto-objects;-flto=auto;-ffat-lto-objects;-fstack-protector-strong;-Wformat;-Werror=format-security
Id flags: -c 

The output was:
0
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]


Compilation of the C compiler identification source "CMakeCCompilerId.c" produced "CMakeCCompilerId.o"

Compiling the C compiler identification source file "CMakeCCompilerId.c" succeeded.
Compiler: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang 
Build flags: -g;-O2;-ffile-prefix-map=/tmp/swift-deb-builder/swiftlang-5.7.2=.;-flto=auto;-ffat-lto-objects;-flto=auto;-ffat-lto-objects;-fstack-protector-strong;-Wformat;-Werror=format-security
Id flags: -c;-I__does_not_exist__ 

The output was:
0
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]


Compilation of the C compiler identification source "CMakeCCompilerId.c" produced "CMakeCCompilerId.o"

Compiling the C compiler identification source file "CMakeCCompilerId.c" succeeded.
Compiler: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang 
Build flags: 
Id flags:  

The output was:
0


Compilation of the C compiler identification source "CMakeCCompilerId.c" produced "a.out"

The C compiler identification is Clang, found in "/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/3.25.0/CompilerIdC/a.out"

Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" succeeded.
Compiler: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang++ 
Build flags: -g;-O2;-ffile-prefix-map=/tmp/swift-deb-builder/swiftlang-5.7.2=.;-flto=auto;-ffat-lto-objects;-flto=auto;-ffat-lto-objects;-fstack-protector-strong;-Wformat;-Werror=format-security
Id flags: -c 

The output was:
0
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]


Compilation of the CXX compiler identification source "CMakeCXXCompilerId.cpp" produced "CMakeCXXCompilerId.o"

Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" succeeded.
Compiler: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang++ 
Build flags: -g;-O2;-ffile-prefix-map=/tmp/swift-deb-builder/swiftlang-5.7.2=.;-flto=auto;-ffat-lto-objects;-flto=auto;-ffat-lto-objects;-fstack-protector-strong;-Wformat;-Werror=format-security
Id flags: -c;-I__does_not_exist__ 

The output was:
0
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]


Compilation of the CXX compiler identification source "CMakeCXXCompilerId.cpp" produced "CMakeCXXCompilerId.o"

Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" succeeded.
Compiler: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang++ 
Build flags: 
Id flags:  

The output was:
0


Compilation of the CXX compiler identification source "CMakeCXXCompilerId.cpp" produced "a.out"

The CXX compiler identification is Clang, found in "/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/3.25.0/CompilerIdCXX/a.out"

Checking whether the ASM compiler is Clang using "--version" matched "(clang version)":
clang version 13.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin

... and the CMakeError.log as follows:

The C compiler identification could not be found in "/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/3.25.0/CompilerIdC/CMakeCCompilerId.o"

The C compiler identification could not be found in "/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/3.25.0/CompilerIdC/CMakeCCompilerId.o"

The CXX compiler identification could not be found in "/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/3.25.0/CompilerIdCXX/CMakeCXXCompilerId.o"

The CXX compiler identification could not be found in "/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/3.25.0/CompilerIdCXX/CMakeCXXCompilerId.o"

Checking whether the ASM compiler is GNU using "--version" did not match "(GNU assembler)|(GCC)|(Free Software Foundation)":
clang version 13.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin
Detecting C compiler ABI info failed to compile with the following output:
Change Dir: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-1sLM87

Run Build Command(s):/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/ninja-build/ninja cmTC_33cd8 && [1/2][ 50%][0.111s] Building C object CMakeFiles/cmTC_33cd8.dir/CMakeCCompilerABI.c.o
clang version 13.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Candidate multilib: .;@m64
Selected multilib: .;@m64
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
 (in-process)
 "/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/bin/clang-13" -cc1 -triple x86_64-unknown-linux-gnu -emit-llvm-bc -flto=full -flto-unit -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name CMakeCCompilerABI.c -mrelocation-model static -mframe-pointer=none -fmath-errno -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -mllvm -treat-scalable-fixed-error-as-warning -debug-info-kind=constructor -dwarf-version=4 -debugger-tuning=gdb -v -fcoverage-compilation-dir=/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-1sLM87 -resource-dir /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/lib/clang/13.0.0 -dependency-file CMakeFiles/cmTC_33cd8.dir/CMakeCCompilerABI.c.o.d -skip-unused-modulemap-deps -MT CMakeFiles/cmTC_33cd8.dir/CMakeCCompilerABI.c.o -sys-header-deps -internal-isystem /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/lib/clang/13.0.0/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/11/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -fmacro-prefix-map=/tmp/swift-deb-builder/swiftlang-5.7.2=. -fcoverage-prefix-map=/tmp/swift-deb-builder/swiftlang-5.7.2=. -O2 -Wformat -fdebug-compilation-dir=/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-1sLM87 -fdebug-prefix-map=/tmp/swift-deb-builder/swiftlang-5.7.2=. -ferror-limit 19 -stack-protector 2 -fgnuc-version=4.2.1 -vectorize-loops -vectorize-slp -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o CMakeFiles/cmTC_33cd8.dir/CMakeCCompilerABI.c.o -x c /tmp/swift-deb-builder/swiftlang-5.7.2/cmake/Modules/CMakeCCompilerABI.c
clang -cc1 version 13.0.0 based upon LLVM 13.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/11/../../../../x86_64-linux-gnu/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/lib/clang/13.0.0/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
[2/2][100%][0.163s] Linking C executable cmTC_33cd8
FAILED: cmTC_33cd8 
: && /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -g -O2 -ffile-prefix-map=/tmp/swift-deb-builder/swiftlang-5.7.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -v CMakeFiles/cmTC_33cd8.dir/CMakeCCompilerABI.c.o -o cmTC_33cd8   && :
clang version 13.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/usr/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o cmTC_33cd8 /lib/x86_64-linux-gnu/crt1.o /lib/x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/11/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/bin/../lib -L/lib -L/usr/lib -plugin /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so -plugin-opt=mcpu=x86-64 -plugin-opt=O2 -Bsymbolic-functions -z relro CMakeFiles/cmTC_33cd8.dir/CMakeCCompilerABI.c.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/11/crtend.o /lib/x86_64-linux-gnu/crtn.o
/usr/bin/ld: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: error loading plugin: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.




Determining if the C compiler works failed with the following output:
Change Dir: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/tools/clang/runtime/compiler-rt-bins/CMakeFiles/CMakeScratch/TryCompile-suBkX9

Run Build Command(s):/tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/ninja-build/ninja cmTC_313e0 && [1/2][ 50%][0.058s] Building C object CMakeFiles/cmTC_313e0.dir/testCCompiler.c.o
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
[2/2][100%][0.095s] Linking C executable cmTC_313e0
FAILED: cmTC_313e0 
: && /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/./bin/clang -g -O2 -ffile-prefix-map=/tmp/swift-deb-builder/swiftlang-5.7.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro CMakeFiles/cmTC_313e0.dir/testCCompiler.c.o -o cmTC_313e0   && :
/usr/bin/ld: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: error loading plugin: /tmp/swift-deb-builder/swiftlang-5.7.2/build/buildbot_linux/llvm-linux-x86_64/bin/../lib/LLVMgold.so: cannot open shared object file: No such file or directory
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

mz2 · December 28, 2022, 6:31pm

The issue is actually something to do with the environment that debuild is building the package in in case of jammy, since this specific build fail does not surface if I call the build script directly in the same Docker container where it fails when going via build_deb.sh.

Environment variables? Something to do with focal / jammy differences of how fakeroot behaves? Ideas very welcome of how to debug this further.

mz2 · December 29, 2022, 2:48pm

Right, so that problem was down to flto=auto and -ffat-lto-objects which are set by default on Jammy. Manipulating them with dpkg-buildflags in debian/rules sorted that out.

sebsto · April 19, 2023, 12:11am

Same error (LLVMgold.so not found) when building on Amazon Linux 2023.

I added this line to swiftlang.spec as per https://fedoraproject.org/wiki/LTOByDefault

%global _lto_cflags %nil