[Help Needed] Failed to build Swift toolchain on Linux due to search path conflict

I have the experience of building the toolchain on macOS successfully.

But recently I want to build a Linux toolchain to debug the swift_newtype issue on Linux.

Following the instruction on GettingStarted.md with both release/5.9 and release/5.10 gives the same error message.

# Build Command
utils/build-script --release-debuginfo --skip-early-swift-driver \
       --skip-early-swiftsyntax --sccache
FAILED: SwiftCompilerSources/CMakeFiles/importedHeaderDependencies.dir/HeaderDependencies.cpp.o 
/usr/local/bin/sccache /usr/bin/clang++ -DCMARK_STATIC_DEFINE -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_LIBCPP_ENABLE_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/swift-linux-aarch64/SwiftCompilerSources -I/Users/kyle/Developer/SwiftProject/swift/SwiftCompilerSources -I/Users/kyle/Developer/SwiftProject/swift/stdlib/public/SwiftShims -I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/swift-linux-aarch64/include -I/Users/kyle/Developer/SwiftProject/swift/include -I/Users/kyle/Developer/SwiftProject/llvm-project/llvm/include -I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/llvm-linux-aarch64/include -I/Users/kyle/Developer/SwiftProject/llvm-project/clang/include -I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/llvm-linux-aarch64/tools/clang/include -I/Users/kyle/Developer/SwiftProject/cmark/src/include -I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/cmark-linux-aarch64/src -I/Users/kyle/Developer/SwiftProject/swift-corelibs-libdispatch/src/BlocksRuntime -I/Users/kyle/Developer/SwiftProject/swift-corelibs-libdispatch -I/Users/kyle/Developer/SwiftProject/swift/SwiftCompilerSources/../include/swift -Wno-unknown-warning-option -Werror=unguarded-availability-new -fno-stack-protector -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-class-memaccess -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Werror=switch -Wdocumentation -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -DOBJC_OLD_DISPATCH_PROTOTYPES=0 -O2 -DNDEBUG -Werror=gnu -Werror=c++98-compat-extra-semi -UNDEBUG -std=c++17 -MD -MT SwiftCompilerSources/CMakeFiles/importedHeaderDependencies.dir/HeaderDependencies.cpp.o -MF SwiftCompilerSources/CMakeFiles/importedHeaderDependencies.dir/HeaderDependencies.cpp.o.d -o SwiftCompilerSources/CMakeFiles/importedHeaderDependencies.dir/HeaderDependencies.cpp.o -c /Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/swift-linux-aarch64/SwiftCompilerSources/HeaderDependencies.cpp
In file included from /Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/swift-linux-aarch64/SwiftCompilerSources/HeaderDependencies.cpp:3:
In file included from /Users/kyle/Developer/SwiftProject/swift/SwiftCompilerSources/../include/swift/Basic/BasicBridging.h:19:
In file included from /Users/kyle/Developer/SwiftProject/llvm-project/llvm/include/llvm/CAS/CASReference.h:12:
In file included from /Users/kyle/Developer/SwiftProject/llvm-project/llvm/include/llvm/ADT/ArrayRef.h:12:
In file included from /Users/kyle/Developer/SwiftProject/llvm-project/llvm/include/llvm/ADT/Hashing.h:53:
In file included from /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/cstring:42:
In file included from /usr/include/string.h:462:
In file included from /Users/kyle/Developer/SwiftProject/swift/SwiftCompilerSources/../include/swift/strings.h:16:
In file included from /Users/kyle/Developer/SwiftProject/swift/include/swift/Basic/LLVM.h:24:
In file included from /Users/kyle/Developer/SwiftProject/llvm-project/llvm/include/llvm/Support/Casting.h:17:
In file included from /Users/kyle/Developer/SwiftProject/llvm-project/llvm/include/llvm/ADT/Optional.h:20:
In file included from /Users/kyle/Developer/SwiftProject/llvm-project/llvm/include/llvm/ADT/None.h:19:
/usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/optional:64:3: error: templates must have C++ linkage
  template<typename _Tp>
  ^~~~~~~~~~~~~~~~~~~~~~
/usr/include/string.h:28:1: note: extern "C" language linkage specification begins here
__BEGIN_DECLS
^
/usr/include/aarch64-linux-gnu/sys/cdefs.h:133:24: note: expanded from macro '__BEGIN_DECLS'
# define __BEGIN_DECLS  extern "C" {
                        ^

It looks to be a search path conflict issue to me. When encountering #include <string.hs> in /usr/include/string.h - Line 462 on my host Linux machine, it is wrongly parsed as ${PROJECT_ROOT}/swift/SwiftCompilerSources/../include/swift/strings.h instead of the correct one - /usr/include/strings.h.

...
#ifdef __USE_MISC
# include <strings.h>
...

How can I fix the issue here? Is it related with the param I pass to utils/build-script script? Any guidance would be appreciated.

# Env
Linux ubuntu-2204 6.5.13-orbstack-00122-g57b8027e2387 #1 SMP Tue Feb  6 07:48:26 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

# Swift Project Version
branch: release/5.10
commit: 17ca88c94a34b34c3c354891e899e82ce98f46ee

I can reproduce it by running the following command in Ninja-RelWithDebInfoAssert/swift-linux-aarch64 folder.

/usr/local/bin/sccache \
/usr/bin/clang++ \
-DCMARK_STATIC_DEFINE \
-D_DEBUG \
-D_GLIBCXX_ASSERTIONS \
-D_LIBCPP_ENABLE_ASSERTIONS \
-D__STDC_CONSTANT_MACROS \
-D__STDC_FORMAT_MACROS \
-D__STDC_LIMIT_MACROS \
-I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/swift-linux-aarch64/SwiftCompilerSources \
-I/Users/kyle/Developer/SwiftProject/swift/SwiftCompilerSources \
-I/Users/kyle/Developer/SwiftProject/swift/stdlib/public/SwiftShims \
-I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/swift-linux-aarch64/include \
-I/Users/kyle/Developer/SwiftProject/swift/include \
-I/Users/kyle/Developer/SwiftProject/llvm-project/llvm/include \
-I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/llvm-linux-aarch64/include \
-I/Users/kyle/Developer/SwiftProject/llvm-project/clang/include \
-I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/llvm-linux-aarch64/tools/clang/include \
-I/Users/kyle/Developer/SwiftProject/cmark/src/include \
-I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/cmark-linux-aarch64/src \
-I/Users/kyle/Developer/SwiftProject/swift-corelibs-libdispatch/src/BlocksRuntime \
-I/Users/kyle/Developer/SwiftProject/swift-corelibs-libdispatch \
-I/Users/kyle/Developer/SwiftProject/swift/SwiftCompilerSources/../include/swift \
-Wno-unknown-warning-option \
-Werror=unguarded-availability-new \
-fno-stack-protector \
-fPIC \
-fno-semantic-interposition \
-fvisibility-inlines-hidden \
-Werror=date-time \
-Werror=unguarded-availability-new \
-Wall \
-Wextra \
-Wno-unused-parameter \
-Wwrite-strings \
-Wcast-qual \
-Wmissing-field-initializers \
-Wimplicit-fallthrough \
-Wcovered-switch-default \
-Wno-class-memaccess \
-Wno-noexcept-type \
-Wnon-virtual-dtor \
-Wdelete-non-virtual-dtor \
-Wsuggest-override \
-Wstring-conversion \
-Wmisleading-indentation \
-Wctad-maybe-unsupported \
-fdiagnostics-color \
-ffunction-sections \
-fdata-sections \
-Werror=switch \
-Wdocumentation \
-Wimplicit-fallthrough \
-Wunreachable-code \
-Woverloaded-virtual \
-DOBJC_OLD_DISPATCH_PROTOTYPES=0 \
-O2 \
-DNDEBUG \
-Werror=gnu \
-Werror=c++98-compat-extra-semi \
-UNDEBUG \
-std=c++17 \
-MD \
-MT \
SwiftCompilerSources/CMakeFiles/importedHeaderDependencies.dir/HeaderDependencies.cpp.o \
-MF \
SwiftCompilerSources/CMakeFiles/importedHeaderDependencies.dir/HeaderDependencies.cpp.o.d \
-o \
SwiftCompilerSources/CMakeFiles/importedHeaderDependencies.dir/HeaderDependencies.cpp.o \
-c \
/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/swift-linux-aarch64/SwiftCompilerSources/HeaderDependencies.cpp

Also I noticed 2 suspicious log message in the build output

...
symlinking the system headers (/usr/include/c++) into the local clang build directory (/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/llvm-linux-aarch64/include).
...
-- LLVM host triple: aarch64-unknown-linux-gnu
-- LLVM default target triple: aarch64-unknown-linux-gnu
-- Building with -fPIC
-- Can't get errc messages in cross-compilation mode

First things first, did you install all dependencies?

Of course.

OK, the Swift build on linux pretty much requires a prebuilt host toolchain since 5.9: do you have one installed and added to your PATH? If so, try building from scratch with those two --skip-early-* flags removed, as shown in the trunk doc for linux.

Thanks for the suggestion. I'll try them both and give my result later.

Back to the original issue, I saved the clang++ compile command to a shell script and play around to debug it.

First, I add -I/usr/include \ as the first -I statement. But it does not help. (Weird :face_with_monocle:)

Second, I added an almost empty test.cpp file in the same location and update the build command to build it.

#include <strings.h>
void hello();
void hello() {
    bzero(0, 0);
}
/usr/local/bin/sccache \
/usr/bin/clang++ \
-DCMARK_STATIC_DEFINE \
-D_DEBUG \
-D_GLIBCXX_ASSERTIONS \
-D_LIBCPP_ENABLE_ASSERTIONS \
-D__STDC_CONSTANT_MACROS \
-D__STDC_FORMAT_MACROS \
-D__STDC_LIMIT_MACROS \
-I/usr/include \
-I/Users/kyle/Developer/SwiftProject/swift/SwiftCompilerSources \
-I/Users/kyle/Developer/SwiftProject/swift/stdlib/public/SwiftShims \
-I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/swift-linux-aarch64/include \
-I/Users/kyle/Developer/SwiftProject/swift/include \
-I/Users/kyle/Developer/SwiftProject/llvm-project/llvm/include \
-I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/llvm-linux-aarch64/include \
-I/Users/kyle/Developer/SwiftProject/llvm-project/clang/include \
-I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/llvm-linux-aarch64/tools/clang/include \
-I/Users/kyle/Developer/SwiftProject/cmark/src/include \
-I/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/cmark-linux-aarch64/src \
-I/Users/kyle/Developer/SwiftProject/swift-corelibs-libdispatch/src/BlocksRuntime \
-I/Users/kyle/Developer/SwiftProject/swift-corelibs-libdispatch \
-I/Users/kyle/Developer/SwiftProject/swift/SwiftCompilerSources/../include/swift \
-Wno-unknown-warning-option \
-Werror=unguarded-availability-new \
-fno-stack-protector \
-fPIC \
-fno-semantic-interposition \
-fvisibility-inlines-hidden \
-Werror=date-time \
-Werror=unguarded-availability-new \
-Wall \
-Wextra \
-Wno-unused-parameter \
-Wwrite-strings \
-Wcast-qual \
-Wmissing-field-initializers \
-Wimplicit-fallthrough \
-Wcovered-switch-default \
-Wno-class-memaccess \
-Wno-noexcept-type \
-Wnon-virtual-dtor \
-Wdelete-non-virtual-dtor \
-Wsuggest-override \
-Wstring-conversion \
-Wmisleading-indentation \
-Wctad-maybe-unsupported \
-fdiagnostics-color \
-ffunction-sections \
-fdata-sections \
-Werror=switch \
-Wdocumentation \
-Wimplicit-fallthrough \
-Wunreachable-code \
-Woverloaded-virtual \
-DOBJC_OLD_DISPATCH_PROTOTYPES=0 \
-O2 \
-DNDEBUG \
-Werror=gnu \
-Werror=c++98-compat-extra-semi \
-UNDEBUG \
-std=c++17 \
-MD \
-MT \
SwiftCompilerSources/CMakeFiles/importedHeaderDependencies.dir/test.cpp.o \
-MF \
SwiftCompilerSources/CMakeFiles/importedHeaderDependencies.dir/test.cpp.o.d \
-o \
SwiftCompilerSources/CMakeFiles/importedHeaderDependencies.dir/test.cpp.o \
-c \
/Users/kyle/Developer/SwiftProject/build/Ninja-RelWithDebInfoAssert/swift-linux-aarch64/SwiftCompilerSources/test.cpp

It still failed with the same reason. I then delete all other -I except the first -I/usr/include. Then it finally compiles successfully.

So it looks like some extra -I is the root cause for the issue.

Update: The extra -I which cause the issue is -I/Users/kyle/Developer/SwiftProject/swift/SwiftCompilerSources/../include/swift \

Stepping back a little, maybe the <cstring.h> in Hashing.h should use llvm's libc++ instead of the host Linux machine's glibc header?

In file included from /Users/kyle/Developer/SwiftProject/llvm-project/llvm/include/llvm/ADT/Hashing.h:53:
In file included from /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/cstring:42:
In file included from /usr/include/string.h:462:
In file included from /Users/kyle/Developer/SwiftProject/swift/SwiftCompilerSources/../include/swift/strings.h:16:

If you really want to figure this particular build issue out, I would use that known-good Docker image I linked, build the compiler successfully in there, then extract the equivalent passing compile command and compare it to your failing one. These header includes are so labyrinthine, there's no way you're going to figure it out just by staring at this failing command.

Very likely, you are simply building an untested configuration, as the linux CI only builds with the early-swiftsyntax parser these days, because it is required for macros, and getting back on that tested path will make everything work again.

1 Like
utils/build-script --release-debuginfo > log_without_skip.txt

Will get the same error message here if I do not have any swift in my $PATH.

➜  SwiftProject which swift  
/home/kyle/.swiftbox/toolchain/swift-5.10/usr/bin/swift

After adding one to my $PATH, and run it again. Still the same result.

I am going to set up a fresh Linux env and try it again.

Are you rebuilding "from scratch," ie deleting the previous failed compiler build with rm -rf build/Ninja-RelWithDebInfoAssert/swift-linux-aarch64/? If not, these commands will not override your old config. If fresh builds don't work, try the Docker image, as it may be some oddity of your linux setup.

Yes. I have tried it with rm -rf build/Ninja-RelWithDebInfoAssert/swift-linux-aarch64/ performed. And I also disable sccache. But still fails for the same reason.

make sure you are using a 5.8.1 prebuilt toolchain, if i remember correctly the 5.9.2 toolchain was defective and cannot be used to build a compiler from source. (i have not tried using the 5.10 toolchain yet)

Recreating a clean Ubuntu 22.04 env and use the default build command utils/build-script --release-debuginfo --skip-early-swift-driver --skip-early-swiftsyntax --sccache without a pre-build swift toolchain is now building successfully. :tada:

Thanks everyone for giving the help.

I suspect it may be relevant with the location. (The original workspace build is created on macOS. But I've run utils/update-checkout --scheme release/5.10 again on Linux env to update. Since I am using OrbStack, I'm sharing the same workspace here)

1 Like