Building Swift compiler from source fails when not using Ninja

xwu · 2022-01-17T00:04:26.561Z

[SR-13635] CMark Test fails

opened 01:20PM - 30 Sep 20 UTC

bug

| | | |------------------|-----------------|… |Previous ID | SR-13635 | |Radar | None | |Original Reporter | pushpinderpalsingh (JIRA User) | |Type | Bug | <details> <summary>Environment</summary> master branch, macOS Big Sur Beta, Xcode 12 </details> <details> <summary>Additional Detail from JIRA</summary> | | | |------------------|-----------------| |Votes | 2 | |Component/s | | |Labels | Bug | |Assignee | None | |Priority | Medium | md5: e0f5c41f8df8d01a23a4aebd07ad1f5e </details> **relates to**: * [SR-14724](https://bugs.swift.org/browse/SR-14724) --skip-test-cmark does not skip CTest **Issue Description:** I am trying to compile Swift for a while now. The compilation goes perfectly but some test case fails which results in the process to terminate with an exit status of 65. I am using Xcode 12 ( not beta ) and macOS Big Sur Beta. I am using this command to build the Swift compiler. > {{utils/build-script --skip-build-benchmarks }} > --skip-ios --skip-watchos --skip-tvos --swift-darwin-supported-archs "x86_64" > --cmake-c-launcher="$(which sccache)" --cmake-cxx-launcher="(which sccache)" > --release-debuginfo --test > --xcode And getting the following output (I am just adding the failing test cases) > === BUILD AGGREGATE TARGET RUN_TESTS OF PROJECT cmark WITH CONFIGURATION RelWithDebInfo === > > Check dependencies > > PhaseScriptExecution CMake\\ PostBuild\\ Rules /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/cmark.build/RelWithDebInfo/RUN_TESTS.build/Script-209180F299BC48E5B78D44FC.sh > cd /Volumes/ExtremeSSD/swift-project/cmark > /bin/sh -c /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/cmark.build/RelWithDebInfo/RUN_TESTS.build/Script-209180F299BC48E5B78D44FC.sh > /usr/local/Cellar/cmake/3.18.3/bin/ctest --force-new-ctest-process -C RelWithDebInfo > Test project /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64 > Start 1: api_test > 1/9 Test #1: api_test ......................... Passed 0.02 sec > Start 2: html_normalization > 2/9 Test #2: html_normalization ............... Passed 0.94 sec > Start 3: spectest_library > 3/9 Test #3: spectest_library .................\*\*\*Failed 0.11 sec > Start 4: pathological_tests_library > 4/9 Test #4: pathological_tests_library .......\*\*\*Failed 0.09 sec > Start 5: roundtriptest_library > 5/9 Test #5: roundtriptest_library ............\*\*\*Failed 0.08 sec > Start 6: entity_library > 6/9 Test #6: entity_library ...................\*\*\*Failed 0.06 sec > Start 7: spectest_executable > 7/9 Test #7: spectest_executable ..............\*\*\*Failed 0.08 sec > Start 8: smartpuncttest_executable > 8/9 Test #8: smartpuncttest_executable ........\*\*\*Failed 0.07 sec > Start 9: regressiontest_executable > 9/9 Test #9: regressiontest_executable ........\*\*\*Failed 0.07 sec > > 22% tests passed, 7 tests failed out of 9 > > Total Test time (real) = 1.53 sec > > The following tests FAILED: > 3 - spectest_library (Failed) > 4 - pathological_tests_library (Failed) > 5 - roundtriptest_library (Failed) > 6 - entity_library (Failed) > 7 - spectest_executable (Failed) > 8 - smartpuncttest_executable (Failed) > 9 - regressiontest_executable (Failed) > Errors while running CTest > make: \*\*\* \[RUN_TESTS_buildpart_0\] Error 8 > Command /bin/sh failed with exit code 2 > > === BUILD AGGREGATE TARGET ZERO_CHECK OF PROJECT cmark WITH CONFIGURATION RelWithDebInfo === > > Check dependencies > > PhaseScriptExecution CMake\\ Rules /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/cmark.build/RelWithDebInfo/ZERO_CHECK.build/Script-8A9EE83BA8FD420BB7903C9C.sh > cd /Volumes/ExtremeSSD/swift-project/cmark > /bin/sh -c /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/cmark.build/RelWithDebInfo/ZERO_CHECK.build/Script-8A9EE83BA8FD420BB7903C9C.sh > echo "" > > make -f /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/CMakeScripts/ReRunCMake.make > make\[1\]: \`/Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/CMakeFiles/cmake.check_cache' is up to date. > > - - BUILD FAILED \*\* > > The following build commands failed: > PhaseScriptExecution CMake\\ PostBuild\\ Rules /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/cmark.build/RelWithDebInfo/RUN_TESTS.build/Script-209180F299BC48E5B78D44FC.sh > (1 failure) > ERROR: command terminated with a non-zero exit status 65, aborting Additional configuration information (utils/update-checkout --dump-hashes): > { > > "https-clone-pattern": "https://github.com/%s.git", > > "ssh-clone-pattern": "git@github.com:%s.git", > > "repos": { > > "sourcekit-lsp": { > > "remote": { > > "id": "apple/sourcekit-lsp" > > } > > }, > > "indexstore-db": { > > "remote": { > > "id": "apple/indexstore-db" > > } > > }, > > "llbuild": { > > "remote": { > > "id": "apple/swift-llbuild" > > } > > }, > > "swift-xcode-playground-support": { > > "remote": { > > "id": "apple/swift-xcode-playground-support" > > } > > }, > > "swift-syntax": { > > "remote": { > > "id": "apple/swift-syntax" > > } > > }, > > "swift-corelibs-xctest": { > > "remote": { > > "id": "apple/swift-corelibs-xctest" > > } > > }, > > "swiftpm": { > > "remote": { > > "id": "apple/swift-package-manager" > > } > > }, > > "swift-tools-support-core": { > > "remote": { > > "id": "apple/swift-tools-support-core" > > } > > }, > > "cmake": { > > "platforms": \[ > > "Linux" > > \], > > "remote": { > > "id": "KitWare/CMake" > > } > > }, > > "swift-driver": { > > "remote": { > > "id": "apple/swift-driver" > > } > > }, > > "yams": { > > "remote": { > > "id": "jpsim/Yams" > > } > > }, > > "icu": { > > "platforms": \[ > > "Linux" > > \], > > "remote": { > > "id": "unicode-org/icu" > > } > > }, > > "swift-corelibs-foundation": { > > "remote": { > > "id": "apple/swift-corelibs-foundation" > > } > > }, > > "llvm-project": { > > "remote": { > > "id": "apple/llvm-project" > > } > > }, > > "swift-argument-parser": { > > "remote": { > > "id": "apple/swift-argument-parser" > > } > > }, > > "ninja": { > > "remote": { > > "id": "ninja-build/ninja" > > } > > }, > > "swift-integration-tests": { > > "remote": { > > "id": "apple/swift-integration-tests" > > } > > }, > > "swift": { > > "remote": { > > "id": "apple/swift" > > } > > }, > > "swift-stress-tester": { > > "remote": { > > "id": "apple/swift-stress-tester" > > } > > }, > > "cmark": { > > "remote": { > > "id": "apple/swift-cmark" > > } > > }, > > "swift-format": { > > "remote": { > > "id": "apple/swift-format" > > } > > }, > > "swift-corelibs-libdispatch": { > > "remote": { > > "id": "apple/swift-corelibs-libdispatch" > > } > > } > > }, > > "branch-schemes": { > > "repro": { > > "repos": { > > "sourcekit-lsp": "241744da9ba73ab9951e7fb70a75e04779642302", > > "swift-syntax": "94fc5ae3f34fac87380756b9c17ea7c6752a227b", > > "llbuild": "3f236e84ebf77fbc4a1c6a41500ffa6b6709180a", > > "swift-xcode-playground-support": "88043d7d320f92598efb39408c3f4b1903a4fff6", > > "indexstore-db": "b7cc40f64a72ce1368b9b2f7bcfa9d9950698e0b", > > "swift-corelibs-xctest": "d728b6456d453be212038bd8fe01ceebf5b7ce4f", > > "swiftpm": "726eff2c35fc3620bc0afde6befecc9b0246b0b5", > > "swift-tools-support-core": "cf5db868eeed2f41ddf1d7ac9b202b5738d67dc1", > > "cmake": "skip", > > "swift-driver": "5a7b75cac10aeee72f89cb965754167739dcd497", > > "yams": "81a65c4069c28011ee432f2858ba0de49b086677", > > "icu": "skip", > > "swift-corelibs-foundation": "e9d8347e64ece3de453d617887af7229f73358d8", > > "llvm-project": "40349c774225a0989b6a70bd16122b7d486c1dfb", > > "swift-argument-parser": "15351c1cd009eba0b6e438bfef55ea9847a8dc4a", > > "ninja": "a1f879b29c9aafe6a2bc0ba885701f8f4f19f772", > > "swift-integration-tests": "3ceb61d33ea8885938eace0add830d143a3059cd", > > "swift": "44c8052bb53108464a43ac8d8f3297e487439d7d", > > "swift-stress-tester": "4d8f8a790d32a3ced310c6b68c46ca8e6f71f52c", > > "cmark": "9c8096a23f44794bde297452d87c455fc4f76d42", > > "swift-format": "d4bba6e22891ff1813e8267e36f2b00307684366", > > "swift-corelibs-libdispatch": "f13ea5dcc055e5d2d7c02e90d8c9907ca9dc72e1" > > }, > > "aliases": \[ > > "repro" > > \] > > } > > } > > }**%**

CPPLib_log error when building with Xcode Compiler

I'm trying to build Swift with Xcode following the Getting Started Guide. I'm running the following command: utils/build-script --verbose-build --skip-build-benchmarks \ --skip-ios --skip-watchos --skip-tvos --swift-darwin-supported-archs "$(uname -m)" \--sccache --release-debuginfo --swift-disable-dead-stripping --test \ --xcode --skip-test-cmark The error I get is the following: User/swift-project/swift/validation-test/BuildSystem/swift-cmake/Klass.swift:14:5: error: cannot find 'CPPLi…

Note the following from that last thread:

CPPLib_log error when building with Xcode

The Xcode generator has many issues and isn't the normal build people use (we generally use ninja).

philipturner · 2022-01-17T00:11:26.897Z

Thanks. The Xcode build without tests just succeeded, but the way.

SR-13635 is exactly the set of tests that failed for me.

philipturner · 2022-01-17T00:22:06.661Z

Now I'm getting the following error when I press Cmd + B with the Xcode project open:

clang-13: error: no such file or directory: '/Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/llvm-macosx-arm64/Debug/lib/libLLVMSupport.a'
clang-13: error: no such file or directory: '/Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/llvm-macosx-arm64/Debug/lib/libLLVMDemangle.a'
Command Ld failed with a nonzero exit code

Full log

Ld /Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/swift-macosx-arm64/Debug/bin/swift-serialize-diagnostics normal (in target 'swift-serialize-diagnostics' from project 'Swift')
cd /Users/philipturner/Documents/Swift-Compiler/swift-project/swift
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2022-01-09-a.xctoolchain/usr/bin/clang++ -target arm64-apple-macos12.1 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk -L/Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/swift-macosx-arm64/Debug/bin -L/Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/llvm-macosx-arm64/RelWithDebInfo/lib/Debug -L/Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/llvm-macosx-arm64/RelWithDebInfo/lib -F/Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/swift-macosx-arm64/Debug/bin -filelist /Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/swift-macosx-arm64/tools/swift-serialize-diagnostics/Swift.build/Debug/swift-serialize-diagnostics.build/Objects-normal/arm64/swift-serialize-diagnostics.LinkFileList -Xlinker -object_path_lto -Xlinker /Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/swift-macosx-arm64/tools/swift-serialize-diagnostics/Swift.build/Debug/swift-serialize-diagnostics.build/Objects-normal/arm64/swift-serialize-diagnostics_lto.o -stdlib=libc++ -target arm64-apple-macosx10.9 -Xlinker /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/lib/libz.tbd -Xlinker /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk/usr/lib/libcurses.tbd -Xlinker /Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/llvm-macosx-arm64/Debug/lib/libLLVMDemangle.a -Wl,-search_paths_first -Wl,-headerpad_max_install_names /Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/swift-macosx-arm64/Debug/lib/libswiftLocalization.a /Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/llvm-macosx-arm64/Debug/lib/libLLVMSupport.a -lm /Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/llvm-macosx-arm64/Debug/lib/libLLVMDemangle.a -Xlinker -no_adhoc_codesign -Xlinker -dependency_info -Xlinker /Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/swift-macosx-arm64/tools/swift-serialize-diagnostics/Swift.build/Debug/swift-serialize-diagnostics.build/Objects-normal/arm64/swift-serialize-diagnostics_dependency_info.dat -o /Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-RelWithDebInfoAssert/swift-macosx-arm64/Debug/bin/swift-serialize-diagnostics

Any ideas how to fix this? Should I edit with Xcode but still compile with Ninja from command line? The error happens when I try each of the recommended schemes in GettingStarted.md.

LucianoPAlmeida · 2022-01-18T00:36:20.088Z

Not entirely sure, but I think that is a problem with LLVM paths when building llvm in a different mode than swift (in this case llvm RelWithDebugInfo and swift in Debug) note that you are building with --release-debuginfo so llvm libraries will probably be at RelWithDebugInfo directory but error is that it doesn't find them in the Debug directory(which is swift build mode). You probably find those libraries in the RelWithDebugInfo folder.

A work around you may be able to do it to build all in debug mode if the goal is only to build, this command SKIP_XCODE_VERSION_CHECK=1 ./swift/utils/build-script --xcode --debug --skip-build-benchmarks --swift-darwin-supported-archs "$(uname -m)" --sccache --skip-ios --skip-tvos --skip-watchos should work.

philipturner · 2022-01-18T12:19:03.093Z

That seems to be the problem! All the executables I need are in the RelWithDebugInfo/lib directory instead of the Debug directory, which doesn't even have a lib folder. I'll test your command soon.

philipturner · 2022-01-18T13:15:26.255Z

Now I got this problem, where the build tried to reference the lib/Debug folder instead of the plain Debug folder:

clang: error: no such file or directory: '/Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-DebugAssert/swift-macosx-arm64/lib/Debug/libswiftCompilerModules-bootstrapping1.a'
Command Ld failed with a nonzero exit code

I opened up the Xcode project and pressed Cmd + B. It seems to have encountered that error just as a warning and continued with the build, so hopefully it will work!

Update: the same error happened later on in the build. I just wasted 30% of my MBP's battery on that build when it already started at 80%, and I have a full day ahead!

clang-13: error: no such file or directory: '/Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-DebugAssert/swift-macosx-arm64/lib/Debug/libswiftCompilerModules-bootstrapping1.a'

LucianoPAlmeida · 2022-01-19T00:41:21.964Z

This is an issue that should be fixed by now, see [SR-15705] Compiler build failed (PhaseScriptExecution CMake PostBuild\ Rules) · Issue #57984 · apple/swift · GitHub for more info. I would suggest to try pass --bootstrapping=off

ksluder · 2022-01-19T15:38:43.059Z

I don’t follow… if it’s fixed, then why is it necessary to pass --bootstrapping=off?

LucianoPAlmeida · 2022-01-19T20:40:36.483Z

Bootstrapping was off by default in Xcode build because it was not supported so it would never try bootstrapping when --xcode and it should build just fine. Fix printing Xcode version from CMake when building with Xcode by cbjeukendrup · Pull Request #40870 · apple/swift · GitHub enabled bootstrapping for Xcode, but by the error message something may not be working as expected since it failing. So what I'm simply suggesting is that if the goal is build the compiler with --xcode enforce --bootstrapping=off should do it as a workaround at least util we figure out the cause of the bootstrapping issue.

Best Regards,
Luciano

philipturner · 2022-01-19T22:01:11.851Z

It's still failing. From the command line, I tried this:

./swift/utils/build-script --xcode --debug --skip-build-benchmarks --swift-darwin-supported-archs "$(uname -m)" --sccache --skip-ios --skip-tvos --skip-watchos --bootstrapping=off

And ended up with this:

clang: error: no such file or directory: '/Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-DebugAssert/swift-macosx-arm64/lib/Debug/libswiftCompilerModules-bootstrapping1.a'
Command Ld failed with a nonzero exit code

** BUILD FAILED **


The following build commands failed:
	Ld /Users/philipturner/Documents/Swift-Compiler/swift-project/build/Xcode-DebugAssert/swift-macosx-arm64/bootstrapping1/Debug/bin/swift-frontend normal (in target 'swift-frontend-bootstrapping1' from project 'Swift')
(1 failure)
ERROR: command terminated with a non-zero exit status 65, aborting

Did I do anything wrong? Or do you have the capacity to test it out yourself and reproduce my error?

ksluder · 2022-01-19T22:25:16.913Z

I successfully built with --bootstrapping=off this morning. I did have to nuke my build directory first.

philipturner · 2022-01-19T22:28:16.347Z

I'll try doing that. In the meantime, what machine specifications (platform, CPU architecture) were you using?

ksluder · 2022-01-19T22:28:40.356Z

M1 Mac mini, building for arm64.

philipturner · 2022-01-19T22:29:29.756Z

I have M1 Max Macbook Pro. You only had to nuke the sub-folder for Xcode-DebugAssert right? I'm free to keep the others like Xcode-RelWithDebInfoAssert?

ksluder · 2022-01-19T22:30:22.005Z

I deleted the entire build directory. I don’t know if there’s a more limited subset you can delete.

LucianoPAlmeida · 2022-01-19T22:52:18.126Z

Before deleting all there are a couple of options to try first because delete all will make you need to rebuild everything again: So you can try using --reconfigure to force cmake to re-generate everything or delete only swift-macosx-arm64 because llvm, cmark and driver probably don't need rebuilding =]

philipturner · 2022-01-19T22:53:21.031Z

Already deleted the Xcode-DebugAssert, although I just moved the other two (Xcode release, Ninja release) out of the directory. Xcode-DebugAssert is in the trash, so I could retain it. At this point, I'd rather not mess with the build.

philipturner · 2022-01-19T23:28:46.622Z

It built successfully both from the command line and in Xcode!

peter1234 · 2023-10-31T07:54:30.556Z

Hello, what is your compiler instruction @philipturner

peter1234 · 2023-10-31T07:55:39.737Z

@philipturner Because I had the same error as you