Failing Tests while Compiling Swift on Xcode 12 with MacOS Big Sur Beta

swiftlysingh · 2020-09-29T11:51:36.683Z

I am trying to compile Swift for a while now. The compilation goes perfectly but some test case fails which results in the process to terminate with an exit status of 65.
I am using Xcode 12 ( not beta ) and macOS Big Sur Beta. I am using this command to build the Swift compiler.

utils/build-script --skip-build-benchmarks \
--skip-ios --skip-watchos --skip-tvos --swift-darwin-supported-archs "x86_64"
--cmake-c-launcher="$(which sccache)" --cmake-cxx-launcher="$(which sccache)"
--release-debuginfo --test
--xcode

And getting the following output (I am just adding the failing test cases)

=== BUILD AGGREGATE TARGET RUN_TESTS OF PROJECT cmark WITH CONFIGURATION RelWithDebInfo ===

Check dependencies

PhaseScriptExecution CMake\ PostBuild\ Rules /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/cmark.build/RelWithDebInfo/RUN_TESTS.build/Script-209180F299BC48E5B78D44FC.sh
cd /Volumes/ExtremeSSD/swift-project/cmark
/bin/sh -c /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/cmark.build/RelWithDebInfo/RUN_TESTS.build/Script-209180F299BC48E5B78D44FC.sh
/usr/local/Cellar/cmake/3.18.3/bin/ctest --force-new-ctest-process -C RelWithDebInfo
Test project /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64
Start 1: api_test
1/9 Test #1: api_test ......................... Passed 0.02 sec
Start 2: html_normalization
2/9 Test #2: html_normalization ............... Passed 0.94 sec
Start 3: spectest_library
3/9 Test #3: spectest_library .................***Failed 0.11 sec
Start 4: pathological_tests_library
4/9 Test #4: pathological_tests_library .......***Failed 0.09 sec
Start 5: roundtriptest_library
5/9 Test #5: roundtriptest_library ............***Failed 0.08 sec
Start 6: entity_library
6/9 Test #6: entity_library ...................***Failed 0.06 sec
Start 7: spectest_executable
7/9 Test #7: spectest_executable ..............***Failed 0.08 sec
Start 8: smartpuncttest_executable
8/9 Test #8: smartpuncttest_executable ........***Failed 0.07 sec
Start 9: regressiontest_executable
9/9 Test #9: regressiontest_executable ........***Failed 0.07 sec

22% tests passed, 7 tests failed out of 9

Total Test time (real) = 1.53 sec

The following tests FAILED:
3 - spectest_library (Failed)
4 - pathological_tests_library (Failed)
5 - roundtriptest_library (Failed)
6 - entity_library (Failed)
7 - spectest_executable (Failed)
8 - smartpuncttest_executable (Failed)
9 - regressiontest_executable (Failed)
Errors while running CTest
make: *** [RUN_TESTS_buildpart_0] Error 8
Command /bin/sh failed with exit code 2

=== BUILD AGGREGATE TARGET ZERO_CHECK OF PROJECT cmark WITH CONFIGURATION RelWithDebInfo ===

Check dependencies

PhaseScriptExecution CMake\ Rules /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/cmark.build/RelWithDebInfo/ZERO_CHECK.build/Script-8A9EE83BA8FD420BB7903C9C.sh
cd /Volumes/ExtremeSSD/swift-project/cmark
/bin/sh -c /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/cmark.build/RelWithDebInfo/ZERO_CHECK.build/Script-8A9EE83BA8FD420BB7903C9C.sh
echo ""

make -f /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/CMakeScripts/ReRunCMake.make
make[1]: `/Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/CMakeFiles/cmake.check_cache' is up to date.

** BUILD FAILED **

The following build commands failed:
PhaseScriptExecution CMake\ PostBuild\ Rules /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/cmark.build/RelWithDebInfo/RUN_TESTS.build/Script-209180F299BC48E5B78D44FC.sh
(1 failure)
ERROR: command terminated with a non-zero exit status 65, aborting

typesanitizer · 2020-09-29T17:20:34.408Z

Those are CMark tests (CMark is one of our dependencies); you can skip them using --skip-test-cmark to get unblocked.

I wonder if this is related to [SR-13534] html_normalization test hangs · Issue #55971 · apple/swift · GitHub . Could you file a bug report on bugs.swift.org with information from the post and additional configuration information (utils/update-checkout --dump-hashes).

(Also, I think you are missing $ before the (which sccache), I don't think substitution works without the $.)

benrimmington · 2020-09-29T19:32:12.403Z

typesanitizer:

(Also, I think you are missing $ before the (which sccache) , I don't think substitution works without the $ .)

The $ exists in the raw text.

swiftlysingh · 2020-09-30T12:22:20.740Z

I have filed the bug on swift jira. Now, I am gonna try to compile swift by skipping cmark tests and get back to you.

swiftlysingh · 2020-09-30T14:05:55.340Z

I skipped the cmark tests but still my build failed. I am getting the following output

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 

4 warning(s) in tests
********************
Failed Tests (20):
  Swift-Unit :: runtime/RelWithDebInfo/SwiftRuntimeTests/RefcountingTest.unknown_retain_release_n
  Swift-Unit :: runtime/RelWithDebInfo/SwiftRuntimeTests/RefcountingTest.unowned_retain_release_n
  Swift-Unit :: runtime/RelWithDebInfo/SwiftRuntimeTests/RefcountingTest.unowned_retain_release_n_overflow
  Swift-Unit :: runtime/RelWithDebInfo/SwiftRuntimeTests/RefcountingTest.nonatomic_unknown_retain_release_n
  Swift-Unit :: runtime/RelWithDebInfo/SwiftRuntimeTests/RefcountingTest.immortal_retain_release
  Swift(macosx-x86_64) :: IRGen/prespecialized-metadata/class-class-flags-run.swift
  Swift(macosx-x86_64) :: IRGen/prespecialized-metadata/enum-extradata-run.swift
  Swift(macosx-x86_64) :: IRGen/prespecialized-metadata/struct-extradata-run.swift
  Swift(macosx-x86_64) :: IRGen/prespecialized-metadata/enum-trailing-flags-run.swift
  Swift(macosx-x86_64) :: IRGen/prespecialized-metadata/struct-multi-conformance.swift
  Swift(macosx-x86_64) :: IRGen/prespecialized-metadata/struct-outmodule-run.swift
  Swift(macosx-x86_64) :: IRGen/prespecialized-metadata/struct-trailing-flags-run.swift
  Swift(macosx-x86_64) :: ModuleInterface/ModuleCache/force-module-loading-mode-archs.swift
  Swift(macosx-x86_64) :: ModuleInterface/ModuleCache/force-module-loading-mode-framework.swift
  Swift(macosx-x86_64) :: ModuleInterface/ModuleCache/force-module-loading-mode.swift
  Swift(macosx-x86_64) :: Interpreter/SDK/libc.swift
  Swift(macosx-x86_64) :: Index/Store/output-failure.swift
  Swift(macosx-x86_64) :: remote-run/upload-stderr.test-sh
  Swift(macosx-x86_64) :: remote-run/upload-and-download.test-sh
  Swift(macosx-x86_64) :: Reflection/capture_descriptors.sil


Testing Time: 1240.71s
  Unsupported      :  210
  Passed           : 6598
  Expectedly Failed:   23
  Failed           :   20
make: *** [/Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/swift-macosx-x86_64/test/CMakeFiles/check-swift-macosx-x86_64] Error 1

** BUILD FAILED **


The following build commands failed:
	PhaseScriptExecution CMake\ Rules /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/swift-macosx-x86_64/test/Swift.build/RelWithDebInfo/check-swift-macosx-x86_64.build/Script-B96A5872945F49F0A564B464.sh
(1 failure)
ERROR: command terminated with a non-zero exit status 65, aborting

Jumhyn · 2020-09-30T14:17:22.627Z

I've had similar issues with unexpected failures when running tests via an Xcode build. I'd recommend running tests via ninja!

swiftlysingh · 2020-09-30T14:24:24.943Z

How should I go about it? You are saying to build with xcode and run tests with ninja right?

Jumhyn · 2020-09-30T14:33:25.961Z

I don't know if it's possible to run tests with ninja from an Xcode build. I generally maintain a separate ninja build that I use for running tests.

swiftlysingh · 2020-10-01T07:38:25.302Z

I am now building with ninja using the following command

utils/build-script --skip-build-benchmarks --skip-test-cmark
--skip-ios --skip-watchos --skip-tvos --swift-darwin-supported-archs "x86_64"
--cmake-c-launcher="$(which sccache)" --cmake-cxx-launcher="$(which sccache)"
--release-debuginfo --test

But getting the following error

4 warning(s) in tests

FAILED: test/CMakeFiles/check-swift-macosx-x86_64

cd /Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test && /usr/local/Cellar/cmake/3.18.3/bin/cmake -E remove_directory /Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./swift-test-results/x86_64-apple-macosx10.9 && /usr/local/Cellar/cmake/3.18.3/bin/cmake -E make_directory /Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./swift-test-results/x86_64-apple-macosx10.9 && /Applications/Xcode.app/Contents/Developer/usr/bin/python3 /Volumes/ExtremeSSD/swift-project/llvm-project/llvm/utils/lit/lit.py --incremental -sv -j 8 --xunit-xml-output=/Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./swift-test-results/x86_64-apple-macosx10.9/lit-tests.xml --param differentiable_programming --param concurrency --param swift_test_subset=primary --param swift_test_mode=optimize_none /Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64

ninja: build stopped: subcommand failed.

ERROR: command terminated with a non-zero exit status 1, aborting

Is compiling a programming language this involved or there is a problem with my system?

typesanitizer · 2020-10-01T16:23:07.998Z

swiftlysingh:

FAILED: test/CMakeFiles/check-swift-macosx-x86_64

It's not clear what failed from this... is it the same 20 tests as earlier?

From my limited experience, both compilers and C++ projects do tend to grow hairy build systems. Compilers also have a bunch of cross-platform differences, so the testing infrastructure needs to be flexible enough to accommodate that (and grows complex due to this requirement).

benn · 2020-10-07T04:32:46.069Z

I'm getting this same error on Catalina with Xcode 12. @swiftlysingh were you able to get yours to work?

Victor_Guerra · 2020-10-07T07:30:04.133Z

hi all!

I had a look at the CMark tests that fail. Within the build directory, there is a log file with all details about passing and failing tests ( .../swift/compiler/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/Testing/Temporary/LastTest.log ).

The first test that fails is spectest_library, and the issue is that the library dir passed to the test to find libcmark.dyl is not right. The command that launches the test is:

...python3 .../swift/compiler/cmark/test/spec_tests.py --no-normalize --spec .../swift/compiler/cmark/test/spec.txt --library-dir .../swift/compiler/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/testdir/../src

but the libcmark.dyl is located in ../swift/compiler/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/testdir/../src/RelWithDebInfo. Passing this path will make the test pass.

../python3 /usr/local/swift/compiler/cmark/test/spec_tests.py --no-normalize --spec /usr/local/swift/compiler/cmark/test/spec.txt --library-dir /usr/local/swift/compiler/build/Xcode-RelWithDebInfoAssert/cmark-macosx-x86_64/testdir/../src/RelWithDebInfo
650 passed, 0 failed, 0 errored, 0 skipped

What I did not find yet is why the lib file is being produced within the RelWithDebInfo dir and not where the tests expect it to be. Anyone has an idea why that is?

benn · 2020-10-07T23:58:46.359Z

Nice find, don't know why that's the case. Moving the files out of RelWithDebInfo made my cmark tests pass. Now I'm getting the Swift-Unit/Swift(macosx-x86_64) tests failures mentioned above.

swiftlysingh · 2020-10-19T15:29:09.796Z

I am still not able to pass all tests successfully. The following command is failing when building with Xcode.

	PhaseScriptExecution CMake\ Rules /Volumes/ExtremeSSD/swift-project/build/Xcode-RelWithDebInfoAssert/swift-macosx-x86_64/test/Swift.build/RelWithDebInfo/check-swift-macosx-x86_64.build/Script-560842E5FBAC45A5B7F9C421.sh

What about you?

benn · 2020-10-31T20:08:18.218Z

Same, I've been unable to. I ended up just building with Ninja.

swiftlysingh · 2020-11-03T08:25:14.918Z

Did it work? I am having trouble building with ninja also.

benn · 2020-11-11T01:18:15.085Z

Ya, didn't run into any errors with Ninja. Maybe try again when Big Sur's released on Thursday?

swiftlysingh · 2020-11-24T11:05:00.320Z

I have been trying for weeks but it isn't working my clone is completely up to date and have even cleaned it a couple of times. All dependencies are installed as expected. I have even tried building it with both Xcode and ninja with no success. What commands are you running for ninja? I am using the commands from the getting started guide on Github with

--skip-test-cmark

Victor_Guerra · 2020-11-24T11:41:37.720Z

For ninja I am using the following:

SKIP_XCODE_VERSION_CHECK=1 swift/utils/build-script --skip-build-benchmarks \
  --skip-ios --skip-watchos --skip-tvos \
  --sccache --swift-darwin-supported-archs "x86_64" \
  --release-debuginfo --test

I need to skip xcode version checks b/c I have Xcode beta 12.3

swiftlysingh · 2020-11-25T17:09:00.671Z

No luck for me. I ran

utils/build-script --skip-build-benchmarks \
  --skip-ios --skip-watchos --skip-tvos \
  --sccache --swift-darwin-supported-archs "x86_64" \
  --release-debuginfo --test

That caused my build to fail the following output:

FAILED: test/CMakeFiles/check-swift-macosx-x86_64

cd /Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test && /usr/local/Cellar/cmake/3.19.0_1/bin/cmake -E remove_directory /Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./swift-test-results/x86_64-apple-macosx10.9 && /usr/local/Cellar/cmake/3.19.0_1/bin/cmake -E make_directory /Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./swift-test-results/x86_64-apple-macosx10.9 && /Applications/Xcode.app/Contents/Developer/usr/bin/python3 /Volumes/ExtremeSSD/swift-project/llvm-project/llvm/utils/lit/lit.py --incremental -sv -j 8 --xunit-xml-output=/Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./swift-test-results/x86_64-apple-macosx10.9/lit-tests.xml --param differentiable_programming --param concurrency --param swift_test_subset=primary --param swift_test_mode=optimize_none /Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64

ninja: build stopped: subcommand failed.

Then I tried skipping cmark tests with following command

 utils/build-script --skip-build-benchmarks \           
  --skip-ios --skip-watchos --skip-tvos \
   --sccache --swift-darwin-supported-archs "x86_64" \
   --release-debuginfo --test --skip-test-cmark

But still my build failed with following output

FAILED: test/CMakeFiles/check-swift-macosx-x86_64 
cd /Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test && /usr/local/Cellar/cmake/3.19.0_1/bin/cmake -E remove_directory /Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./swift-test-results/x86_64-apple-macosx10.9 && /usr/local/Cellar/cmake/3.19.0_1/bin/cmake -E make_directory /Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./swift-test-results/x86_64-apple-macosx10.9 && /Applications/Xcode.app/Contents/Developer/usr/bin/python3 /Volumes/ExtremeSSD/swift-project/llvm-project/llvm/utils/lit/lit.py --incremental -sv -j 8 --xunit-xml-output=/Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./swift-test-results/x86_64-apple-macosx10.9/lit-tests.xml --param differentiable_programming --param concurrency --param swift_test_subset=primary --param swift_test_mode=optimize_none /Volumes/ExtremeSSD/swift-project/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64
ninja: build stopped: subcommand failed.
ERROR: command terminated with a non-zero exit status 1, aborting