Unable to compile lib_InternalSwiftSyntaxParser myself from latest commit

I'm trying to see whether I am able to compile a version of lib_InternalSwiftSyntaxParser to embed in my iOS application. Been trying for a while. According to the instruction in the repo, I should follow the following steps:

git clone https://github.com/apple/swift.git
./swift/utils/update-checkout --clone
./swift/utils/build-parser-lib --release --no-assertions --build-dir /tmp/parser-lib-build

For iOS:

./swift/utils/build-parser-lib --release --no-assertions --build-dir /tmp/parser-lib-build-iossim --host iphonesimulator --architectures x86_64
./swift/utils/build-parser-lib --release --no-assertions --build-dir /tmp/parser-lib-build-ios --host iphoneos --architectures arm64

The cloning and the dependencies updates works just fine, however, the build script for both macOS and iOS don't work and fails with the same issue

-- Detecting C compiler ABI info
CMake Error:
  Running

   '/usr/local/bin/ninja' '-C' '/private/tmp/parser-lib-build/x86_64/obj/CMakeFiles/CMakeTmp' '-t' 'cleandead'

  failed with:

   ninja: error: build.ninja:122: unknown target 'all'

  default all

             ^ near here



CMake Error:
  Running

   '/usr/local/bin/ninja' '-C' '/private/tmp/parser-lib-build/x86_64/obj/CMakeFiles/CMakeTmp' '-t' 'recompact'

  failed with:

   ninja: error: build.ninja:122: unknown target 'all'

  default all

             ^ near here



CMake Error at /Applications/CMake.app/Contents/share/cmake-3.18/Modules/CMakeDetermineCompilerABI.cmake:48 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /Applications/CMake.app/Contents/share/cmake-3.18/Modules/CMakeTestCCompiler.cmake:26 (CMAKE_DETERMINE_COMPILER_ABI)
  CMakeLists.txt:49 (project)


-- Configuring incomplete, errors occurred!

Any idea for what could be happening?

macOS 11 beta 1 on Intel
Using toolchain from Xcode 12 beta 1 (12A6159)
Swift master branch was pulled on 6-29

@akyrtzi

I'm not seeing such an issue. This is probably not specific to the parser library, are you able to configure and build the swift compiler? What version of CMake and ninja are you using?

@akyrtzi

Hello, thanks for the response.

tomshen@Toms-Mac-Pro ~ % cmake --version
cmake version 3.18.0-rc2

CMake suite maintained and supported by Kitware (kitware.com/cmake).
tomshen@Toms-Mac-Pro ~ % ninja --version
1.10.0

I didn't try the full build script yet. Before I start that (and do the hour long wait), I have a question, if I build using:

swift/utils/build-script --swiftsyntax --swiftpm --llbuild

Does that also generates the swift syntax parser library along inside the built toolchain (or even for iOS as well?)?

Thanks

It will generate the parser library but for macOS only.

@akyrtzi

Just tried a full build with:

 swift/utils/build-script --swiftsyntax --swiftpm --llbuild

Seems to work fine.

Any other way I can build this library for iOS? I'm trying to build a small app that utilizes this library. It would be a pity if I have to resort to using regex based code lexer.

I'm not sure what the issue with the CMake configure is, I'd recommend to investigate what is the difference in your CMake invocation from build-script vs build-parser-lib, e.g. the script invocations may somehow end up picking up different CMake or ninja binaries.

I'm not too familiar with the build script. It doesn't look like it outputted the invocation of cmake. It simply logs the error from the build-parser-lib script. Any flag that I need to pass to show it?

You can pass -v for more verbose output.

Here are the output with -v flag:

tomshen@Toms-Mac-Pro swift-2 % ./swift/utils/build-parser-lib -v --release --no-assertions --build-dir /tmp/parser-lib-build
Symlinking '/Volumes/960GB-SSD/swift-2/swift' to '/Volumes/960GB-SSD/swift-2/llvm-project/swift'
+ ln -s /Volumes/960GB-SSD/swift-2/swift /Volumes/960GB-SSD/swift-2/llvm-project/swift
+ mkdir -p /tmp/parser-lib-build/x86_64/obj
+ pushd /tmp/parser-lib-build/x86_64/obj
+ /usr/local/bin/cmake -G Ninja -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 -DSWIFT_DARWIN_DEPLOYMENT_VERSION_OSX=10.12 -DLLVM_HOST_TRIPLE:STRING=x86_64-apple-macosx10.12 -DLLVM_TARGET_ARCH=x86_64 -DSWIFT_HOST_VARIANT=macosx -DSWIFT_HOST_VARIANT_SDK=OSX -DSWIFT_HOST_VARIANT_ARCH=x86_64 '-DCMAKE_C_FLAGS=-arch x86_64' '-DCMAKE_CXX_FLAGS=-arch x86_64' -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -DCMAKE_INSTALL_PREFIX:PATH=/ '-DLLVM_ENABLE_PROJECTS=clang;swift' -DLLVM_EXTERNAL_PROJECTS=swift -DSWIFT_BUILD_ONLY_SYNTAXPARSERLIB=TRUE -DSWIFT_BUILD_PERF_TESTSUITE=NO -DSWIFT_INCLUDE_DOCS=NO -DSWIFT_BUILD_REMOTE_MIRROR=FALSE -DSWIFT_BUILD_DYNAMIC_STDLIB=FALSE -DSWIFT_BUILD_STATIC_STDLIB=FALSE -DSWIFT_BUILD_DYNAMIC_SDK_OVERLAY=FALSE -DSWIFT_BUILD_STATIC_SDK_OVERLAY=FALSE -DLLVM_ENABLE_LIBXML2=FALSE -DLLVM_ENABLE_LIBEDIT=FALSE -DLLVM_ENABLE_TERMINFO=FALSE -DLLVM_ENABLE_ZLIB=FALSE -DCMARK_MAIN_INCLUDE_DIR=/Volumes/960GB-SSD/swift-2/cmark -DCMARK_BUILD_INCLUDE_DIR=/tmp/parser-lib-build/x86_64/obj/cmark -DLLVM_INCLUDE_TESTS=FALSE -DCLANG_INCLUDE_TESTS=FALSE -DSWIFT_INCLUDE_TESTS=FALSE /Volumes/960GB-SSD/swift-2/llvm-project/llvm
-- The C compiler identification is AppleClang 12.0.0.12000022
-- The CXX compiler identification is AppleClang 12.0.0.12000022
-- The ASM compiler identification is Clang
-- Found assembler: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Detecting C compiler ABI info
<Here shows the error message in the original post>

I also checked CMakeError.log, there seems to be some linker errors, not sure if it's related or not:

Compiling the C compiler identification source file "CMakeCCompilerId.c" failed.
Compiler: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc 
Build flags: -arch;x86_64
Id flags:  

The output was:
1
ld: library not found for -lSystem
clang: error: linker command failed with exit code 1 (use -v to see invocation)


Compiling the CXX compiler identification source file "CMakeCXXCompilerId.cpp" failed.
Compiler: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ 
Build flags: -arch;x86_64
Id flags:  

The output was:
1
ld: library not found for -lc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Hi @TomShen1234, I was able to reproduce this issue using cmake-3.18-r3, but it builds correctly with the latest released version cmake-3.17.3.

@compnerd any idea what might have caused this regression in cmake-3.18? I'm not sure where to find the build.ninja file that is failing during configure to be able to see what it contains.

Not sure if I have too much to add. A couple of notes:

  1. CMakeError.txt is expected to contain many errors. Its just a collection of the failed commands that ran during the detection (and the expectation is that there will be failures, its just trying different things to learn about the environment)

  2. There is one piece of detail that I think hasn't been discussed yet:

-- Found assembler: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc

Why is it falling back to cc? I would expect that to be clang not cc (cc is the traditional Unix name for the C compiler). This is yanked from a random CI run, but this is what I would expect:

-- The C compiler identification is AppleClang 12.0.0.12000022
-- Check for working C compiler: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
-- Check for working C compiler: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -- works

I'm left wondering if CMake is correctly identifying the compiler or not. That is probably the first thread that I would chase, trying to figure out why its not picking up clang/clang++.

Why is it falling back to cc ?

Note: I see the same behaviour with cmake-3.17.3 when running build-parser-lib (in which case it seems to work).

In build-script we explicitly pass CMAKE_C[XX]_COMPILER, but in build-parser-lib we are not. If I change the script to explicitly set clang, it still fails the same way.

Okay, with --debug-trycompile I can see that indeed the ninja file is incorrect. It contains:

default all

But it is missing

build all: ....

Hmm, interesting. If there are no build rules, does that imply that no target was added?

Sorry should have been clearer. There is

build /tmp/parser-lib-build-ios/x86_64-tblgen/CMakeFiles/CMakeTmp/all: ....

and other targets, but it's missing the build all: phony /tmp/parser-lib-build-ios/x86_64-tblgen/CMakeFiles/CMakeTmp/all that would tie it all together

@blangmuir Can confirm that cmake 3.17.3 is able to build the parser lib successfully. Thanks!

Let me know if anyone needs anymore tests from me!