Building the Swift Benchmark Suite Independently

Development Compiler
1

I ran into some trouble while trying to build the Swift Benchmark Suite Independently.

Since I didn't want to clone the entire Swift repo, I did a shallow clone:

$ git init SwiftBenchmarkSuite
$ cd SwiftBenchmarkSuite/
$ git remote add origin https://github.com/apple/swift.git
$ git config core.sparsecheckout true
$ echo "benchmark/*" >> .git/info/sparse-checkout
$ git pull --depth=1 origin master

Which was quick and seems to work fine.

So now I wanted to build, following the instructions (see above link):

$ sudo xcode-select -s /Applications/Xcode.app/
$ cd SwiftBenchmarkSuite/
$ cd benchmark/
$ mkdir build
$ cd build
$ cmake ..
-bash: cmake: command not found

I downloaded and installed:
https://cmake.org/files/v3.11/cmake-3.11.3-Darwin-x86_64.dmg
(I'm trying to keep this computer "clean", without brew, macports etc, so I went with the dmg)

$ /Applications/CMake.app/Contents/bin/cmake ..
-- The C compiler identification is AppleClang 9.1.0.9020039
-- The CXX compiler identification is AppleClang 9.1.0.9020039
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at cmake/modules/SwiftBenchmarkUtils.cmake:61 (message):
  Unable to find Clang driver
Call Stack (most recent call first):
  cmake/modules/AddSwiftBenchmarkSuite.cmake:27 (runcmd)
  CMakeLists.txt:253 (configure_build)


-- Configuring incomplete, errors occurred!
See also "/Users/jens/sf/SwiftBenchmarkSuite/benchmark/build/CMakeFiles/CMakeOutput.log".

I don't know what is meant by "Unable to find Clang driver", as I do have clang:

$ clang --version
Apple LLVM version 9.1.0 (clang-902.0.39.2)
Target: x86_64-apple-darwin17.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Is the clang driver something other than clang?

Also looking at

benchmark/cmake/modules/AddSwiftBenchmarkSuite.cmake:27

It seems to essentially do

$ xcrun -toolchain macosx10.17 -f clang
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

Which works (as above).

2

The benchmark suite's cmake is a sort of hacked up thing. My suggestion would be to invoke it like so:

cmake ../swift/benchmark -G Ninja -DSWIFT_EXEC=${BIN_DIR}/usr/bin/swiftc -DSWIFT_DARWIN_XCRUN_TOOLCHAIN=XcodeDefault.

Also I added support recently from swiftpm. You should be able to just download a developer snapshot and edit in Xcode by using swiftpm's generate-xcodeproj.

2 Likes
3

Thanks!

4

Is this warning to be expected:

$ /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2018-06-08-a.xctoolchain/usr/bin/swift package generate-xcodeproj
'swiftbench' /Users/jens/sf/SwiftBenchmarkSuite/benchmark: warning: Ignoring duplicate product 'PrimsSplit' (static)
generated: ./swiftbench.xcodeproj
$

?

5

Yes. I don't know how to get rid of that = (. I haven't had the time to look into it. If you have a good idea, a patch would be appreciated.

Also another thing to be aware of: this just uses the default swiftpm setup, so you need to probably need to make sure to change the settings to build with release by default.

6

Also, this is pretty new, so I would appreciate any other feedback.

7

Question. Are you trying to add a benchmark? Give me a little bit to update the documentation around that. It looks stale.

8

@Jens I just updated the docs. Can you give me some feedback. [WIP] Update benchmark readme by gottesmm · Pull Request #17112 · apple/swift · GitHub?

2 Likes
9

Sure! I can add notes here about anything that comes to mind regarding the benchmark suite in general, not just the swiftpm part. Don't expect too much though, as I'm only looking into this as time permits.

One thing I noticed was that if I allowed Xcode 10 beta to update the project to 4.2, I got over 300 errors or something, enough for me to give up and restart the process using Xcode 9.4, or perhaps try with cmake instead.

No, my goal is just to get acquainted with the SBS, since I've, from the perspective of an amateur, been thinking a bit about how to benchmark issues that are highly context-dependent (eg compiling with -static-stdlib can make some specific code faster or slower depending on context, importing a module that isn't even used can sometimes make some code slower, adding or removing some spaces in a string that is printed after measuring the time of a hot loop can impact the optimizability of the timed code, using &+ can sometimes be slower than using + , etc.).

In short, some issues are impossible to demonstrate or benchmark unless the test is completely stand-alone, compiled with some specific flags etc. So such tests can't eg import TestUtils, or have some specific runFunction or BenchmarkInfo struct, since anything like that might interfere with what's meant to be tested.

So the software that is responsible for compiling, running and collect data from these kinds of tests would have to be as separated from the tests as possible.

This is why I asked the following:

Perhaps you know if and where I can find it?

That's a big improvement! Perhaps the docs could mention how to do a shallow clone, I had to look that up.

10

Just to say, I encountered the same problems as the OP today. I just found this thread and [WIP] Update benchmark readme by gottesmm · Pull Request #17112 · apple/swift · GitHub which looks like a solution.

It would be great if the docs could be fixed.

11

I spoke too soon. Now I see this when I build master of the benchmarks in Xcode 10.2 beta 2 using swift-DEVELOPMENT-SNAPSHOT-2019-02-03-a.

ld: file not found: /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2019-02-03-a.xctoolchain/usr/lib/arc/libarclite_macosx.a

They seem to build OK with the toolchain shipped with Xcode 10.2 beta 2 though.

Is it expected that I cannot build the benchmark suite with the latest snapshot?