Hi,
I am working on generating code coverage for the compiler. This would allow in testing more of the Swift language. I've had limited progress so far.
Here are the different approaches I took:
Approach 1. Build compiler with build-script, modify flags CMAKE_CXX_FLAGS and CMAKE_Cwift_FLAGS in build/Ninja-/swift-/CMakeCache.txt , build swift folder again with ninja again.
CMAKE_CXX_FLAGS = ... -fprofile-instr-generate -fcoverage-mapping and CMAKE_Swift_FLAGS=-profile-generate -profile-coverage-mapping.
Rebuilding the swift-* folder: run `ninja -C build/Ninja-/swift-
Approach 2. Extract steps to build compiler using --dry-run. Build the compiler and intercept (stop) it before it builds the swift-* folder. Manually run the commands for generating the swift folder from the --dry-run's output and edit CMAKE_CXX_FLAGS flags at this line to include profiling coverage report generation information flags
Approach 3. build with the -swift-analyze-code-coverage merged flag from here
So far, only Approach 1 has allowed me to build the compiler, because I'm using the built compiler and modifying its CMakeCache.txt file. When I run tests, most of the executable tests fail along with other tests, with undefined symbols error
Approach 2 and 3 fail at link time (error) for undefined symbols
Undefined symbols for architecture arm64:
"___llvm_profile_runtime", referenced from:
___llvm_profile_runtime_user in SwiftDemangle.cpp.o
...
(maybe you meant: ___llvm_profile_runtime_user)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Is there a known way to generate code coverage for the compiler? Are there suggestions on the path/method that I should use?
Update
We were able to get our LIT executable tests to work with code coverage.
To recap the issue, our LIT executable tests were failing with errors like this:
Undefined symbols for architecture arm64:
"___llvm_profile_runtime", referenced from:
___llvm_profile_runtime_user in SwiftDemangle.cpp.o
...
(maybe you meant: ___llvm_profile_runtime_user)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
The executable tests (written in Swift) were likely trying to access some runtime libraries (written in C++ and originally compiled with code-coverage enabled), which needed to know about the missing ___llvm_profile_runtime symbol mentioned in the error above.
We needed to find out where the ___llvm_profile_runtime symbol was being exported from and then make its location available to our executable LIT tests. The ___llvm_profile_runtime symbol is exported from the libclang_rt.profile_*.a static library, that can be found under the accompanying LLVM build folder that is created when you build Swift. The exact path of the library should be similar to <PATH_TO_BUILD_SUBDIR>/llvm-OS-ARCH/lib/clang/17/lib/darwin/.
Once we had the specifics of the source library of the concerned symbol we simply passed that information along to our LIT executable tests as linker flags.
@harlanhaskins it goes back quite a while and but it looks like you authored the --swift-analyze-code-coverage option in the build-script, that let's users build the compiler with code-coverage enabled and can consolidate profiling information in one place.
We've tried using that option to avoid having manual touch-points, but it we get failures like this:
[444/1598][ 27%][17.485s] Linking CXX shared library lib/libswiftDemangle.dylib
FAILED: lib/libswiftDemangle.dylib
Undefined symbols for architecture arm64:
"___llvm_profile_runtime", referenced from:
___llvm_profile_runtime_user in SwiftDemangle.cpp.o
...
(maybe you meant: ___llvm_profile_runtime_user)
ld: symbol(s) not found for architecture arm64
ERROR: command terminated with a non-zero exit status 1, aborting
I imagined that simply specifying some linker flags, for the search path and name for the libclang_rt.profile_*.a library, to CMake, while building executables and shared libraries would have solved the issue. But it looks like it did not.
We tried specifying these flags in the top-level CMakeCache.txt file in the Swift build sub-directory, as follows:
//Flags used by the linker during all build types.
CMAKE_EXE_LINKER_FLAGS:STRING= -L/Users/kshitij/workspace/swift-project/build/coverage-test/llvm-macosx-arm64/lib/clang/17/lib/darwin/ -lclang_rt.profile_osx
//Flags used by the linker during the creation of shared libraries
// during all build types.
CMAKE_SHARED_LINKER_FLAGS:STRING= -L/Users/kshitij/workspace/swift-project/build/coverage-test/llvm-macosx-arm64/lib/clang/17/lib/darwin/ -lclang_rt.profile_osx
Would you happen to know what we might be missing?
warning: build/ap1/swift-macosx-arm64/test-macosx-arm64/AutoDiff/Sema/DerivedConformances/default.profraw: failed to uncompress data (zlib)
warning: build/ap1/swift-macosx-arm64/test-macosx-arm64/AutoDiff/Serialization/default.profraw: failed to uncompress data (zlib)
warning: build/ap1/swift-macosx-arm64/test-macosx-arm64/AutoDiff/SILOptimizer/default.profraw: failed to uncompress data (zlib)
warning: build/ap1/swift-macosx-arm64/test-macosx-arm64/AutoDiff/SIL/default.profraw: failed to uncompress data (zlib)
warning: build/ap1/swift-macosx-arm64/test-macosx-arm64/AutoDiff/SILGen/default.profraw: failed to uncompress data (zlib)
warning: build/ap1/swift-macosx-arm64/test-macosx-arm64/AutoDiff/Parse/default.profraw: malformed instrumentation profile data: function name is empty
error: no profile can be merged
@hamishknight thank you for fixing this! Out of curiosity, how are the linker flags you're setting, different from what I was setting in the CMakeCache.txt file myself?
Update: I happened to build with flags --release-debuginfo --debug-swift -swift-analyze-code-coverage merged and got this error while merging the generated profile data with llvm-macosx-arm64/bin/llvm-profdata merge -sparse $(all profraw files in build folder) :
warning: ./build/Ninja+cmark-RelWithDebInfoAssert+llvm-RelWithDebInfoAssert+swift-RelWithDebInfoAssertCoverage+stdlib-RelWithDebInfoAssert/swift-macosx-arm64/
stdlib/private/SwiftPrivateThreadExtras/default.profraw: unrecognized instrumentation profile encoding format
stdlib/private/RuntimeUnittest/default.profraw: failed to uncompress data (zlib)
stdlib/public/core/default.profraw: malformed instrumentation profile data: function name is empty
stdlib/public/Platform/default.profraw: failed to uncompress data (zlib)
stdlib/private/StdlibUnittestFoundationExtras/default.profraw: failed to uncompress data (zlib)
With @hamishknight 's patch on adding the compiler flags, I was able to get code coverage analysis working.
Here are the steps for someone's future reference:
To generate a coverage enabled build: utils/build-script --release -t --swift-analyze-code-coverage merged
The Coverage Raw files, *.profraw are generated inside BUILDFOLDER/swift-macosx-arm64/swift-test-results/
These files can be merged by using llvm-profdata merge command as follows:
With the merged *.profdata file, it can be used to generate a report or brought into the editor to show code coverage. I've been using [ryanluker.vscode-coverage-gutters](Name: Coverage Gutters Id: ryanluker.vscode-coverage-gutters Description: Display test coverage generated by lcov or xml - works with many languages Version: 2.11.1 Publisher: ryanluker VS Marketplace Link: Coverage Gutters - Visual Studio Marketplace) with vscode to display coverage information inside the editor directly.