Compiler code coverage, need help

Viranchee · 2023-10-26T03:30:03.458Z

Hi,
I am working on generating code coverage for the compiler. This would allow in testing more of the Swift language. I've had limited progress so far.

Here are the different approaches I took:

Approach 1. Build compiler with build-script, modify flags CMAKE_CXX_FLAGS and CMAKE_Cwift_FLAGS in build/Ninja-/swift-/CMakeCache.txt , build swift folder again with ninja again.

CMAKE_CXX_FLAGS = ... -fprofile-instr-generate -fcoverage-mapping and CMAKE_Swift_FLAGS=-profile-generate -profile-coverage-mapping.
Rebuilding the swift-* folder: run `ninja -C build/Ninja-/swift-

Approach 2. Extract steps to build compiler using --dry-run. Build the compiler and intercept (stop) it before it builds the swift-* folder. Manually run the commands for generating the swift folder from the --dry-run's output and edit CMAKE_CXX_FLAGS flags at this line to include profiling coverage report generation information flags

Approach 3. build with the -swift-analyze-code-coverage merged flag from here

So far, only Approach 1 has allowed me to build the compiler, because I'm using the built compiler and modifying its CMakeCache.txt file. When I run tests, most of the executable tests fail along with other tests, with undefined symbols error

Approach 2 and 3 fail at link time (error) for undefined symbols

Undefined symbols for architecture arm64:
  "___llvm_profile_runtime", referenced from:
      ___llvm_profile_runtime_user in SwiftDemangle.cpp.o
      ...
     (maybe you meant: ___llvm_profile_runtime_user)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Is there a known way to generate code coverage for the compiler? Are there suggestions on the path/method that I should use?

jkshtj · 2023-10-27T20:41:49.888Z

Update
We were able to get our LIT executable tests to work with code coverage.

To recap the issue, our LIT executable tests were failing with errors like this:

Undefined symbols for architecture arm64:
  "___llvm_profile_runtime", referenced from:
      ___llvm_profile_runtime_user in SwiftDemangle.cpp.o
      ...
     (maybe you meant: ___llvm_profile_runtime_user)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

The executable tests (written in Swift) were likely trying to access some runtime libraries (written in C++ and originally compiled with code-coverage enabled), which needed to know about the missing ___llvm_profile_runtime symbol mentioned in the error above.

We needed to find out where the ___llvm_profile_runtime symbol was being exported from and then make its location available to our executable LIT tests. The ___llvm_profile_runtime symbol is exported from the libclang_rt.profile_*.a static library, that can be found under the accompanying LLVM build folder that is created when you build Swift. The exact path of the library should be similar to <PATH_TO_BUILD_SUBDIR>/llvm-OS-ARCH/lib/clang/17/lib/darwin/.

Once we had the specifics of the source library of the concerned symbol we simply passed that information along to our LIT executable tests as linker flags.

jkshtj · 2023-10-27T20:59:48.134Z

@harlanhaskins it goes back quite a while and but it looks like you authored the --swift-analyze-code-coverage option in the build-script, that let's users build the compiler with code-coverage enabled and can consolidate profiling information in one place.

We've tried using that option to avoid having manual touch-points, but it we get failures like this:

[444/1598][ 27%][17.485s] Linking CXX shared library lib/libswiftDemangle.dylib
FAILED: lib/libswiftDemangle.dylib
Undefined symbols for architecture arm64:
  "___llvm_profile_runtime", referenced from:
      ___llvm_profile_runtime_user in SwiftDemangle.cpp.o
      ...
     (maybe you meant: ___llvm_profile_runtime_user)
ld: symbol(s) not found for architecture arm64
ERROR: command terminated with a non-zero exit status 1, aborting

I imagined that simply specifying some linker flags, for the search path and name for the libclang_rt.profile_*.a library, to CMake, while building executables and shared libraries would have solved the issue. But it looks like it did not.

We tried specifying these flags in the top-level CMakeCache.txt file in the Swift build sub-directory, as follows:

//Flags used by the linker during all build types.
CMAKE_EXE_LINKER_FLAGS:STRING= -L/Users/kshitij/workspace/swift-project/build/coverage-test/llvm-macosx-arm64/lib/clang/17/lib/darwin/ -lclang_rt.profile_osx

//Flags used by the linker during the creation of shared libraries
// during all build types.
CMAKE_SHARED_LINKER_FLAGS:STRING= -L/Users/kshitij/workspace/swift-project/build/coverage-test/llvm-macosx-arm64/lib/clang/17/lib/darwin/ -lclang_rt.profile_osx

Would you happen to know what we might be missing?

hamishknight · 2023-10-29T14:30:39.377Z

Looks like we need to be setting the linker flags in cmake, try it with this patch [cmake] Fix `SWIFT_ANALYZE_CODE_COVERAGE` by hamishknight · Pull Request #69491 · apple/swift · GitHub.

Viranchee · 2023-10-29T16:52:43.491Z

@hamishknight Thank you for updating this
I have a compiler building in background for your patch.

Also I'd like to highlight that there's also an issue affecting merging of the profile data files:

LLPROFDATA=build/ap1/llvm-macosx-arm64/bin/llvm-profdata
BUILDFOLDER=build/ap1/swift-macosx-arm64

$LLPROFDATA merge -sparse \
$BUILDFOLDER/stdlib/public/Differentiation/default.profraw \
$BUILDFOLDER/stdlib/private/DifferentiationUnittest/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/SILOptimizer/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/validation-test/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/SILGen/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/Serialization/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/Parse/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/ModuleInterface/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/compiler_crashers/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/IRGen/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/Sema/DerivedConformances/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/Sema/DerivativeRegistrationCrossFile/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/Sema/DerivativeRegistrationCrossModule/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/Sema/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/Sema/ExportedDifferentiationModule/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/Sema/ImplicitDifferentiableAttributeCrossFile/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/TBD/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/compiler_crashers_fixed/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/stdlib/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/SIL/Serialization/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/SIL/Parse/default.profraw \
$BUILDFOLDER/test-macosx-arm64/AutoDiff/SIL/default.profraw \
/Volumes/compiler/apple/swift/SwiftCompilerSources/default.profraw \
-o lol.profdata

warning: build/ap1/swift-macosx-arm64/test-macosx-arm64/AutoDiff/Sema/DerivedConformances/default.profraw: failed to uncompress data (zlib)
warning: build/ap1/swift-macosx-arm64/test-macosx-arm64/AutoDiff/Serialization/default.profraw: failed to uncompress data (zlib)
warning: build/ap1/swift-macosx-arm64/test-macosx-arm64/AutoDiff/SILOptimizer/default.profraw: failed to uncompress data (zlib)
warning: build/ap1/swift-macosx-arm64/test-macosx-arm64/AutoDiff/SIL/default.profraw: failed to uncompress data (zlib)
warning: build/ap1/swift-macosx-arm64/test-macosx-arm64/AutoDiff/SILGen/default.profraw: failed to uncompress data (zlib)
warning: build/ap1/swift-macosx-arm64/test-macosx-arm64/AutoDiff/Parse/default.profraw: malformed instrumentation profile data: function name is empty
error: no profile can be merged

Viranchee · 2023-10-30T14:07:59.716Z

Update:
The generated profraw files can be merged except these 3:

swift-macosx-arm64/stdlib/public/core/default.profraw \
swift-macosx-arm64/stdlib/private/RuntimeUnittest/default.profraw \
swift-macosx-arm64/stdlib/private/StdlibUnittestFoundationExtras/default.profraw \

Thanks again @hamishknight

hamishknight · 2023-10-30T14:53:46.425Z

Glad to hear it's working! Interesting that those profiles aren't merging, I'll try investigate when I get a moment.

jkshtj · 2023-10-30T16:57:39.761Z

@hamishknight thank you for fixing this! Out of curiosity, how are the linker flags you're setting, different from what I was setting in the CMakeCache.txt file myself?

Viranchee · 2023-11-02T14:19:11.866Z

Update: I happened to build with flags --release-debuginfo --debug-swift -swift-analyze-code-coverage merged and got this error while merging the generated profile data with llvm-macosx-arm64/bin/llvm-profdata merge -sparse $(all profraw files in build folder) :

warning: ./build/Ninja+cmark-RelWithDebInfoAssert+llvm-RelWithDebInfoAssert+swift-RelWithDebInfoAssertCoverage+stdlib-RelWithDebInfoAssert/swift-macosx-arm64/

stdlib/private/SwiftPrivateThreadExtras/default.profraw: unrecognized instrumentation profile encoding format
stdlib/private/RuntimeUnittest/default.profraw: failed to uncompress data (zlib)
stdlib/public/core/default.profraw: malformed instrumentation profile data: function name is empty
stdlib/public/Platform/default.profraw: failed to uncompress data (zlib)
stdlib/private/StdlibUnittestFoundationExtras/default.profraw: failed to uncompress data (zlib)

Viranchee · 2023-11-17T22:09:20.570Z

Update:

With @hamishknight 's patch on adding the compiler flags, I was able to get code coverage analysis working.

Here are the steps for someone's future reference:
To generate a coverage enabled build:
utils/build-script --release -t --swift-analyze-code-coverage merged

The Coverage Raw files, *.profraw are generated inside BUILDFOLDER/swift-macosx-arm64/swift-test-results/

These files can be merged by using llvm-profdata merge command as follows:

find $RESULTS -name "*.profraw" -print0 | xargs -0 llvm-profdata merge -sparse -o /tmp/merged.profdata

With the merged *.profdata file, it can be used to generate a report or brought into the editor to show code coverage. I've been using [ryanluker.vscode-coverage-gutters](Name: Coverage Gutters Id: ryanluker.vscode-coverage-gutters Description: Display test coverage generated by lcov or xml - works with many languages Version: 2.11.1 Publisher: ryanluker VS Marketplace Link: Coverage Gutters - Visual Studio Marketplace) with vscode to display coverage information inside the editor directly.

llvm-cov export -format="lcov" -object $BUILD/swift/bin/swiftc -instr-profile=$TMPPROFDATA > coverage/lcov.info

Thank you again Hamish!