Code coverage?

Has there been any thought put into generating code coverage reports for
the compiler? Or, are there some known steps community members could take
to help out (such as adding a flag to build-script which enables
-fprofile-arcs -ftest-coverage)?

I think it'd be really interesting to see what's covered by the
(validation) tests and what isn't.

Jacob

Luke Larson and I added a few things to support testing code coverage of the compiler.

You can build with

build-script -t --swift-analyze-code-coverage merged

which will rebuild the compiler with coverage instrumentation and run all the tests you've selected with coverage enabled.

Once all the tests are done, you'll get a set of .profraw files in the swift-test-results directory. Just merge them together with

llvm-profdata merge *.profraw -o swiftc.profdata

and you can use llvm-cov to inspect that data.

Note: you'll need to build swift with an upstream version of llvm and clang, because there's a very recent 'pooled' coverage system that we use.

You'll probably see that coverage is really low, somewhere around ~50%. It's that low because the coverage technically considers _all files_, including those from llvm and clang. Those files are, naturally, not tested by the Swift test suite, so they contribute a significant amount of 'un-covered' areas.

When we isolate the Swift compiler's files, I found that coverage was somewhere around 79.45%

If you need any help, let me know!

-- Harlan

ยทยทยท

On Aug 25, 2016, at 2:46 AM, Jacob Bandes-Storch via swift-dev <swift-dev@swift.org> wrote:

Has there been any thought put into generating code coverage reports for the compiler? Or, are there some known steps community members could take to help out (such as adding a flag to build-script which enables -fprofile-arcs -ftest-coverage)?

I think it'd be really interesting to see what's covered by the (validation) tests and what isn't.

Jacob
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

1 Like

I wanted to try running the code coverage for the compiler but have been hitting a wall
Any clues would be helpful

/Applications/CMake.app/Contents/bin/cmake --build /Users/viranchee/appleSwift/build/Ninja+cmark-DebugAssert+llvm-DebugAssert+swift-DebugAssertCoverage+stdlib-DebugAssert/swift-macosx-arm64 -- -j10 all swift-test-stdlib-macosx-arm64 swift-libexec-macosx-arm64 swift-benchmark-macosx-arm64

[1/759] Linking CXX shared library lib/libswiftDemangle.dylib
FAILED: lib/libswiftDemangle.dylib 
: && /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -Wno-unknown-warning-option -Werror=unguarded-availability-new -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-class-memaccess -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -index-store-path "/Users/viranchee/appleSwift/build/Ninja+cmark-DebugAssert+llvm-DebugAssert+swift-DebugAssertCoverage+stdlib-DebugAssert/swift-macosx-arm64/IndexStore" -fdiagnostics-color -Werror=switch -Wdocumentation -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -DOBJC_OLD_DISPATCH_PROTOTYPES=0 -g -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -dynamiclib -Wl,-headerpad_max_install_names -Wl -lobjc   -target arm64-apple-macosx10.13 -Xlinker -compatibility_version -Xlinker 1 -o lib/libswiftDemangle.dylib -install_name @rpath/libswiftDemangle.dylib lib/SwiftDemangle/CMakeFiles/swiftDemangle.dir/SwiftDemangle.cpp.o lib/SwiftDemangle/CMakeFiles/swiftDemangle.dir/MangleHack.cpp.o -L/Users/viranchee/appleSwift/build/Ninja+cmark-DebugAssert+llvm-DebugAssert+swift-DebugAssertCoverage+stdlib-DebugAssert/llvm-macosx-arm64/./lib   -L/Users/viranchee/appleSwift/build/Ninja+cmark-DebugAssert+llvm-DebugAssert+swift-DebugAssertCoverage+stdlib-DebugAssert/swift-macosx-arm64/lib   -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/lib/swift   -L/Users/viranchee/appleSwift/build/Ninja+cmark-DebugAssert+llvm-DebugAssert+swift-DebugAssertCoverage+stdlib-DebugAssert/swift-macosx-arm64/./lib/swift/macosx   -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx -Wl,-rpath,/usr/lib/swift  lib/libswiftDemangling.a && :
Undefined symbols for architecture arm64:
  "___llvm_profile_runtime", referenced from:
      ___llvm_profile_runtime_user in SwiftDemangle.cpp.o
      ___llvm_profile_runtime_user in MangleHack.cpp.o
      ___llvm_profile_runtime_user in libswiftDemangling.a(Demangler.cpp.o)
      ___llvm_profile_runtime_user in libswiftDemangling.a(NodePrinter.cpp.o)
      ___llvm_profile_runtime_user in libswiftDemangling.a(Context.cpp.o)
      ___llvm_profile_runtime_user in libswiftDemangling.a(Remangler.cpp.o)
      ___llvm_profile_runtime_user in libswiftDemangling.a(NodeDumper.cpp.o)
      ...
     (maybe you meant: ___llvm_profile_runtime_user)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[10/759] Building CXX object lib/SIL/CMakeFiles/swiftSIL.dir/IR/SILDeclRef.cpp.o
ninja: build stopped: subcommand failed.

Hi @Viranchee,

This is a thread from the last decade about incorporating features into the compiler. Please post any questions about using Swift in a new, separate thread under "Using Swift," and any issues with compiling the compiler itself in a new thread under the "Development" category. Thanks!

Hi @xwu I want to generate code coverage on the Swift Compiler itself