Testing code coverage when using macros

Using Swift
1

I'm playing around with the latest Xcode 15 beta but don't see any option to enable (or view) code coverage for unit tests for my Swift package.

They used to be generated under the "Report navigator". Is anyone aware of where they end up in the latest release?

2

Slight update: it seems like the issue was due to the coverage files not generating.

My package uses macros and has a plugin dependency; something about building with coverage causes the build to fail with:

Undefined symbol: ___llvm_profile_runtime

Commenting out the plugin dependency works, but I'd like to get coverage for that as well. Any idea why this might not be working? Perhaps it's something similar to this issue in the context of the plugin?

1 Like
3

I'm running into this since yesterday on 15b4 when I added a macros package. My other packages don't seem to cause it. In my case, the root is an Xcode project, not a Swift package. How does one enable and disable code coverage?

4

Hey @JetForMe - been debugging this more this am.

It seems like there are two issues:

  1. Code coverage on macro targets causes the compiler to crash
  2. Code coverage seems automatically enabled on the latest Xcode beta and it's not clear how to turn it off.

Some janky partial solutions I've discovered:

  1. If you are purely looking to see the code coverage number, you can remove the macro plugin as a dependency from the library you are looking to view coverage for. You'll need to re-add it when you "ship" your library. I don't believe there is a way to view code coverage for plugin targets without things crashing at the moment.
  2. You can disable all coverage and get things to compile by creating a custom test plan and setting that as the test target for the scheme under Edit Scheme -> Test. This is confusing because there used to be an "Options" tab over the Test Scheme editor where you could easily disable test coverage, however this is not showing for me for any project with macros.

I'm hoping this is just an issue with the early betas and that code coverage will be possible and easy to use for packages with macros in future betas.

1 Like
5

Not sure how to do #1 if I'm actually using the macros to compile my code. But for now I just removed the macro library dependency altogether (it was the SwiftUI macro for creating environment keys, tedious but easy to do without macros).

I did try disabling the test coverage in the test project I created to repro the issue, and that also solves the issue, at the expense of not having code coverage. Since I've never actually looked at a code coverage report before, I can live with that for now, but now that I can't have them I really want them :crazy_face:

1 Like
6

I believe you should be able to disable code coverage by editing the test plan?

2 Likes
7

Yes unfortunately #1 isn't possible if you use the macros to compile the code you want to test. In my case I wanted to test a library that exported the macros, so it was feasible to do this to get coverage.

1 Like
8

Update - this seems to be fixed in the latest beta (Xcode 15 beta 7). I'm now seeing coverage properly generated across libraries and their corresponding plugins.

Unfortunately referencing a macro inside a test file does cause the compiler to crash with the given error:

1.	Apple Swift version 5.9 (swiftlang-5.9.0.128.106 clang-1500.0.40.1)
2.	Compiling with the current language version
3.	While evaluating request TypeCheckSourceFileRequest(source_file "/Users/josh/src/papyrus/PapyrusCore/Tests/APITests.swift")
4.	While type-checking declaration 0x140238b60 (at @__swiftmacro_16PapyrusCoreTests5Todos33_9BE5E177D81FA58DE7A262F2F04D5989LL3APIfMp_.swift:1:1)
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  swift-frontend           0x000000010359b7b4 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  swift-frontend           0x0000000106119f0c llvm::sys::RunSignalHandlers() + 112
2  swift-frontend           0x0000000101e5b720 SignalHandler(int) + 352
3  libsystem_platform.dylib 0x000000018de02a24 _sigtramp + 56
4  swift-frontend           0x0000000105a30364 (anonymous namespace)::DeclChecker::visit(swift::Decl*) (.llvm.14083370151234498882) + 440
5  swift-frontend           0x0000000103ad20f0 swift::Decl::visitAuxiliaryDecls(llvm::function_ref<void (swift::Decl*)>, bool) const + 516
6  swift-frontend           0x0000000105a3023c (anonymous namespace)::DeclChecker::visit(swift::Decl*) (.llvm.14083370151234498882) + 144
7  swift-frontend           0x0000000103259e94 swift::TypeChecker::typeCheckDecl(swift::Decl*, bool) + 220
8  swift-frontend           0x00000001034da040 swift::TypeCheckSourceFileRequest::evaluate(swift::Evaluator&, swift::SourceFile*) const + 432
9  swift-frontend           0x00000001034e22ec llvm::Expected<swift::TypeCheckSourceFileRequest::OutputType> swift::Evaluator::getResultUncached<swift::TypeCheckSourceFileRequest>(swift::TypeCheckSourceFileRequest const&) + 664
10 swift-frontend           0x00000001034d9cc8 swift::TypeCheckSourceFileRequest::OutputType swift::evaluateOrDefault<swift::TypeCheckSourceFileRequest>(swift::Evaluator&, swift::TypeCheckSourceFileRequest, swift::TypeCheckSourceFileRequest::OutputType) + 228
11 swift-frontend           0x00000001021473c0 swift::CompilerInstance::performSema() + 296
12 swift-frontend           0x00000001060d13f8 performCompile(swift::CompilerInstance&, int&, swift::FrontendObserver*) + 464
13 swift-frontend           0x00000001060d5454 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 4568
14 swift-frontend           0x0000000105a1b4c0 swift::mainEntry(int, char const**) + 4116
15 dyld                     0x000000018da7bf28 start + 2236
LLVM Profile Error: Failed to write file "default.profraw": Operation not permitted
Command SwiftEmitModule failed with a nonzero exit code

Not entirely sure what a solution here might be. But at least test coverage for libraries with macros in them is possible now (just not coverage of the actual macros).