How is --enable-code-coverage intended to be used?

Development Package Manager
1

How is --enable-code-coverage intended to be used?

It feels like it lives in a grey area in between public interface and and implementation detail.

  • It is a documented part of the CLI, but --help leaves you in the dark about how to use it:

    $ swift test --help
[...]
  --enable-code-coverage
                          Test with code coverage enabled
[...]

  • The Xcode 10.2 release notes provide a little more information, but still not enough to actually get the coverage information:

    • The swift test command can generate code coverage data in a standard format suitable for consumption by other code coverage tools using the flag --enable-code-coverage . The generated code coverage data is available inside <build-dir> / <configuration> /codecov . (44567442)

To actually use it, one must reverse engineer two important details. Since neither is documented anywhere (that I can find), it is not clear to what extent these details should be relied on.

  1. Where is the data actually? [...]/codecov is actually a folder containing .profdata, .profraw and .json files. The .json file seems to be the relevant one; it does contain all the information in an inspectable way. But what is it actually called? Experimentation suggests it resides at the root of codecov and is named according to the declared package name. But are there any hidden surprises? What happens if the package name is not a valid filename? How is a script supposed to predict the filename and locate the file without relying on implementation details?

  2. How should it be loaded? The Xcode release notes state in a cryptic way that the data is in “a standard format suitable for consumption”? What is that standard format? Entering the JSON keys into a search engine leads to a comment in the LLVM source code. How stable is that format? Is it considered part of SwiftPM’s API contract? How are tools supposed to load the data without relying on implementation details?

Some of this has already been briefly mentioned over on this thread.

(To be clear, I have already figured out how to load the data. The question is only whether it can be done “properly” without assuming undocumented details. If it can’t, then the following question is what would need to be changed so that it can.)

@Aciid, @ddunbar

4 Likes
2

If everything worked correctly, there should be a file present in that directory with name of the root package. If a package name is not a valid file name, my guess is that SwiftPM will fail to write the JSON file. Although, I'd expect something else might go wrong during the build itself. We could add an option in SwiftPM that returns the path of this JSON file. That way, clients won't need to rely on the structure of the build directory.

SwiftPM uses llvm-cov's export command to create this JSON file. @vedantk should be able to confirm if it's a stable format.

3

Are you thinking CLI (à la --show-bin-path) or as a part of libSwiftPM’s Swift API? To me it doesn’t matter which.

4

@SDGGiesbrecht you are right, it is at the moment closer to an "implementation detail" which could be used to build more code coverage stuff. We didn't know exactly what form that stuff should take though, so we hoped it would unblock development of coverage related features.

What I personally would like to work towards is tooling which would allow things like annotated PRs with code coverage impact. What isn't very clear is how much of something like that would be part of SwiftPM's responsibility, versus other projects. The idea of going ahead and putting this in was so that such things could be explored, not necessarily because the current flag has much value.

2 Likes
5

That is roughly what I use it for, though my system simply rejects anything less than 100% instead of reporting the measurement (with some modification). SwiftPM’s coverage report is much easier to consume than Xcode’s and it works wonderfully on Linux! So thank you for putting it in there!

It is the last remaining “abuse of an implemantation detail” in my code, so I am hoping to help move it to the point where some way of getting at the data can be formalized.

1 Like
6

@vedantk, do you have an answer for me?

7

llvm-cov's JSON output includes a semver field (https://semver.org), which you may parse to detect breaking changes.

1 Like
8

I just submitted a pull request with --show-codecov-path.

1 Like