Proposed changes to Swift 5 code coverage

Evolution Pitches
1

I've started to look at the Swift Code Coverage. I wanted to pitch my ideas here before submitting a formal proposal / pull request.

1. enable code coverage during build step (only when building debug configuration)
current behavior:

swift test --enable-code-coverage

This current behavior assumes you will build and test every time.

attempting:

swift test --enable-code-coverage --skip-build

does not produce the code coverage folder.

Proposed behavior

swift build --build-tests --enable-code-coverage

Why:
There may be a need to run filtered tests

swift test --skip-update skip-build --filter AppTests.OpenApiTests/testReadOpenApiYAML

Rebuilding each time would not be the preferred behavior.

2. Demangled function names
Function name mangling. It does not appear that the tools are revised yet to demangle the function names defined in the json file output.

"$s3App20Auth0AuthenticatablePAA3JWT10JWTPayloadRzrlE12authenticate5using2on3NIO15EventLoopFutureCyxSgG14Authentication19BearerAuthorizationV_11DatabaseKit0P11Connectable_ptFZ"

Proposed behavior
Provide demangled Swift 5 function names in json
or
Provide updated tooling to demangle Swift 5 function name

  1. Specify output director for code coverage results

Current behavior:
Output is placed in the .build folder

Problem statement
My .gitignore file excludes the .build folder. In some instances, I may want the code coverage data saved so that I can run analysis during CI/CD without rebuilding and rerunning tests.

Proposed behavior

swift build --build-tests --enable-code-coverage --code-coverage-dir Tests/coverage

I would then be able to manage/analyze the coverage data before running any further CI/CD steps.

Feedback?

2 Likes
2

1 and 3 seem reasonable to me, although for 3 is having another option significantly better than just copying the data yourself?

For 2, I’m not really sure is a good idea. The current behavior only exposes the low level data, where it is more compact and some other tools might want the mangled version (for example if they in some way need to correlate with other data). It seems like this might be a reasonable thing to expect current clients to handle. It is too bad that the Swift demangler isn’t easily exposed as API from the stdlib.

3

On #2
I agree with your comment regarding swift demangle being exposed as an API. That would make life easier. I haven't run any time studies to see how long it would take to demangle each function. I could write something to process the tree export (would love that in json or YAML instead).

On #3
My goal here was to have the output directive in line with the ask. I do currently have a cp command before clean-up of the .build folder. The ask is to simplify those two steps into 1.

4

I know of no API guarantee about where the data can be found in the first place. If you cannot find it, you cannot copy it. Some experimentation allowed me to decipher the current implementation details to find it: BuildParamaters.codeCovPath + package name + ".json". But not knowing how stable that assumption is forces me to gate the functionality to a specific patch of Swift.

That could be solved either by the proposed ability to specify the output file location, or instead by some means to query the package manager about where it put the data (comparable to swift build --show-bin-path).

The JSON format also seems to be an implementation detail, so in the same vein, it would be nice if the package manager vended a library module capable of loading it. I have a library that does so, but since it requires relying on implementation details, it would be nice to sink such functionality into the SwiftPM package.

5

I did find a Swift implementation of a demangler (GitHub - MariusCiocanel/CwlDemangle: An implementation of Swift mangled symbol parsing and demangled printing in Swift.). I spent some time today wrapping it in a package so I can try to use it programmatically. If it works, I can use that for post processing and should be good to go.

6

Given a demangler exists, I think I will stick with items 1 and 3 for now in the proposal.