Currently, swift test supports generating a JSON coverage report, which is great for ingesting into various systems. The JSON coverage report generation is processed by llvm-cov where SwiftPM is simply orchestrating the command line invocation. The JSON, however, is not very "human consumable friendly" while iterating at-desk.
This pitch proposes adding an additional command line argument to swift test that would specify whether to generate a JSON or HTML report. Selecting HTML would produce a llvm-cov HTML report, that is more easily consumable by a developer at desk. Similar to the JSON coverage report, Swift Package Manager would only be responsible for constructing the command line argument that will be passed to llvm-cov, which would produce the HTML report.
Since llvm-cov show (which would produce the HTML) has numerous CLI arguments that allow configurations of the HTML report, we need to ensure this functionality does not have a "command line arguments" explosion to swift test. To counteract this, a response file will be introduced (that will be stores in the Swift Package project). In addition, we can optionally add a command line argument similar to -Xlinker, -Xcxx, etc.., which would override the arguments in the response file.
Looking forward to hearing your feedback and thoughts.
Because when working from IDE, I typically look up coverage there, this feature for me has most value in CI.
Off course, when running in CI, I’m also interested in the JSON representation. But having an HTML one that I can store as an artifact with the pipeline run is very useful. For instance to have a quick look. But also for troubleshooting the CI.
From your demonstration, I understand you can choose to generate the report from the tests directly, or from a previous run? I’m most interested in creating the report from a previous run. In CI, I would first run `swift test` with JSON as coverage output (to send to Sonarqube). And then use that same test run data to create an HTML representation as a pipeline artifact.
Do you already have an implementation we can try out and/or can you share an example report?
From the demonstration I presented to the Testing Working Group, I did have the ability to generate a report from a previous run. However, after speaking with some folks, it was deemed maybe unnecessary - I had forgotten about this specific use case (sorry). More specifically, a coverage subcommand was deemed possibly unnecessary.
I understand the need and desire for wanting to produce two reports (a JSON and HTML) as they each have their purposes.
Do you have a pany preferences between (1) the ability of specify the new option multiple times (e.g.: --formats json,html or --format json --format html) or (2) have the ability to generate a coverage report from a previous run?
If it’s possible to specify two formats (i.e. option 1), that would have my preference. It’s more intuitive and will also make the pipeline file easier to read. `–format json,html` would have my preference within option 1.
Do note, to get the actual JSON in a usable format is currently a two step process in our pipeline:
swift test --enable-code-coverage
/home/vsts/.local/share/swiftly/bin/llvm-cov show -instr-profile=.build/debug/codecov/default.profdata .build/debug/ExampleProjectPackageTests.xctest > Coverage.report
Would that second step (the llvm-cov call) still be necessary?
If we opt for option1, we will need to reconsider how the --show-coverage-path output will be presented. I propose adding a --print-format (argument names to be determined) that will accept the same values as the --type option in swift package describe, per @dschaefer2 post above. The text output could look something like:
$ swift test --show-coverage-path --coverage-format html, json --print-format text
JSON: /path/to/json/file
HTML: /path/to/html/report
I'm sorry that you need to have a 2-stage approach in your CI environment. however this proposal was not intended to touch the JSON report. With that said, SwiftPM could most definitely improve the experience by potentially incorporating ways to configure the JSON. Could I ask you to please file an enhancement GitHub issue against swift-package-manager so we can investigate improvements in the JSON coverage report?
To try it, build the project and then invoke the built swift-test binary directly. On *nix system, this could look like:
❯ swift build
❯ $(swift build --show-bin-path)/swift-test --help
<...SNIP..>
COVERAGE OPTIONS:
--show-codecov-path, --show-code-coverage-path, --show-coverage-path
Print the path of the exported code coverage JSON file.
--print-codecov-path-mode, --print-code-coverage-path-mode, --print-coverage-path-mode <print-codecov-path-mode>
How to display the paths of the selected code coverage file formats. (default: text)
json - Display the output in JSON format.
text - Display the output as plain text.
--enable-codecov, --enable-code-coverage, --enable-coverage/--disable-codecov, --disable-code-coverage, --disable-coverage
Enable code coverage. (default: --disable-codecov)
--codecov-format, --code-coverage-format, --coverage-format <format>
Format of the code coverage output. Can be specified multiple times. (values: json, html; default: Produces a JSON coverage report.)
<...SNIP...>
The help text for the --coverage-format option does not appear correctly as each item should have an associated help text. This is tracked by apple/swift-argument-parser#818.
The PR currently allows specifying multiple coverage format by providing --coverage-format <format> multiple times.
I am very much in favor of having this option in SwiftPM. Currently I use a third-party Swift script to process code coverage output and turn it into HTML for my CI. Having an option to generate this from SwiftPM would be very useful and help clean that up significantly.
Why would having it in SPM be better than a tool that can consume the JSON output and produce any format you want?
I'll ask that generally as well: why burden SPM, which is already under-resourced, with building and maintaining a single HTML export, when it already, and always will, supports JSON output of the same information. That should make it trivial for any tool to ingest. So what the value of putting this in SPM itself?
The JSON output format will remain. So existing workflow will continue to work.
Since SwiftPM knows where the sources, and the profile data are located, it can more easily produce an HTML version of the report by invoking llvm-cov utility. It's meant for a quality of life improvements to remove the need of having to manually create a script to produce HTML
SwiftPM is not generating the report itself. As with the JSON report, SwiftPM delegate the report generation to llvm-cov.
So this really is to just add an additional output formatting option that in the end forwards to llvm-cov as is already done, but that can also output to html and json at the same time as well. Seems simple enough to me…
Wait, so is this a new output format, a sibling to the existing JSON output, or is it just a way to get the llvm-cov report natively? AFAIK, those two outputs don't cover the same thing.
This will be a new output format in addition to the existing JSON output. When this new format is specified/requested, SwiftPM plays the role of "orchestrator" and will display a path to the report, which is generated by llvm-cov.
Maybe the Swift Evolution proposal (currently in draft) will provide additional context around the proposed functionality/feature..
❯ swift build
❯ $(swift build --show-bin-path)/swift-test --help
<...SNIP...>
COVERAGE OPTIONS:
--show-codecov-path, --show-code-coverage-path, --show-coverage-path
Print the path of the exported code coverage files.
--print-codecov-path-mode, --print-code-coverage-path-mode, --print-coverage-path-mode <print-codecov-path-mode>
How to display the paths of the selected code coverage file formats. (default: text)
json - Display the output in JSON format.
text - Display the output as plain text.
--enable-codecov, --enable-code-coverage, --enable-coverage/--disable-codecov, --disable-code-coverage, --disable-coverage
Enable code coverage. (default: --disable-codecov)
--codecov-format, --code-coverage-format, --coverage-format <format>
Format of the code coverage output. Can be specified multiple times. (values: json, html; default: Produces a JSON
coverage report.)
<...SNIP...>
❯ $(swift build --show-bin-path)/swift-test --filter AbsolutePathTests --enable-coverage --coverage-format html --coverage-format json --scratch-path .test
Building for debugging...
[5/5] Write swift-version-7104982ABD005795.txt
Build complete! (1.60s)
Test Suite 'Selected tests' started at 2025-10-01 22:10:43.894.
Test Suite 'SwiftPMPackageTests.xctest' started at 2025-10-01 22:10:43.895.
Test Suite 'SwiftPMPackageTests.xctest' passed at 2025-10-01 22:10:43.895.
Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
Test Suite 'Selected tests' passed at 2025-10-01 22:10:43.895.
Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
◇ Test run started.
↳ Testing Library Version: 1400
↳ Target Platform: arm64e-apple-macos14.0
◇ Suite PathTests started.
◇ Suite AbsolutePathTests started.
<...SNI...>
✔ Suite AbsolutePathTests passed after 0.002 seconds.
✔ Suite PathTests passed after 0.003 seconds.
✔ Test run with 20 tests in 8 suites passed after 0.003 seconds.
Code coverage report:
- HTML: /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/.test/arm64-apple-macosx/debug/codecov/SwiftPM-html/index.html
- JSON: /Users/bkhouri/Documents/git/public/swiftlang/swift-package-manager/.test/arm64-apple-macosx/debug/codecov/SwiftPM.json
Hi folks, I was thinking about merging the --show-coverage-path and --show-coverage-path-mode [mode] into a single try-state argument. what do folks think about this?
So instead of
--show-codecov-path, --show-code-coverage-path, --show-coverage-path
Print the path of the exported code coverage files.
--print-codecov-path-mode, --print-code-coverage-path-mode, --print-coverage-path-mode <print-codecov-path-mode>
How to display the paths of the selected code coverage file formats. (default: text)
json - Display the output in JSON format.
text - Display the output as plain text.
it would be
--show-codecov-path, --show-code-coverage-path, --show-coverage-path <show-codecov-path>
Print the path of the exported code coverage files.
json - Display the output in JSON format.
text - Display the output as plain text.
So running:
swift test --> will not output coverage paths swift test --show-coverage-path --> will output coverage path in text format (the default) swift test --show-coverage-path json --> will output coverage path in json format. swift test --show-coverage-path text --> will output coverage path in text format.
What do folks think about this? Do you prefer having a single tri-state argument, or shall we keep two separate arguments?
I prefer this over having two separate arguments. Not only is it more concise, but it also prevents two arguments that look very alike, yet have an interdependency: --print-codecov-path-mode makes little sense without --show-codecov-path. I feel this will help keep commands more readable. And the optional format specifier and default behavior make a lot of sense to me in merging these two arguments.
