Experimenting with documentation coverage metrics - a few questions and feedback

Development Swift-DocC
1

I've started digging in and experimenting with the documentation coverage metrics, as a way to just track how much API surface exists on the library I'm contributing into, and to track updates over time.

(FYI: All of this is with the version of DocC included with Xcode 13.1)

After a bit of digging in the code and experimentation, I ended up using the following pattern:

  1. make an explicit symbol graph directory
  2. run the swift build with the -emit-symbol-graph option aimed at that directory
  3. run a docc convert command to expose the coverage metrics.

An example of such:

mkdir -p .build/symbol-graphs

swift build --target Automerge \
-Xswiftc -emit-symbol-graph \
-Xswiftc -emit-symbol-graph-dir -Xswiftc .build/symbol-graphs

xcrun docc convert Source/Automerge.docc \
--fallback-display-name Automerge \
--fallback-bundle-identifier org.automerge.Automerge-swift \
--fallback-bundle-version 0.1.6 \
--additional-symbol-graph-dir .build/symbol-graphs \
--experimental-documentation-coverage \
--level brief

Example brief metrics output:

   --- Experimental coverage output enabled. ---
                | Abstract        | Curated         | Code Listing
Types           | 40% (10/25)     | 96% (24/25)     | 0.0% (0/25)
Members         | 3.2% (16/507)   | 74% (374/507)   | 0.20% (1/507)
Globals         | 2.9% (1/35)     | 97% (34/35)     | 0.0% (0/35)

(By the way, a HUGE THANK YOU!!!! for including a detail that shows how many of the samples include a code listing, that's just amazing!)

This particular library has a dependency to another library (ZippyJSON), and I didn't want to include it's symbols into my coverage metrics. The solution I've found for now is to explicitly remove it's set of symbols from my symbol-graphs directory, but that's kind of awkward and specific to this package.

Is there a way to "exclude" a set of symbols (other than removing them) for the purposes of getting these metrics (or likewise, getting hints from the --analyze option)?

When I run out a detailed "report":

# exclude ZippyJSON symbols from analysis
rm -f .build/symbol-graphs/ZippyJSON.symbols.json

xcrun docc convert Source/Automerge.docc \
--fallback-display-name Automerge \
--fallback-bundle-identifier org.automerge.Automerge-swift \
--fallback-bundle-version 0.1.6 \
--additional-symbol-graph-dir .build/symbol-graphs \
--experimental-documentation-coverage \
--level detailed > report.txt

The report is sort of a table, with the synopsis at the top, but sort of not. Is there any consideration to getting the detailed output report in a structured format of any kind? CSV perhaps?

What does the USR field that is reported represent?

Example:

Symbol Name                      Kind                             Abstract?      Curated?       Code Listing?     Parameters     Language          USR
applyPatch(patch:)             | Instance Method                | true         | false        | false           | 100% (1/1)   | Swift           | doc://org.automerge.Automerge-swift/documentation/Automerge/Document/applyPatch(patch:)

Is USR something that we could potentially use to link documentation between frameworks?

1 Like
2

Hi @Joseph_Heck, when you run swift build with -Xswiftc -emit-symbol-graph, symbol graph files are generated for Automerge and that target's dependencies.

Since your docc invocation builds the Automerge.docc catalog, the directory passed to --additional-symbol-graph-dir should only contain symbol graph files for the Automerge module. That's something that Xcode takes care of when using "Build Documentation" in Xcode and that an integration with SwiftPM will offer in the future.

If you're using Xcode, you can run the xcodebuild docbuild command (described here) on your target and set the OTHER_DOCC_FLAGS to set extra flags for the docc invocation, like so:

xcodebuild docbuild -scheme SlothCreator \
  -derivedDataPath ~/Desktop/SlothCreatorBuild \
  -destination platform=macOS \
  OTHER_DOCC_FLAGS="--experimental-documentation-coverage --level brief"

This will output the coverage table and also generate a documentation-coverage.json file in the DocC Archive placed in the derived data folder.

What does the USR field that is reported represent?

The column name "USR" is actually a misnomer in this case because what gets output is the documentation link of the page. "USR" is a term refers to the unique string generated by the Swift compiler for a symbol. Thanks for pointing that out, I filed a bug about this: [SR-15540] Documentation coverage "USR" column is named incorrectly · Issue #205 · apple/swift-docc · GitHub.

Also, it's worth noting that support for emitting documentation coverage information is experimental, so I really appreciate you giving it a shot! Any feedback is highly appreciated.

1 Like
3

Ooh - very interesting! Thank you for the Xcode build notes!

Is there a path to getting to that structured JSON file from DocC directly on the command line? In particular, if the project (library/framework) I'm working for a server-side library, then Xcode isn't my immediate build environment, but I'd still like to get the structured/delimited report from my CI systems.

The synopsis information (e.g. 'brief' output) groups into 3 sections: 'Globals', 'Types', and 'Members' - and its not immediately obvious which symbol types fall into which category. As the feature moves forward out of experimental status, it would be nice to have this detailed somewhere.

As it stands, the synopsis numbers are super useful (even if I was confused about some of what it was telling me - I wasn't entirely certain where it was telling me about gaps), and I absolutely love the explicit metric of "code snippet" coverage. I had a self-goal of providing code snippets for all the major classes, and it's already helped me get a view of what's there from a first rush through, and what's still pending. This is also where the detailed (structured) report comes in exceptionally handy - right now I'm manually pasting that file into a spreadsheet and filtering down to the "important" symbols - so I'd love a way to take that raw structured report output and more easily get to a filtered set based on the framework design.

Finally, I wasn't sure how (or if) Articles were exposed in this. I suspected they might be 'Globals', and in my initial case here, I have an article in the docc with a quick overview of the key ways to use the framework (the kind of stuff that had previously been dropped in the repo's README to show off the framework). However, I'm not sure that's showing that the article has a code listing.

The article is included within the larger report set when run at 'detailed' - and has several code listings within it - but it doesn't track into the 'brief' (or I didn't know how to spot it in there).

4

If you're not using Xcode, there currently isn't a compiler-driven way to get the symbol graph files for a specific module unfortunately—the symbol graph files for all the modules involved in the build get emitted in the same directory. To publish the DocC docs, we run a command to delete symbol graph files via a heuristic: https://github.com/apple/swift-docc/blob/main/bin/preview-docs#L69. This is going to be resolved once DocC is integrated with SwiftPM.

Agreed, we'll want to have documentation for this feature once it's ready.

That's a great idea. I think one way we could achieve this would be to provide an ability to visualize coverage data based on how you've curated your APIs. For instance, the symbols that are curated in the Topics section of the top-level page of the framework are likely the most important ones, so the documentation coverage data could put these at the forefront in some way.

Articles aren't currently included in the 'brief' report indeed. Looking at the codebase (https://github.com/apple/swift-docc/blob/main/Sources/SwiftDocC/Infrastructure/CoverageDataEntry.swift#L223), the 'Globals' category encapsulates global functions, modules, global variables, and operators.

1 Like
5

Hello, is there any news about this feature? Is documentation coverage still an experimental feature? Has it evolved since then? Thanks.

6

From the code, It's still in the experimental phase - it's functional, but the internal team's development focus has been elsewhere.

FWIW, I continue to use it and find it useful for overviewing documentation coverage details.

1 Like
7

Moved this question to a new thread since this one was about @Joseph_Heck's experience with this feature