[Pitch] Documentation generation for any SwiftPM package

Evolution Pitches
21

I can appreciate the idea of making the documentation generation pluggable through some kind of an API. However, in my experience we need to consider at least two, preferably three candidate clients before being able to build a really good abstraction for them.

I don't think we have any other candidates at the moment that could plug in to a docs command, so I suggest that we start with the subcommand that ties directly to DocC for now. When others systems becomes available, and the requirements are there, then we can build out the abstractions.

3 Likes
22

This is awesome, +1 from me for this pitch. I am also leaning towards using something like swift doc or swift docc as the command to generate the docs, with subcommands/options to offer more granularity.

3 Likes
23

It probably won't be surprising that this pitch came up in today's Documentation Workgroup meeting, and while we didn't have any insight into the "how", there is a lot of excitement to see this come to fruition in some form. I did want to reflect some of the conversation here to share it more broadly.

As Chris mentioned in his pitch, there's docc built as part of the toolchain, which constrains itself in terms of dependencies, and acts as a sort of "core driver" kind of executable. It requires symbol graphs (JSON files that have the details about the API as provided by the Swift compiler) as an input, and mixes that in various forms with supplemental markdown content, sourced from what we the "DocC catalog" - really a directory with markdown that aligns to some markup conventions (think of them as semi-structured data).

Today we're doing all that coordination using the swift-docc-plugin, which is what provides the "generate-documentation" (and "preview-documentation") commands, but at it's core, it's an orchestrator. And that coordination would be great to have directly in the swift toolchain, but that also comes with the downside of being locked to the toolchain - which comes with constraints on packages that it's allowed to depend on.

As we're imagining what we might want there, I'd like to throw out that we'd really love to have some option to define and/or use code that isn't bound by the same constraints as the toolchain that we could optionally call for some of this work - be it coordination/orchestration or providing alternate renderings of the documentation content.

I don't have any brilliant ideas on how to provide such plugins, or patterns that the Swift community leans into generally, but as we're doing that for SwiftPM, I'd love to see how we can apply these same options to allow DocC to take advantage of the same mechanism if possible.

3 Likes
24

I assume that parts of the "Proposal" section was intended to summarize how DocC—and the DocC plugin that does build orchestrations work today—but some of what you've described is different enough that it's worth pointing it out.

If for any of these differences, the intention is to follow the behavior that's described in the "Proposal" section rather than the current behavior, then it would be good to clarify that as it could mean that there's extra work in other components.

This command automatically generates DocC archives in HTML for all package products and intelligently merges them into a unified, package-level archive that provides a cohesive browsing experience. The command handles the complexity of target discovery, symbol graph generation, and archive merging behind the scenes, requiring no configuration for basic use cases.

Merging documentation from multiple targets into a unified archive is something that DocC supports and the DocC plugin can orchestrate but it requires passing the --enable-experimental-combined-documentation flag to the plugin to perform that orchestration.

It is a goal that that feature would both get rid of its "experimental" status and be enabled by default, but neither of those are the case today.

For executable targets that use Swift Argument Parser or are capable of generating a compatible tool-info JSON, the command automatically extracts the complete command-line structure and converts it into documentation-ready symbol graphs.

This feature, as described, doesn't exist today. It's been talked about and the Argument Parser is capable of outputting a markdown file that lists the flags, options, and arguments for each command. However, Swift Argument Parser isn't capable of outputting a symbol graph file and the DocC plugin doesn't have any orchestration that calls the Argument Parser to generate the markdown file.

The --internal flag extends documentation generation to include internal targets that aren't exposed as public products.

The DocC plugin defaults to building documentation for all "products" of the package, because those are the only targets that are available to other developers. Developers can specify what products or what targets that the DocC plugin should orchestrate a documentation build for by passing one or more --product <product-name> or one or more --target <target-name>. The plugin doesn't have an --internal flag. Instead the developer would need to spell out all the target names if they wanted to build documentation for the internal targets as well.

Also, I could imagine that someone would think that the --internal flag would configure the access level of what symbols are part in the documentation so that it includes internal-access symbols—which is configured with the --symbol-graph-minimum-access-level <access-level> argument with the DocC plugin—so it might be worth naming this flag something along the lines of --all-targets to avoid that potential confusion.

2 Likes
25

I'm curious what behaviors you think that this preview (sub)command would have.

I'm asking because I feel that preview-documentation command of the DocC plugin has the wrong behaviors today and I would advice against building something new that follows those behaviors.

Specifically, in the DocC plugin the "preview-documentation" command has two main behaviors that are different from the "generate-documentation" command:

  • It watches for changes to files in the documentation catalog and rebuilds the documentation if any of them changes.
  • It starts a local web server so that the developer can preview the rendered documentation pages.

This may sound good at first, but both these behaviors have severe limitations that reduce their usefulness in practice. To source of these limitations is that the both watching the inputs for changes and running the local web server happens in docc preview:

  • Because the inputs to docc preview is there catalog and the symbol graph files, it can't watch the Swift source files for changes and refresh the documentation when a developer adds a new method or updates an in-source documentation comment. Even if DocC found the source files through the symbol graph information and watched them for changes, DocC itself doesn't extract the symbol information from the source code. Rather it's the DocC plugins responsibility to orchestrate the documentation build and call into the Swift complier to extract source information and pass that to DocC.
  • Because docc preview is building the documentation, it can't be used to preview the unified package-level documentation that's produced by docc merge.

If we are talking about having some preview command that includes either "watches files for changes" or "runs a local web server" then I would recommend that we split those responsibilities across different components:

  • The component that does orchestration is the only one capable of watching source files for input and orchestrating a new/updated symbol extraction.
  • The component that runs the local web server should be passed the already built archive to serve, so that that archive can come from any source (which could be docc convert, docc merge, some new command, or potentially even a different tool).

If all of this is out of scope for this pitch, then I would recommend that we defer any documentation preview functionality rather that we start down a path that we may to back out of.

26

The main benefit I see in this pitch is that as a user, I can stop worrying about which logic is implemented in the docc tool vs the plugin, it’s all available under one umbrella, and can get refactored without me having to change my workflow.

As a user, when I start working on a package, I want to run swift doc and let it run, and it’ll do all of the things you’ve described: watch not just catalog contents, but also source files, rebuild as needed, merge docs from modules into a single archive, and run the local server for me, all updating automatically. That’s the dream.

That said, I wouldn’t expect swift doc to immediately do all this (for example, as you mentioned, merging is still experimental) - that’s fine by me, honestly. I’m already used to having to kill and restart the preview server whenever changing a Swift file. I’d expect swift doc to do whatever the plugin preview command does today, and improve from there.

But I’d much rather get the simple workflow through swift doc even before it’s perfect - and I understand it’ll keep evolving with every version of the toolchain. Since all I ever manually do is preview. Archive building all otherwise happens in CI, so I’m not bothered if those commands are convoluted, I’m only affected by how easy/difficult it is to preview docs locally for a package.

3 Likes
27

I think we have the same goals and hopes for this feature.

As long as this pitch doesn't go into detail about which component is responsible for which behavior and there isn't an expectation that everything in this pitch will be available immediately, I think that there is sufficient freedom to design and implement this new preview command in a way that doesn't accidentally carrying over the constraints of the DocC plugin's "preview-documentation" command.

2 Likes