Hi. This is a continuation of this topic which focused on goals and use cases for combined documentation of multiple targets. In this new topic I'd like to focus on the technical aspects of how to build combined documentation for multiple targets and find a high level design with the right tradeoffs to support the goals and use cases from the previous thread.
The "sloth" development team is working on a Swift package that contains 4 libraries; "A", "B", "C", and "D" which depend on each other as follows:
A ├─ B │ ╰─ C ╰─ D
The team also has two related Swift packages with one library each; "E" and "F".
The targets in these 3 Swift packages are all conceptually related as one project "A-F" so the team host the documentation for all the 6 targets in one place. The combined documentation has an overview page that explains what the project is about, links to each target's documentation, and links to a handful of repositories from other team's who use this project.
The "capybara" team is working on an app and a framework. The framework depend on the "B" target from one of the "sloth" team's Swift packages but doesn't depend the "A" or "D" targets. In the "capybara" team's project, the targets depend on each other as follows:
App ╰─ Framework ╰─ ─ ─ ─ ─ ─ ─ ─ B (external dependency) ╰─ C
Public types from "B" appear as arguments and return values in function declarations in the "capybara" team's framework and some types in the framework conform to public protocols from "B". When the team locally builds documentation it also builds documentation for "B" and "C". Following a link from the locally built framework documentation navigates to the locally built documentation for "B" and from there to "C".
The "capybara" team's company hosts documentation for shared code on an internal website and the team's Framework documentation is hosted there. The team doesn't re-host the "B" and "C" documentation. Instead the links from the Framework to "B" is replaced with absolute links to where the "sloth" team hosts their documentation for the "B" target.
Other teams at the company also host documentation for their shared code on the internal website. The "antelope" team hosts documentation for a big framework with 3 example projects and a handful of snippets to help explain how to best use the different parts of the big frameworks's API.
This section gives an overview of how Swift-DocC works, establishes some terminology, and describes how Swift-DocC fit into scripts and build workflows.
Viewed in isolation, Swift-DocC is a documentation compiler that takes symbol information, markup, and media as input and outputs a directory of documentation data that can be hosted to view the rendered documentation. We call the directory of input files a "documentation catalog" and the directory of output files a "documentation archive".
One way to pass symbol graph files to DocC is as files in the documentation catalog but, because of how DocC is integrated into scripts and build workflows, it's more common to pass a separate directory of "additional" symbol graph files and only use the documentation catalog for markup files and media.
Both the documentation catalog and the symbol graph files is optional input but at least one of them is needed to build documentation. Regardless of how input is passed to DocC, one call to
docc convert creates one documentation archive. A documentation archive may cover one or more modules (target documentation), tutorials, and/or manually designated "technology roots".
For the remainder of this post I'll be using "target" in place of "module" to unify the terminology used in build workflows and in DocC.
Some use case pass more than one symbol graph file to DocC. The two main cases are:
- To combine information for one target across multiple source languages.
- To combine information for one target across multiple platforms.
It's possible— although not officially supported and with some issues—to pass symbol graph files for multiple targets to one
docc convert call. In this case the output documentation archive will contain documentation for all targets.
DocC uses links to connect documentation pages. Links in "Topics sections" (level-2 headings titled “Topics”) are used to define the documentation hierarchy. We call this "curation". DocC also creates links based on some references found in symbol graph files, for example for types in declarations.
All inputs in a documentation build can link to each other. Links are resolved relative to the page where they're written but developers can write links with absolute paths to refer to pages from different top-level documentation hierarchies.
DocC can also link to "external documentation sources" but doesn't ship with any concrete implementations that interface with any specific sources. Content from external documentation sources doesn't become their own pages in the documentation archive.
DocC can optionally write an extra file into the documentation archive that lists each page, and on-page landmark, that can be linked to along with a summary of its content. We call this file the "linkable entities" file.
Most developers don't interact with DocC directly. Instead their documentation workflows are connected to their build workflows, leveraging systems like Swift Package Manager or Xcode to extract symbol information and pass the relevant inputs to DocC. Targets in these systems may have dependencies and can use public symbols from their dependencies. Circular dependencies are not allowed between these targets.
Two different strategies—each with their own tradeoffs—could both be used as the foundation for building combined documentation for multiple targets:
- Making a single call to DocC with one documentation catalog and the combined symbol graph files for multiple targets.
- Making separate calls to DocC with separate documentation catalogs for each target and only that target’s symbol graph files and then combining the documentation archives into a combined archive.
It's my opinion that the design for this feature needs to deeply consider how it will work when DocC is integrated with a build workflow. When considering DocC in this context I feel that the single call strategy opens up to too many possible problems to be worth the benefits that it provides to DocC in a direct-call context. More on this in the Alternatives considered section.
Because of this I would like to propose a solution that's based around separate calls to
docc with separate documentation catalogs for each target.
All proposed names for actions, command line arguments, types in Swift-DocC, etc. are placeholder names.
To support linking between targets, I propose adding a new
--dependency argument to the
docc convert (and
docc preview ) actions. Each dependency argument would pass the file path of a dependency’s' already built documentation archive. DocC will treat dependencies' documentation as external sources of documentation—meaning that resolved content does not result in distinct pages in the documentation archive—and will use a new
DocumentationArchiveResolver resolver (that conforms to the existing
ExternalSymbolResolver protocols) to resolve both links written by developers and links from references in symbol graph files to content from dependencies' documentation.
When building multi-target documentation as part of a build workflow each target's documentation would depend on its dependencies' documentation. This means that each target can link to its direct dependencies but not their dependencies.
From the example at the top of this post, the “B” target can link to symbols in “C” and but not symbols in “A”, “D”, “E”, or “F”.
Building documentation for “B” and “C” would result in two separate documentation archives where “B” has relative links to “C”. These local links would be valid as long as both documentation archives are “hosted”, for example by a local preview server or local renderer (like the documentation window in Xcode).
Developers can specify documentation dependencies in any order when calling DocC directly, for example in scripts.
To write links to other targets’ documentation, developers can use the existing
doc: link syntax and use other documentation archive's unique identifier as the "host" of the documentation URI.
doc://com.example/path/to/documentation/page#optional-heading ╰─────┬─────╯╰────────────┬────────────╯╰───────┬───────╯ bundle ID path in docs hierarchy heading name
As a more convenient way to link to other target's documentation, I propose that the existing symbol link syntax be extended with a leading slash for links to symbols in other targets. A leading slash for links to symbols in the current target would be optional but supported. For example, to link to the "something" property of "SomeClass" in "SomeDependency" from the documentation for "MyTarget", I would write:
If "MyTarget" have a symbol also named "SomeDependency" I would like to it with:
In addition to "authored" links, automatic links from references in symbol graph files should resolve to dependencies symbols. By registering a resolver conforming to
ExternalSymbolResolver , DocC will let the resolver try to resolve all symbol references from that aren't found in any local symbol graph file. For the resolver to have this data, the documentation archive would need to contain a list of all the symbol page's unique identifiers. I propose that these identifiers be added to the “linkable entities” file (the file that lists each page, and on-page landmark, that can be linked to) and that this file will be created by default.
To create a combined documentation archive for multiple targets I propose adding a new
docc merge action. The merge action will take 3 types of inputs:
- A list of documentation archives to be combined.
- An optional documentation catalog of conceptual (non-symbol) content for the “landing page” of the combined documentation
- Optional configuration for how to handle links to documentation archives other than the input list.
The output of the merge action will be a new documentation archive with the data for the combined documentation. The combined archive will contain separate data hierarchies, navigation hierarchies, and possibly themes for each target.
If no documentation catalog is passed, DocC will generate top-level "/documentation" and "/tutorial" pages, as applicable, that will list all the targets and all the tutorials respectively. If a documentation catalog is passed, the developer is responsible for linking to each target and/or tutorial as they see fit. The merge action’s documentation catalog can link to documentation from all its input documentation archives (same as convert action dependencies).
When merging documentation archives, cross-target links to documentation archives that that are not passed as input will be transformed by DocC. Follow up proposals should cover both versioned documentation and how DocC can know where external documentation is hosted so that DocC can transform links to this documentation into fully resolved absolute links to the hosted documentation. Until then, DocC will default to transforming links into non-links (but keeping the resolved titles, text style, and abstracts as applicable). Developers who know where an external target (a target that’s not part of the combined documentation archive) hosts its documentation can override this behavior by providing a documentation base URL per-target.
Passing a combined documentation archive as input to
docc merge will remove its landing pages and replace them with new landing pages based on that
docc merge action's inputs.
Developers can mix build workflows—that create per-target documentation archives with links—and scripts or workflows that call DocC directly to create combined documentation archives. This can for example be used to combine documentation across multiple repositories or to combine documentation for targets that cannot be built in a single build workflow (for example due to differences in platform requirements).
For cases where all targets can be built together, the systems where DocC is integrated could let developers configure which targets to combine documentation and run the
docc merge action as part of the build workflow.
Future proposals should cover how to link to symbols in SDKs and other pre-built dependencies and how DocC can transform these links to fully resolved absolute links to where that documentation is hosted. It’s likely that this would be implemented as another external resolver type in DocC.
The “sloth” team can write links between their target dependencies (from “A” to “B” and “D” and from “B” to “C”). If “A”, “E”, and “F” cannot be built in one build workflow the team can still leverage build workflow integrations to build documentation for “A”, “E”, and “F” separately. This avoids needing to extract symbol information directly. Building documentation for “A” also builds documentation for “B”, “C”, and “D”.
The team can customize the "landing pages" that describe the “A-F” project and can host all of their documentation together.
The team can’t use documentation links to highlight other projects that use their "A-F" project. Instead they would use https links. It's likely not more than one link per project—either to repository or its top-level hosted documentation.
When the “capybara” team builds their app documentation they also get local documentation for their framework and for the “B” and “C” targets. Since “B” and “C” targets have their separate documentation catalogs and only link in build-dependency-order, the “capybara” team’s local version of the “B” and “C” documentation appear the same as the "sloth" team's hosted documentation for these targets. No local documentation for “A”, “D”, “E”, of “F” is built.
When the “capybara” teams hosts the framework's documentation they have the choice of transforming links to “B” into non-links or into absolute links to the “sloth” team's hosted documentation.
When the “antelope” team builds documentation for their “Big Framework” and their 3 example apps, each example app can link to individual public symbols from the “Big Framework” but not to other app’s symbols. The “Big Framework” also can’t link to symbols in the example apps or use documentation links to reference the example apps. With the proposed solution the “Big Framework” could use https links to reference the top level page of each sample. Future proposals may cover new functionality specifically related to example apps that improve the ability for the framework to link to its examples.
This proposal aims to design the core support for building and hosting multi-target documentation. Other proposals should build on this to add more features. A few important future directions mentioned in the proposed solution are:
- Versioned documentation
- Creating fully resolved absolute web links to other hosted documentation.
- Linking to symbols in SDKs and other pre-built dependencies.
As mentioned above, an alternate solution could be based around a single
docc convert call with one larger multi-target documentation catalog and symbol graphs for multiple targets.
One key benefit of passing symbol graphs for multiple targets together is that links between targets can be bidirectional.
However, supporting bidirectional cross-target links introduce pose some new problems.
For example, if the “sloth” team had documented its “A-F” project so that the “B” target contained links to “A” and “F”. Then when the “capybara” team used “B” as a dependency and built documentation for their app in a build workflow, “A” would not be built and “F” wouldn’t even be locally cloned. This would mean that the local version of the “B” documentation would contain broken links to “A” and “F”.
Further, if “A-F” is documented in one documentation catalog then conceptual content related to targets other than “B” and “C” would be locally built and links to the other targets from this content would also be broken.
I'm against the idea that documentation workflows should ever build code that's not part of the build workflow but even if that was an option, there are no guarantees that "A" can build in every situation where "B" can. For example, "A" may have higher a required Swift versions or SDK versions than "B" or "A" may be a single platform target but "B" is a cross platform target and the current build workflow is not building for a platform that "A" supports.
Documentation with bidirectional cross-target links will encounter these issues as soon as (at least) one target can be imported individually. The way that developers would avoid this issue is to only write links in the target's dependency order.
Another benefit of a single
docc call for multiple targets is that the output is one documentation archive without introducing a new "merge" action.
However, if the only way to link between targets is to pass them to the same
docc call, then that means that linking to a target also embeds all that target's documentation into the documentation archive. This issue can be solved by also adding support for resolving external links to existing documentation archives (as described in the proposed solution).
Also, if the only way to combine documentation for multiple targets into a single documentation archive is to pass them to the same
docc call, then that means that targets that are not part of the same build workflow need to abandon the integration of DocC in the build workflow and construct the
docc convert call themselves. The developers could either redundantly build documentation in the build workflow to access the symbol graph files for each target that the build workflow created or they would need to construct the calls to generate the symbol graph files for each target themselves.
In the proposed solution, when a developer wants to combine documentation for targets that can’t be built in a single build workflow, they can still leverage the build workflows and only need to construct a
docc merge call themselves.
These issues could be solved by also adding a new "merge" action (as described in the proposed solution) to make it easier for developers to utilize build workflows to generate documentation and combine documentation for targets that they can't build in a single build workflow.
Performing the merge action on the documentation archive—as opposed to the in-memory documentation model—adds the responsibility to alternative output formats to also support merging as it applies to that format. However, most of the steps that this merge action would perform are specific to the documentation archive format and it's not clear if other output formats would need this. I'm not aware of any alternative output format implementations in forks of Swift-DocC, so it's hard to determine the impact this would have on other formats.
I don't think the separation of the "convert" actions work and the "merge" action's work would prevent future output formats or future intermediate format from being implemented.