Use cases for combined documentation of multiple targets in Swift-DocC

[ronnqvist (David Rönnqvist)]

Hello everyone. I would like kick off a discussion about linking between documentation for multiple targets and hosting documentation for multiple targets together.

As a first step I would like to focus the discussion on goals and use cases. Two example use cases that we aim to solve are:

• A team with two apps and a shared framework should be able to write documentation links from each app to the framework but not between the apps or from the framework to either of the apps (more on this later).
• A developer of a Swift package should be able to host documentation for all the targets in that package in one place.

Goals

A few key goals for combined documentation of multiple targets include:

A target’s documentation should be able to link to that target's dependencies' documentation

Just like how targets can use dependencies’ public symbols, the target's documentation should be able to link to public symbols, articles, and tutorials in its dependencies' documentation. The syntax to link to a dependency's documentation and local documentation should be the same. Developers should be able to link to a dependencies' documentation even if the dependency is defined outside of the current package or project.

Developers should be able to host multiple related targets in a single documentation archive.

Developers should be able to create documentation archives which contain documentation for multiple targets. Developers should have control over which targets are part of the multi-target documentation archive and which aren't. Developers should be able to create combined documentation archives for multiple targets even if the targets are not part of the same build workflow.

Documentation for targets sub-hierarchies should continue to be buildable as separate units

If a target can be imported individually it can’t require that its documentation be build together with other related targets (unless they are dependencies of that target). The other targets may not even be buildable in the current build workflow due to differences in platform or SDK requirements.

Documentation archives should not contain known broken links

Links to documentation in other targets are only valid if both are hosted next to each other or if both are hosted together. Developers should have the ability to remove links to targets that won’t be hosted together.

Maintain as much flexibility as possible for direct callers

Developers who write their own tools and scripts that interact with DocC directly should have as much flexibility as possible to define dependencies between targets and define what targets are part of multi-target documentation archives.

Use cases

Below is a list of some of the use cases that we have considered. If there are important use cases that we have missed we would like to hear about it.

Swift package authors documenting their packages

A developer of a Swift package with multiple targets would continue to write documentation for each target, now with the additional ability for each target to link to documentation for its dependencies. They should be able to combine all their targets' documentation into a single documentation archive so that it can all be hosted in one place.

Developers of projects in separate repos that are conceptually related

Projects with conceptually related components across multiple different repos won't necessarilly be part of any single build workflow. They should still be able combine the documentation for these conceptually related targets into one combined documentation archive so that the documentation for all targets can be hosted together.

App, tool, or framework developers wanting to link to shared code and other dependencies

Similar to Swift package developers; teams working on apps, tools, or frameworks would write documentation for each target like with the new ability for each target to link to documentation for its dependencies. Regardless if the dependency is from the same project or an external dependency, developers should be able to link to its documentation. Declarations in the developer's target that use dependencies' public types should form automatic links to those symbols' documentation. The app developer team can optionally host their combined app and framework documentation but they can also read the locally built documentation if that's a better fit for their workflow.

Organizations wanting to host many unrelated documentation archives

Whether it's a group with many teams wanting to host each team's documentation in one place or projects like the Swift Package Index, we don't believe that documentation for unrelated targets or projects should be combined into one large documentation archive. However, some changes for multi-target documentation archives will also make it easier to host multiple separate documentation archives side-by-side.

Custom integrations

Developers who don't rely on build workflows to build their documentation have the ability to define their documentation "dependencies" and control how to combine the documentation into a single archive in their custom tooling. Documentation dependencies doesn't have to match target dependencies. A custom integration could build documentation with a dependency one way and then build it again with the dependency the other way but it wouldn't result in unrestricted back-and-forth linking between the two.

Frameworks or libraries that have example apps

DocC doesn't have specific support for documentation with example projects but it's a feature we'd like to support in the future and it's worth thinking about how an example app and a framework could link to each other. Following the target dependency order; documentation for the example app could link to symbols in the framework or library but the framework of library wouldn't be able to link to symbols in the example app. The framework likely doesn't need to link to individual symbols in the example app but it would be useful to be able to link to the example as a whole from the framework documentation that example app is showcasing. Depending on how documentation with example projects would work—which is far outside the scope for this proposal—there are different ways to address this. It may even be that the page that describe the example project is part of the frameworks documentation and not the example's documentation.

Use cases for future directions

Two important uses cases for future directions—which may be added over time—are:

Developers linking to documentation for SDKs or other pre-built dependencies

App developers who use functionality from SDKs or other pre-built dependencies should be able to link to documentation for those dependencies as well. This would, for example, allow links to protocols in the Swift standard library that the developer's code conforms to.

Connecting hosted documentation with fully qualified links

When a developer hosts their documentation for a target that has external dependencies with their own hosted documentation, the target's hosted documentation should form fully qualified links to the dependencies' hosted documentation. Readers of the target's hosted documentation can click through to the dependencies' hosted documentation.

Call to action

Are there use cases or goals for combined documentation of multiple targets that we've missed? We want to know about them so that upcoming discussions about solutions can focus on the right problems.

[mikenachbaur-okta (Mike Nachbaur)]

Almost all of the above use-cases exactly fit my required scenarios. I maintain several OSS SDKs, across different repositories (some have multiple targets in a single repo, others only one), but they all are interrelated. Each repository has one or more sample applications that could benefit from linking to their associated content.

I have two thoughts to consider:

1. It would be great to be able to link to Apple's own frameworks for documentation. For example, one of my structs has a convenience property that returns a PersonNameComponents object to be used to format a name by the developer. It would be convenient to be able to link to the official documentation in a consistent fashion.

2. Have you considered documentation versioning? We publish our docs to GitHub Pages when PRs are merged into main, and when we tag a new release. This way we can have our bleeding-edge documentation available, and documentation scoped to the features available for a given release.

Either way, I'm very excited for this new feature, and can't wait for it to become available!

[taylorswift]

@ronnqvist

Hello everyone. I would like kick off a discussion about linking between documentation for multiple targets and hosting documentation for multiple targets together.

As a first step I would like to focus the discussion on goals and use cases. Two example use cases that we aim to solve are:

• A team with two apps and a shared framework should be able to write documentation links from each app to the framework but not between the apps or from the framework to either of the apps (more on this later).
• A developer of a Swift package should be able to host documentation for all the targets in that package in one place.

it might help to look to swift-biome, and the instance of it deployed on swiftinit.org as an example of what some of these features might look like. (please note, the repository README is extremely out-of-date, but there are up-to-date instructions for how to use Biome at the bottom of swift-json’s README.)

@ronnqvist

A target’s documentation should be able to link to that target's dependencies' documentation

Just like how targets can use dependencies’ public symbols, the target's documentation should be able to link to public symbols, articles, and tutorials in its dependencies' documentation. The syntax to link to a dependency's documentation and local documentation should be the same. Developers should be able to link to a dependencies' documentation even if the dependency is defined outside of the current package or project.

Swift Biome has had this capability since v0.2.0. for an example, see this page from the docs for swift-json, which contains symbol-links referencing declarations in the Grammar module, which resides in a separate package.

because different packages can define modules with colliding names, Biome can also filter the results by the module’s dependency graph. Biome also supports the @import(_:) markdown block directive (example here) for even finer-grained control, and this block directive can be used in both inline and out-of-line documentation.

@ronnqvist

Developers should be able to host multiple related targets in a single documentation archive.

Biome currently supports this on a modulewise basis, though it doesn’t support combined package archives. you can refer to the swift-biome/ecosystem repository for an example of what this looks like in production.

@ronnqvist

Documentation archives should not contain known broken links

the Biome compiler resolves links ahead of time when symbolgraphs are registered into its database. this is deeply intertwined with documentation versioning and dynamic URL routing.

link breakage is quite common in real-world swift packages; the most common situation is when library authors add function overloads to an API, which breaks all existing links to that API, since every link now requires disambiguation. Biome supports fallback resolutions in the form of disambiguation pages when this happens.

@mikenachbaur-okta

Have you considered documentation versioning? We publish our docs to GitHub Pages when PRs are merged into main, and when we tag a new release. This way we can have our bleeding-edge documentation available, and documentation scoped to the features available for a given release.

Biome has supported documentation versioning since v0.3.0. you can see a live example of it in the swift-nio docs!

[Joseph_Heck]

You've covered a large breadth of what I'd like, and sometimes have tried, to do within DocC in the list above.

I'm heavily using Github Pages to host my content, and would happily reference a site that builds and hosts library documentation (such as Swift Package Index) in a cross-link, or link directly to another libraries documentation site - if they have something publicly available they'd prefer to be used. I do wish that could be easily combined with a version generated from the specific dependency I've resolved, but even referencing a constantly-latest version would be better than the hard-coded HTML links that are the option today.

I've personally been less concerned about hosting documentation (including dependencies) myself, and quite willing to have some link constructs going to external sites (the most common desire is to reference protocols or classes that Apple provides - standard library, Foundation, or some popular frameworks (SwiftUI, Combine, etc) that you can extended). That said, I've recently run into the issue where I want to reference the documentation for a specific version of the documentation.

In this case, I used another OSS library as a dependency to my own library, and as that library evolved I wanted to reference the version I was using as a dependency - The dependency library evolved sufficiently that my references and links had become confusing.

I've been breaking up libraries into subcomponents to keep them focused and specific, breaking pieces out into separate targets, and sometimes into separate packages (to be used as a dependency). In those cases, I want to reference documentation of the dependency when writing the docs and details for the library that's using it.

Example Apps (or code in general) are the hardest to arrange currently, and in particular I'd like to be able to have an example app (executable) that uses a library I've written, and pull in segments of that app's code to walk someone new to the library through a specific example of the library being used. The upcoming Snippets feature will get a fair way there, as I desperately want to verify my examples are (still!) compiling as a library evolves. As that becomes available, I can see where I'd very much like to reference a snippet from a dependency (primarily point to it, for more information/example kind of thing).

I can see that tutorials appear to provide the clearest way to highlight a walk-through like experience on how to use a library with an example app (Apple's examples being brilliant), but haven't stepped into that yet because of the step-function of complexity and difficulty in getting that set up. That said, I'd like to provide a sample/example app that uses one of my libraries as an example, build docs for it (potentially the tutorial content) and refer back to the reference documentation OF that library for the content that goes into that tutorial.

[josephduffy]

I'm mainly hoping to host documentation for multiple libraries in the same package so this covers my use cases.

I don't know if it's covered by this change since it's related to Xcode but I'm also hoping that this would allow links to external (to the framework) symbols to be links. If this linking is powered by DocC then I would guess that it would work, otherwise I realise this is not in the scope of this conversation.

I also wanted to point to some other places this has been discussed:

https://github.com/apple/swift-docc-plugin/issues/12

which led to:

https://github.com/apple/swift-docc/issues/255

[0xTim (Tim)]

We'd love to see this for Vapor. We're currently made up of ~40 packages and top level packages link down to dependencies, both ones created by us and dependencies like NIO and AsyncHTTPClient.

This is definitely one of the things blocking us from adopting DocC, at least for anything other than simple API docs. (There are plenty of other blockers too which can be saved for other threads)

[ronnqvist (David Rönnqvist)]

@mikenachbaur-okta

It would be great to be able to link to Apple's own frameworks for documentation. For example, one of my structs has a convenience property that returns a PersonNameComponents object to be used to format a name by the developer. It would be convenient to be able to link to the official documentation in a consistent fashion.

Agreed. In the original post I meant for the "Developers linking to documentation for SDKs or other pre-built dependencies" use case to cover linking to documentation for frameworks like Foundation or to documentation for the Swift standard library.

@mikenachbaur-okta

Have you considered documentation versioning? We publish our docs to GitHub Pages when PRs are merged into main, and when we tag a new release. This way we can have our bleeding-edge documentation available, and documentation scoped to the features available for a given release.

Yes. Although it's never mentioned in the original post I have been thinking about documentation versioning. I think this probably going to be necessary in order to support the "Connecting hosted documentation with fully qualified links" use case without breaking the "Documentation archives should not contain known broken links" goal.

The ability to browse versioned documentation is probably distinct enough that it should be spelled out as its own use case for future directions. Thank you for bringing that up.

[ronnqvist (David Rönnqvist)]

@Joseph_Heck

I've personally been less concerned about hosting documentation (including dependencies) myself, and quite willing to have some link constructs going to external sites (the most common desire is to reference protocols or classes that Apple provides - standard library, Foundation, or some popular frameworks (SwiftUI, Combine, etc) that you can extended). ...

I feel that it's an unspoken goal that developers shouldn't need to host their dependencies documentation. It's a little less clear—and something worth discussing—what DocC should do about links to dependencies that don't have hosted documentation (or where DocC can't know that the dependency has hosted documentation).

@Joseph_Heck

...That said, I've recently run into the issue where I want to reference the documentation for a specific version of the documentation.

In this case, I used another OSS library as a dependency to my own library, and as that library evolved I wanted to reference the version I was using as a dependency - The dependency library evolved sufficiently that my references and links had become confusing.

I'll take this as another reason why we should consider versioned documentation as an important use case and make sure it's part of the design discussion.

@Joseph_Heck

Example Apps (or code in general) are the hardest to arrange currently, and in particular I'd like to be able to have an example app (executable) that uses a library I've written, and pull in segments of that app's code to walk someone new to the library through a specific example of the library being used. The upcoming Snippets feature will get a fair way there, as I desperately want to verify my examples are (still!) compiling as a library evolves. As that becomes available, I can see where I'd very much like to reference a snippet from a dependency (primarily point to it, for more information/example kind of thing).

I want to try and understand what the cross-linking impacts are for this. Are there use cases that you feel wouldn't be solved by Snippets in the library documentation? Specifically, are there use cases that would be solved by linking to pages or other elements in the example app's potential documentation?

[ronnqvist (David Rönnqvist)]

@josephduffy

I don't know if it's covered by this change since it's related to Xcode but I'm also hoping that this would allow links to external (to the framework) symbols to be links. If this linking is powered by DocC then I would guess that it would work, otherwise I realise this is not in the scope of this conversation.

So far in this topic we're trying to uncover what use cases various people have for combined documentation so that we're not missing anything in the design discussion.

If you are asking about linking to frameworks linking to other frameworks or libraries that they depend on then that is covered by the "A target’s documentation should be able to link to that target's dependencies' documentation" goal.

If you have use cases for a framework linking to another framework without any explicit target dependency between the two frameworks I would love to know more about it so that we can consider that use case in the upcoming design discussion.

[taylorswift]

@ronnqvist

If you have use cases for a framework linking to another framework without any explicit target dependency between the two frameworks I would love to know more about it so that we can consider that use case in the upcoming design discussion.

swift-json’s docs have tooltips that reference SwiftNIO types, even though the library itself has no SwiftNIO dependency.

JSON.Rule<Location>

The swift-grammar and swift-json libraries are transparent! This means that its parsing rules are always zero-cost abstractions, even when applied to third-party collection types, like /swift-nio/NIOCore/ByteBufferView.

doccomment source

/// >   Tip: 
///     The ``/swift-grammar`` and ``/swift-json`` libraries are transparent!
///     This means that its parsing rules are always zero-cost abstractions, 
///     even when applied to third-party collection types, like 
///     ``/swift-nio/NIOCore/ByteBufferView``.

the /swift-nio/NIOCore/ByteBufferView symbol link used to work in Biome v0.2. in fact it is actually a valid URI if you copy and paste it into a browser.

but it is not linked anymore because Biome now registers packages in topological sort order, and swift-nio is no longer available before swift-json is registered.

[ktoso (Konrad 'ktoso' Malawski 🐟🏴‍☠️)]

Others have already chimed in about the usual package author cases but I'll add more examples:

Cross-library (cross-package) links

This is the "A target’s documentation should be able to link to that target's dependencies' documentation" case you mention.

However I'd like to give a few more concrete examples as well as how we'd want to use these.

So libraries which use e.g. Logging (GitHub - apple/swift-log: A Logging API for Swift) would want to be able to link to them like:

``Logging/Logger``

However, not every single library using logging should replicate the logging docs (!), that's bad for SEO, instead they should link to "wherever the logger docs are hosted".

docc by itself can't know where to link to, as it depends where other libraries deploy/host their documentation. Most will do so on github pages or on packageindex.com (like so RevenueCat Documentation – Swift Package Index random example).

We will need to configure the baseURL for links to be formed, therefore I suggest a mapping file like this:

Side note: Preferably this would be configured in the SwiftPM plugin via a nice type-safe configuration, but we're lacking this feature still in SwiftPM, so for now let's go with an external configuration file.

We'd need to configure "when trying to link docs of Logging (which we know is from package swift-log), use this base url", so a mapping file could look something like this (details open to discussion of course):

doccHostingLocations:
  // defaultBaseURL: "..." // could be set optionally if most follow some pattern
  packages: 
    - name: swift-distributed-actors
      baseURL: "https://apple.github.io/swift-distributed-actors/$VERSION/documentation/$TARGET"
    - name: swift-log
      baseURL: "https://apple.github.io/swift-log/$VERSION/documentation/$TARGET"
    - name: swift-metrics
      baseURL: "https://apple.github.io/swift-metrics/docs/$VERSION/documentation/$TARGET"
   - name: mqtt-nio
     baseURL: "https://swiftpackageindex.com/swift-server-community/mqtt-nio/main/documentation/$TARGET"

Note the mqtt-nio example didn't support $VERSION since it only hosts main version for now... Other projects in my example do host docs for specific versions though, so that's why the baseURLs contain $VERSION. As we build docc documentation, we know what versions we depend on and create well-formed links to the exact versions MY project is depending on.

There could be fallbackURL as well, if $VERSION was not published, we could link check them and offer the fallback link but that's future work...

This allows us to strive in an ecosystem regardless where documentation ends up hosted. It could be all on github pages, or all on the package index, or on my website because I happen to own a company that also has swift APIs and want to host it on my site for whatever reason.

Prior art:

This is how it is solved in scaladoc, by e.g. the GitHub - ThoughtWorksInc/sbt-api-mappings: An Sbt plugin that fills apiMappings for common Scala libraries. sbt plugin.

Other examples:

• GitHub - apple/swift-distributed-tracing: Instrumentation library for Swift server applications needs to link to GitHub - apple/swift-service-context: Minimal type-safe context propagation container
• tons of projects need to link to GitHub - apple/swift-log: A Logging API for Swift and GitHub - apple/swift-metrics: Metrics API for Swift
• GitHub - apple/swift-distributed-actors: Peer-to-peer cluster implementation for Swift Distributed Actors needs to link to GitHub - apple/swift-nio: Event-driven network application framework for high performance protocol servers & clients, non-blocking.

Multiple target single-package documentation

This is the "Developers should be able to host multiple related targets in a single documentation archive." case, and I'd just like to give some examples:

Projects like NIO or swift-distributed-actors have their functionality split across many modules. For example:

• NIOCore - https://apple.github.io/swift-nio/docs/current/NIOCore/index.html (https://apple.github.io/swift-nio/docs/current/NIOCore/index.html)
• NIOHTTP1 - https://apple.github.io/swift-nio/docs/current/NIOHTTP1/index.html (https://apple.github.io/swift-nio/docs/current/NIOHTTP1/index.html)

are all the same package, and very often need to cross link between eachother; e.g. A ChannelPipeline documented in NIOHTTP1 needs to refer to NIOCore's EventLoopFuture https://apple.github.io/swift-nio/docs/current/NIOHTTP1/Extensions/ChannelPipeline.html#/s:7NIOCore15ChannelPipelineC8NIOHTTP1E21addHTTPClientHandlers8position21leftOverBytesStrategyAA15EventLoopFutureCyytGAC8PositionO_AD018RemoveAfterUpgradeL0OtF

Such links should work, and they are all hosted in the same place.

Prior art

In a previous life, during my Akka work, we solved this using a build plugin we developed called "unidoc" which collapsed all docs from many modules into a single docs page. Just for reference: GitHub - sbt/sbt-unidoc: sbt plugin to create a unified Scaladoc or Javadoc API document across multiple subprojects.

docc likely can do better than that here, and host them still separately, but know how to link between the targets/modules.

Re: Documentation archives should not contain known broken links

Links to documentation in other targets are only valid if both are hosted next to each other or if both are hosted together. Developers should have the ability to remove links to targets that won’t be hosted together.

This one I wanted to understand better... I agree that dead links are bad, but I would like to make sure that this is an option and not the only way. We definitely want to link across packages (see the first writeup in my post here).

I believe what this also means is that while we build documentation together, we can therefore issue warnings for bad symbol links to a dependency; once that has been built, we can assume the documentation of the dependency at the same version we built against, will be valid at the published url (that we point at with the baseURL).

"Landing page" for docs

You hint at it somewhat but it wasn't explicitly stated so I'd like to ask:

Generally when "hosting many documentation of many, related, modules" we'd want to have a landing page for such pages. In the sense of "Welcome to MyLibrary! This library does..." which is not strictly part of documentation of any specific target, but an overall documentation of the package. Which then goes on and links into specific targets.

This may be is a separate feature to talk about, but having to designate one of the targets "as the main one" in order to put the landing page in there is a bit weird and I'd love to improve upon this.

--

Hope these help and I'm really looking forward to all the great docs ecosystem we can get started building using these improvements

[ktoso (Konrad 'ktoso' Malawski 🐟🏴‍☠️)]

Ah, and side-note we'd very much want to be able to link to swift docs as well: https://developer.apple.com/documentation/swift/actor etc. I believe this would fit the "baseURL" mappings as well.

I forgot to add this to the first example, but this could basically be solved using the same mechanism:

doccHostingLocations:
  modules:
    - name: Swift
      baseURL: "https://developer.apple.com/documentation/swift"
    - name: Distributed
      baseURL: "https://developer.apple.com/documentation/distributed"

etc.

Since those modules are "well known" those we could just automatically understand and link to developer docs, and not need them to be configured, but the mechanism would be the same as people configuring module baseURLs explicitly.

The same applies to the SDK and modules within it, like Foundation etc.

[ronnqvist (David Rönnqvist)]

@ktoso

However, not every single library using logging should replicate the logging docs (!), that's bad for SEO, instead they should link to "wherever the logger docs are hosted".

Yes, for a variety of reasons developers shouldn't host their dependencies' documentation.

@ktoso

Projects like NIO or swift-distributed-actors have their functionality split across many modules. For example:

• NIOCore - NIOCore Reference (https://apple.github.io/swift-nio/docs/current/NIOCore/index.html)
• NIOHTTP1 - NIOHTTP1 Reference (https://apple.github.io/swift-nio/docs/current/NIOHTTP1/index.html)

are all the same package, and very often need to cross link between eachother; e.g. A ChannelPipeline documented in NIOHTTP1 needs to refer to NIOCore's EventLoopFuture ChannelPipeline Extension Reference

AFAICT this link is from a target so a module so that the primary use case that we're thinking of. Do you know of any examples where NIO or swift-distributed-actors need to cross link to non-dependencies?

@ktoso

Links to documentation in other targets are only valid if both are hosted next to each other or if both are hosted together. Developers should have the ability to remove links to targets that won’t be hosted together.

This one I wanted to understand better... I agree that dead links are bad, but I would like to make sure that this is an option and not the only way. We definitely want to link across packages (see the first writeup in my post here).

I believe what this also means is that while we build documentation together, we can therefore issue warnings for bad symbol links to a dependency; once that has been built, we can assume the documentation of the dependency at the same version we built against, will be valid at the published url (that we point at with the baseURL).

My thoughts on this is that based on some signal from the developer about which targets will be hosted together and which targets have hosted documentation at known locations (ideally version specific hosted documentation) DocC would be able to avoid or remove links that it knows would be broken when the documentation is hosted. The details are left for upcoming discussions but it's a simultaneous goal that developers should have control over this, either as direct configuration or as the ability to call DocC directly with custom arguments from a script.

@ktoso

"Landing page" for docs

You hint at it somewhat but it wasn't explicitly stated so I'd like to ask:

Generally when "hosting many documentation of many, related, modules" we'd want to have a landing page for such pages. In the sense of "Welcome to MyLibrary! This library does..." which is not strictly part of documentation of any specific target, but an overall documentation of the package. Which then goes on and links into specific targets.

This may be is a separate feature to talk about, but having to designate one of the targets "as the main one" in order to put the landing page in there is a bit weird and I'd love to improve upon this.

Yes, the intention is that combined documentation would have some form of landing page(s) that are either synthesized or authored or a combination of both. This will probably have its own detailed design discussion since it's a fairly well isolated and developer facing component.

[taylorswift]

@ktoso

Generally when "hosting many documentation of many, related, modules" we'd want to have a landing page for such pages. In the sense of "Welcome to MyLibrary! This library does..." which is not strictly part of documentation of any specific target, but an overall documentation of the package. Which then goes on and links into specific targets.

for many of my packages (like swift-png), this is what the README is for. could there be an option to just render the package README for a landing page?

@ronnqvist

AFAICT this link is from a target so a module so that the primary use case that we're thinking of. Do you know of any examples where NIO or swift-distributed-actors need to cross link to non-dependencies?

dependency graphs are a DAG, but when writing documentation, you often want to have link cycles. one known issue with Biome is docs in NIOHTTP1 can link to symbols in NIOCore, but not the other way around, because doing so would create a dependency cycle.

so far, i haven’t come up with a great solution for this. it wouldn’t be a major change to enable it, since Biome already compiles documentation in two passes, but allowing this would really increase the “blast radius” when documentation for a single package is updated, since everything referring to it would also need to be rebuilt. if swift-foo contains even a single link to swift-bar, then swift-foo’s docs need to be rebuilt every time swift-bar is modified, and this doesn’t scale. as @ktoso mentioned, this can get really complex if swift-bar is hosted externally.

documentation versioning does not solve this problem in practice, because it is actually often necessary to overwrite the documentation (and associated git tags) for a particular version multiple times.

[ronnqvist (David Rönnqvist)]

@taylorswift

dependency graphs are a DAG, but when writing documentation, you often want to have link cycles.

Do you have any examples of link cycles in documentation? I haven't found any but it's very hard to search for.

(I view link cycles and links to non-dependencies (like the swift-json example earlier) as two conceptually different cases to consider even if they may pose some of the same problems and share some of the same solutions.)

[taylorswift]

@ronnqvist

Do you have any examples of link cycles in documentation? I haven't found any but it's very hard to search for.

well, right now there wouldn’t be any because none of the 3 major documentation compilers support link cycles. but the README for swift-grammar starts with:

High-performance constructive parsing, in pure Swift. This module powers the swift-json library!

where swift-json is a consumer of swift-grammar.

---

edit: i forgot about @SDGGiesbrecht ’s compiler. i don’t know if it supports link cycles. i’d be interested to know if it does.

@ronnqvist

(I view link cycles and links to non-dependencies (like the swift-json example earlier) as two conceptually different cases to consider even if they may pose some of the same problems and share some of the same solutions.)

i don’t know if this would be a helpful distinction. the “limited” use-case you’re describing is basically just adding “documentation” dependency edges in addition to the normal edges. it would need corresponding dependency resolution logic, and some kind of counterpart to the Package.resolved format we currently have.

there is also a political problem where if package A decides to link to package B, then package B loses the right to link to package A, and the only reason it can’t is because the author of package A got there first.

[SDGGiesbrecht (Jeremy David Giesbrecht)]

I am not entirely sure what you are asking, partly due to confusion over what you meant by your link. That package does not seem relevant to your conversation. If you meant it as an example of the output of my documentation tool, the tool itself is described here.

Having read the last half dozen comments (but not the whole thread), I can describe the related bits of how my tool was designed, and hope that answers your question. (Although the tool itself has aged and I am in the slow process of shifting its guts over to DocC anyway.)

Unlike DocC, it was designed around packages not modules, and so the main page is itself a node in the graph, its “symbol” being the package declaration in the manifest. It knows about packages, libraries and targets by way of loading the manifest with SwiftPM, but then assembles the corresponding pages by scanning the manifest for matching declarations and extracting conventional documentation comments applied to them. While the documentation comments are familiar in format, as far as I know no other tool recognizes them, because they are technically attached to function calls, not declarations in the AST sense. The scan is also imperfect, as exotic formatting in the manifest can result in them being not found and hence being flagged as undocumented.

The tool expressly does not pick up the read‐me, but rather the other way around. It generates the read‐me along with the documentation. The reason for this is that a read‐me tends to contain a lot of information on one page that is better split up into separate pages when you have a whole website to work with. Whereas the documentation site has separate pages for the main description, importing instructions, and stuff about the author, the read‐me has them in sequential sections. The read‐me also starts with a list of supported platforms, and a link to the full documentation, which would be redundant on the documentation site, since the platform list is already in the navigation frame of every page, and the documentation link would just be circular.

Also unlike DocC, the tool does not build anything, but rather scans the AST with SwiftSyntax. This was a deliberate design decision in order to have accurate information about #if. It also means the tool has no knowledge of mangled names and no unique identifiers for nodes. The tool attempts to syntax‐highlight any code spans, including turning any known symbols into links. But due to having compiled nothing, this is imperfect and does not always do the right thing when name clashes occur. While the tool does scan dependencies (it knows about the standard and core libraries, and anything it learns from SwiftPM), it only does so to detect inheritance. It does not automatically link to anything outside the package, so calls to dependency functions appear the same as calls to hypothetical placeholders.

Manual links with fully qualified URLs can point anywhere, and that is what you have to do if you want to link to something in another package. Knowing both that users will do this, and that developers will refactor code, the tool attempts to guarantee the semi‐persistence of all page URLs. The directory structure of the site (and hence the path segment of a URL) is arranged logically. If the tool is aware of existing documentation (such as if the output directory already exists, or it is aimed at a gh-pages branch), then it starts by overwriting every file with a redirect to its parent. Then the new documentation is generated overtop, obliterating redirects wherever symbols still exist and leaving the rest behind. That way if an instance method is documented and linked to, but then removed, the otherwise dead link is now pointing at a redirect to the type page, whose list of methods most likely contains the removed method’s successor.

For the related projects section, a list of package URLs is supplied to the tool configuration. The configuration supports importing external packages, so an organization is expected to curate a list of URLs in a metadata package, and then have all of its real packages’ configurations import that metadata from the central location. The tool follows the URLs and extracts some data, but this happens outside of any dependency graph, and does not go looking for transitive related packages. While it almost always involves link cycles, it deals only with a flat list, not a deep graph. And nothing learned from related projects is fed back into to the source scanner or renderer, so the main documentation pages have no knowledge of their symbols.

[SDGGiesbrecht (Jeremy David Giesbrecht)]

I have started to think it would be useful for SwiftPM to have a means of declaring known clients for the sake of testing. It would serve a purpose similar to Swift’s source compatibility suite, and the listed clients would be treated almost as though they were the actual roots of separate graphs and the current package were instead an overridden local reference. Such a feature might be equally useful for your upward facing documentation links.

[ktoso (Konrad 'ktoso' Malawski 🐟🏴‍☠️)]

@ronnqvist

AFAICT this link is from a target so a module so that the primary use case that we're thinking of. Do you know of any examples where NIO or swift-distributed-actors need to cross link to non-dependencies?

Right that was within the same package examples.

Cross to non-dependencies I think we'd only ever do "hey, look at this other library if you're interested" but would not need specific symbol references. So those can be plain old href links I suppose.

Swift metrics and logging have examples of this:

API is designed to establish a standard that can be implemented by various metrics libraries which then post the metrics data to backends like Prometheus, Graphite, publish over statsd, write to disk, etc.

but as you can see those are just links to the other packages; we don't need to deep link into non-dependencies ever I think.

@ronnqvist

My thoughts on this is that based on some signal from the developer about which targets will be hosted together and which targets have hosted documentation at known locations (ideally version specific hosted documentation) DocC would be able to avoid or remove links that it knows would be broken when the documentation is hosted.

Sounds good to me!

@taylorswift

for many of my packages (like swift-png), this is what the README is for. could there be an option to just render the package README for a landing page?

Perhaps, I'd only be worried about making the readme not-readable if not docc rendered (people navigating to github, stash etc, the readme should look nice there too). David said that'll be a separate proposal so let's hash it out then though

[andrewtheis]

This problem can probably be solved in two stages with a relatively straightforward solution. Mind you, I’m not well versed in how the internals of DoC works, so how complex this would be to implement is not considered here.

First, Xcode already uses DocC to compile documentation for everything within a project or package in Xcode locally - regardless of whether it has adopted DocC fully or has remote documentation available. Therefore, using the existing Symbol/Reference/Format, it could be updated to support the target name as the first parameter - using the Package.swift file as a reference for SPM dependencies, the project file for target references, and a statically defined list for system frameworks.

• Foundation/String
• swift-nio/NIOCore/Channel
• WidgetExtension/MyWidget

Second, would be to figure out how these references translate to links in remote documentation. System frameworks would be easy, as Apple knows the URLs these are located at and can infer the version based on minimum supported system version of the package or project.

Targets and other local packages could dump their docs into a known folder format that can be referenced by relative href links. Whatever mechanism Xcode uses to determine all targets and local packages within a project/package could be ported to DocC (I assume there is something that is just invoking the docc command for each target after some scan of the project and package file to spit out local documentation).

External packages would need to define where documentation lives. This would probably make the most sense to live inside the Package.swift file as a new parameter at the top level. When documentation is generated for that package it follows the same folder structure mentioned before, so it’s not necessary for the importing package to know anything other than a base URL.

Versioning makes things a bit more complicated, but what makes the most sense would be a requirement that the documentation must be hosted in a way that mirrors the package’s Git tree - meaning that DocC would need to run by a CI step that generates documentation for every commit pushed to the package repository, then dump that in some known folder structure - maybe just commit sha/tags at a top level. The importing package can determine links by looking at its Package.resolved file, to know exactly which version of the docs to point to - whether that’s a version tag or commit SHA.

Alternatively, and what probably makes more sense, is documentation could be generated locally somehow and pushed up as part of the commits to the package itself. This would prevent the documentation from ballooning in size as a separate copy for each commit wouldn’t be needed, it would just diff alongside the rest of the source changes. This would require some implementation of local build step to generate the documentation in a way that developers don’t need to think about doing this, it just happens on a build and the /docs folder changes are there when the developer goes to commit their source changes.

Either of these could be cumbersome for the package author depending on how this is implemented, but it’s the only way to know for sure that a symbol reference won’t be broken if pointing to some arbitrary commit of a package.

These solutions make the most sense to me, if they’re possible to implement. It should be tightly coupled to the existing Package manifest system, Package resolution system, the package Git repository, etc. It should be straightforward for package authors to supply documentation in a way that requires minimal additional information to be provided (ie: no documentation manifest files, no hard coded links, etc).