Announcing Swift-DocC

This post is a continuation from the announcement on the Swift blog, with additional details for members of the Swift community that may want to contribute to Swift-DocC. To quickly summarize, Swift-DocC is a documentation system that parses Markdown comments within Swift code, as well as standalone article and tutorial files contained in the repository, to produce rich, inter-linked documentation.

The documentation team within Apple originally created Swift-DocC for internal authoring of SDK documentation and tutorials. Since those early iterations, the entire stack was re-engineered to be useful for the whole Swift developer community. Swift-DocC is included as part of Xcode 13, and today it officially launches as an open source project.

What makes Swift-DocC special

The main goal for the Swift-DocC project from the very beginning was to unify the documentation experience across API reference docs, conceptual articles, and tutorials. And of course, to manage all the links and interdependencies among all the related docs for the author.

Swift has supported writing API reference documentation, in the form of specially-formatted comments, from its early days. Extra features for these comments have long been supported in Xcode and other tools. The Swift community has since created popular documentation tools such as Jazzy and SwiftDoc, which extract these comments from a codebase and generate documentation websites. This existing support means that a lot of Swift files already have great comments written alongside the code. This sort of API reference documentation has proven itself incredibly helpful to the entire Swift community.

Additionally, conceptual documentation is often equally important, and complementary to API reference. Conceptual articles can provide higher-level instructions for using different APIs together to perform a more complex task. This high level view of a package is essential when introducing a newcomer to a package’s concepts and design patterns. Going further, step-by-step tutorials are a great way to directly help a new user become familiar with a package, by walking them through the code with examples, screenshots, and diagrams.

Finally, bringing documentation right into the developer workflow, integrated into the existing workflow, means more documentation will be written. So it will always be a high priority for Swift-DocC to integrate as seamlessly as possible into the way developers like to work.

How Swift-DocC works

Swift-DocC (GitHub) supports authoring three types of rich documentation — API reference, conceptual articles, and tutorials. Documentation is written using Markdown syntax directly in source code files, or in plain text files that live inside the same repository. Developers can then use whatever editor, IDE, or SCM tools they already prefer. By using the simple file and folder conventions of Swift-DocC, for instance a .docc folder along side your source, the docc compiler tool is able to automatically manage links, produce diagnostics, and build even large documentation libraries very quickly.

Swift-DocC has several powerful features:

  • Integrated with the Swift compiler to generate documentation using rich semantic information
  • Built to integrate tightly with tools to power rich authoring experiences including autocompletion and diagnostics
  • Documentation text that can directly link to other symbols across the doc library using double-backtick syntax
  • Validation engine to verify links, and report other content errors and warnings
  • Conceptual articles can be included within the repository with links to other API and article documents
  • Ability to override portions of the default generated documentation using extension files
  • Integrated tutorial engine using graphics and sample code, great for introducing packages to new users
  • Supports very large codebases of Swift code (with a near-term goal to support C and Objective-C)
  • Excellent build-time performance, appropriate for continuous integration

The Details

For folks that may want to contribute, or are just interested in the inner-workings of Swift-DocC, let’s dive a little deeper into how it all works.

Inputs

At its core, the Swift-DocC compiler accepts two kinds of inputs:

  1. A machine-readable representation of a Swift module’s API in the form of symbol graph files. Symbol graph files list all the APIs available in a module, including their documentation comments and relationship with one another. The format is designed to be programming language–agnostic, in that compilers for other languages can integrate with Swift-DocC by emitting symbol graph files. Symbol graph files are processed by the docc compiler to generate an internal graph of the documentation content. Swift 5.5 already emits symbol graph files as part of the compilation process.

  2. A folder of additional content called the Documentation Catalog, which can include:

    • Article files containing free-form Markdown describing a topic conceptually
    • Assets such as images, videos, and code files
    • Tutorial files containing structured Markdown to produce step-by-step tutorials
    • Documentation extension files, which contain additional documentation for a specific API of the documented module
    • An Info.plist file containing metadata such as the name of the documented module. This file is optional and the information it contains can be passed via the command line

These inputs are provided to docc via the command line:

docc convert \
    <path to .docc catalog> \
    --additional-symbol-graph-dir <path to folder of symbol graph files>

Syntax

Swift-DocC understands the GitHub Flavored Markdown dialect of Markdown and extends the syntax to support:

  • Symbol links, e.g., MySymbol , which provide a lightweight syntax to link to an API's documentation
  • Directives, e.g., @Tutorial { … } , which allow the writing of structured Markdown content. Directives are attributed markdown containers that provide a declarative way to author rich content like tutorials. Because the set of supported directives and their child content is known by the compiler, DocC can emit precise diagnostics and IDEs can provide rich structural autocompletion and folding facilities.

Markdown parsing is provided by the Swift Markdown library, which uses cmark-gfm internally.

To organize documentation into a hierarchical structure, Swift-DocC documentation declares a Topics section with a list of links to documentation organized in topic groups. This structure allows interweaving API reference documentation with conceptual documentation articles, as well as step-by-step tutorials.

## Topics

### Essentials

- <doc:GettingStarted>
- ``Sloth``

### Creating Sloths

- ``SlothGenerator``
- ``NameGenerator``

Outputs

With these inputs, Swift-DocC produces a folder in the Documentation Archive format (extension .doccarchive ), which contains a machine-readable output of the documentation that a renderer can use to produce rendered documentation.

Documentation Archives use the following structure:

<name>.doccarchive
├── data // JSON files for each documentation page.
│ └── documentation
│ ├── <module name>
│ │ ├── <top-level API>.json
│ │ │ └── <member API>.json
│ ┆ ┆── <article name>.json
├── index // An index of the documentation content.
├── metadata.json // Metadata about the produced documentation.
├── …Swift-DocC-Render template files…
┆

The content for each documentation page is written to files in the Render JSON format (see schema), which are emitted in the data/ folder of documentation archives. Render JSON files fully describe documentation pages and allow other documentation tools to integrate with Swift-DocC. Swift-DocC’s renderer, Swift-DocC-Render, is a web app that understands Render JSON and generates styled pages for the web.

Integration in Swift tools

A future proposal will describe how to implement Swift-DocC with Swift Package Manager to support the creation of rendered documentation through a single command-line invocation:

swift package build-documentation

This will be a great opportunity for the community to identify common workflows, such as generating documentation to deploy it to a web server, or generating documentation locally to learn about a package’s dependencies.

What’s Next

Today Swift-DocC starts its journey, and there is a lot more work to do, even in the initial feature set. The code will appear in nightly toolchains shortly, so give it a spin!

Among the first tasks to add to the project will be to integrate with SwiftPM, so the doc building experience is more seamless. Additionally, there is already a pretty exciting list of new features and other short-term improvements set for this project, including:

  • Support for Objective-C documentation
  • Enable hosting documentation in static file server environments (e.g., GitHub Pages)
  • Custom themes for the web renderer
  • Generate documentation from app code, including private or internal APIs

Of course, the community will surely have even more ideas. Let’s get started! As mentioned in the blog post, there are many ways to get involved:

59 Likes

Am I correct in my observation that Swift-DocC-Render uses Highlight.js for syntax highlighting?

Earlier this year, I made a lot of improvements to the Swift support for Highlight.js (Improved Swift support for Highlight.js). However, that only covered the features I needed at that time, so it's far from complete. If Swift-DocC-Render has made further improvements, can these be contributed back upstream?

Am I correct in my observation that Swift-DocC-Render uses Highlight.js for syntax highlighting?

Yes, DocC-Render uses Highlight.js for syntax highlighting.

Earlier this year, I made a lot of improvements to the Swift support for Highlight.js (Improved Swift support for Highlight.js ). However, that only covered the features I needed at that time, so it's far from complete. If Swift-DocC-Render has made further improvements, can these be contributed back upstream?

Any improvements to the syntax highlighting for Swift (or any other languages) should be contributed directly to the highlight.js project, and DocC-Render can bump the version of highlight.js that it depends on to bring in those changes.

2 Likes

This is very cool, congratulations!

I know that native concurrency has sort of stolen the show since WWDC, but personally, this is the thing I've most been looking forward to.

Writing documentation is really hard. For me, it's the slowest work that I do; it takes several revisions, clarifying things which could be unclear, removing parts which seem too verbose, adding examples which exercise the API but are still relatable, etc.

I definitely think that better tooling can help with that, especially when it comes to organisation and presentation. Docs need to be understandable and navigable even though they might contain a lot of nuanced information. There are so many details, from the order in which information is presented, or which other information is presented nearby, to small details like the way lines break in long generic function declarations or a code snippet, and they all have a really big impact on readability.

So far, I've found DocC to be very good. There are a few issues which prevent me from switching to it today (e.g. extensions on types from outside the module are not documented), and hosting from a static file server would be excellent, but I'm glad that it's open-source so I can either contribute improvements or use those improvements as soon as they land.

(Also, the documentation for DocC itself is extremely impressive. Those ASCII diagrams are next-level)

14 Likes

Installing into Xcode

You can test a locally built version of Swift-DocC in Xcode 13 or later by setting the DOCC_EXEC build setting to the path of your local docc :

  1. Select the project in the Project Navigator.
  2. In the Build Settings tab, click '+' and then 'Add User-Defined Setting'.
  3. Create a build setting DOCC_EXEC with the value set to /path/to/docc .

The next time you invoke a documentation build with the "Build Documentation" button in Xcode's Product menu, your custom docc will be used for the build. You can confirm that your custom docc is being used by opening the latest build log in Xcode's report navigator and expanding the "Compile documentation" step.

Is this only possible using a build setting in an Xcode project?

Plenty of Swift libraries do not maintain Xcode project files, and even SwiftPM is deprecating the --generate-xcodeproj feature, because Xcode can open Swift packages directly. Unfortunately, AFAIK there is no way to set custom build settings if you open your project that way.

I also tried setting DOCC_EXEC as an environment variable when starting Xcode, just in case that happened to work:

open --env DOCC_EXEC=<path to my built docc> /Applications/Xcode.app 

Alas, it doesn't seem to.

Sounds like package integration will be an upcoming proposal.

But I agree, it would be great to see the Xcode dependency removed so we can generate docs for all Swift platforms simultaneously. (On that topic, I wonder if it would be possible for DocC to detect when certain bits of functionality are behind platform compile guards and not just availability checks.)

5 Likes

Awesome to see this land now and very happy to see SPM and static page serving on the near-term roadmap, will make documentation life a lot easier!

6 Likes

Would you want to integrate something like docc2html (the static site generator) or servedocc? (presumably there is no reason for DocCArchive anymore?)

What about things like the GitHub action?

Congratulations on launching this! I'm excited to start using it, but also the Markdown bindings, in some of my projects.

It looks like the gfm branch of apple/swift-cmark contains a handful of implementation changes and API additions compared to the main github/cmark-gfm repo. Do you have plans to upstream those back to github/cmark-gfm soon? This would be very helpful for projects that already have a dependency on cmark-gfm and don't want to or can't take on a second fork.

These are all great topics for future features and enhancements. We setup a Swift-DocC forum topic for the community to discuss and prioritize ideas like this. Swift-DocC - Swift Forums

May be good to spin up specific ideas and proposals there. Thanks!

2 Likes

For some of our changes, we've in fact already proposed them upstream to github/cmark-gfm ! For others, we would love to work with the cmark-gfm maintainers and community to build something that works for everyone.

2 Likes

I was very excited when Xcode announced DocC, and then dismayed to learn it only works on library targets, not application targets. Why this limitation? It seems arbitrary. There are many reasons to want to document the code in an application/executable target as well.

Does Swift-DocC have the same limitation?

2 Likes

And, if it does, can we get a Pitch to extend it to application targets?

1 Like

The initial release of DocC in Xcode and Swift-DocC focused on documentation of public symbols of a framework or library. To support application targets we would also want to support documenting internal symbols which adds some interesting complexities, for example controlling what documentation is exported to a framework’s end users.

The announcement post lists “Generate documentation from app code, including private or internal API” as one of our first tasks for new features and improvements. You can expect a future pitch about this.

10 Likes