Support hosting DocC archives in static hosting environments

philipturner · December 5, 2021, 8:35pm

Today, I'm making some changes to ARHeadsetUtil to polish it up before using it in MetalFFT. I have to synchronize changes with ARHeadsetKit, which is not a very productive workflow.

ethankusters · December 9, 2021, 11:14pm

Thank you for all of the great feedback here! The proposed implementation has been merged to Swift-DocC's main branch.

https://github.com/apple/swift-docc/commit/4ee8b3a571185a038a553ad025752fbbf05d0bc3

Joseph_Heck · December 10, 2021, 12:22am

Am I correct in thinking that "merged to main" means it'll be out with the next successful swift-nightly toolchain release? Or am I being too optimistic (I wasn't sure if there was another branch delay in there or such...)

ethankusters · December 10, 2021, 12:52am

You are correct! Now that Swift-DocC is part of the Swift toolchain, the repository's main branch is pulled every night for the nightly toolchain builds. So I would expect this to be included in the next successful build.

philipturner · December 10, 2021, 2:25am

Does this mean, for example, that I can compile ARHeadsetKit’s DocC catalog into HTML code tomorrow and host it on GitHub pages?

gwendal.roue · December 10, 2021, 7:49am

Full support for Github hosting would be convenient for so many reasons, but there is one more. Granting the right to publish an update to the doc is as easy as granting the right to modify the code: just grant a push access to the repository that contains the generated documentation. This will be really helpful for teams.

I hope that Github hosting is planned to enter the end-to-end test suite of DocC.

Feature requests (just in case those use cases were not properly identified yet):

Ability to host several versions of the doc (so that users of an older version of a lib can still access the documentation for this version): http://author.github.com/repo/docs/1.0, http://author.github.com/repo/docs/2.1, etc.
The final lovely touch: document a way for the old versions of the doc to warn they are outdated, and to provide a link to the latest version, without regeneration. This would be really nice because as long as old versions of the doc are online, one can never tell which one Google will put in the results. Besides, immutable links to old versions will exist online (in online comments, forums posts, stack overflow answers, etc.). Telling the user he's looking at outdated documentation would be a nice touch. I agree that a well-designed robots.txt can help pruning old docs from the search engine indexes, but immutable links remain.

Max_Desiatov · December 10, 2021, 8:13am

Huge +1 from me for supporting multiple versions. This is crucial for hosting documentation for any publicly available library. Although it is orthogonal to static hosting, I couldn't resist voicing my support for it here since you mentioned it

philipturner · December 10, 2021, 10:45am

I hope that this is successful. I would like to have that process go easily, as I’m working on a different project right now and don’t want to spend a lot of time manually recompiling the ARHeadsetKit documentation.

ethankusters · December 11, 2021, 12:57am

You'll be able to use the next Swift.org nightly toolchain to produce DocC archives that are compatible with GitHub Pages. I’ll post in this thread with detailed instructions on how to use the new functionality when the first toolchain that includes these changes is released.

If you'd like to get a jump-start on things, you can pull Swift-DocC's main branch and use the latest version of docc to transform an existing DocC archive to make it compatible with GitHub pages. For example:

git clone https://github.com/apple/swift-docc.git
cd swift-docc
swift run docc transform-for-static-hosting <path-to-your-doccarchive> --hosting-base-path <github-repo-name>

This will transform an existing DocC archive to make it compatible with GitHub pages.

Again, once functionality this is available in a nightly toolchain, I’ll post instructions on how to create a new DocC archive that can be hosted on GitHub pages.

Philip_Turner · December 11, 2021, 1:32am

I’m thinking of using the new DocC to publish online documentation for my MetalFFT (GitHub - philipturner/metal-fft: 1D, 2D, and 3D variations of Fast Fourier Transforms) framework instead this weekend.

ethankusters · December 11, 2021, 1:32am

@gwendal.roue I agree! If there's anything you'd like to see beyond what is described in the original forums post that you would consider necessary for "Full support" of GitHub hosting, please let me know. This is a really important feature for Swift-DocC to get right.

Regarding your feature requests:

Proper versioning support in Swift-DocC is definitely something we'd like to tackle in the future but you may be able to achieve something like what you're describing with what's supported today.

I haven't tried this- but I think this should be possible today with the --hosting-base-path flag described in the original post. I can imagine configuring your CI to pass a different base path depending on the current branch. So maybe:

--hosting-base-path my-repo-name/main
--hosting-base-path my-repo-name/v1.0
--hosting-base-path my-repo-name/v1.5

And then storing the output in various root folders also depending on that branch like:

main
├ data
│  │ example.json
│  └ ...
├ documentation
│  ├ example
│  │   └ index.html
│  └ ...
├ js
├ css
└ ...
v1.0
├ data
│  └ ...
├ documentation
│  └ ...
└ ...
v1.5
└ ...

GitHub Pages would then make your documentation available at various base urls like:

This isn't a great user experience and there's a lot we can do to make this process easier when we address versioning more directly (and we're tracking some feedback like [SR-15541] Support hosting of multiple DocC archive(s) (versions) · Issue #204 · apple/swift-docc · GitHub on this) but I'm excited about the possibilities already.

I agree this would be amazing and is exactly what I've been thinking of as "proper" versioning support in Swift-DocC. I don't believe we're currently tracking this as a feature request- if you'd like to post some details on how you'd like this to work on bugs.swift.org and start a dedicated thread about it, I'd love to chat more.

gwendal.roue · December 11, 2021, 10:15am

Thank you very much @ethankusters. I'm very glad the team behind DocC shows such focus on user experience, not only UX for doc readers, but UX for doc producers as well. This level of empathy is quite refreshing :-)

rauhul · December 12, 2021, 1:10am

It would be awesome if multiple versions hosted together allowed users to see API diffs like the docs on developer.apple.com, example: Apple Developer Documentation

philipturner · February 1, 2022, 1:45pm

I look forward to using this to replace Google's archived documentation for Swift for TensorFlow.

brianmichel · February 14, 2022, 3:31pm

Does anyone here have an example of this working on GitHub Pages? Every time I end up transforming a .doccarchive and hosting the result with a simple Python web server, I seem to have to go a few levels deep to actually see the generated documentation. Meaning if I generate documentation with the following command (assuming I'm generating multiple sets of documentation for all of our frameworks):

swift run docc process-archive transform-for-static-hosting CoolCode.doccarchive \ 
--output /path/to/output \
--hosting-base-path CoolCode

I end up having to browse to the path CoolCode/documentation/coolcode before I see anything rendering. I would assume if I generated multiple transformed archives into the same folder I could serve that folder and navigate to /CoolCode to see the documentation at the root level.

I was trying to follow the example from Support hosting DocC archives in static hosting environments - #27 by ethankusters to figure out a solution to host generated documentation for my team's many frameworks, but this doesn't seem to completely do it. Any thoughts?

(N.B. I'm running the latest sha's from both swift-docc and swift-docc-render to generate my examples)

ethankusters · February 14, 2022, 5:28pm

Hi @brianmichel!

Sorry to hear you're having trouble here.

Here are a few examples of Swift-DocC docs up on GitHub Pages today:

This is the expected behavior today. Because you passed --hosting-base-path CoolCode, the generated site expects to be hosted at the /CoolCode base path. This functionality is provided to support hosts like GitHub pages, where your site is essentially name-spaced by the name of your repo.

By default, we expect Swift-DocC docs on GitHub pages to be available at something like:

https://<username>.github.io/<repository-name>/documentation/<target-name>

Since you're using a simple Python HTTP server, you can skip the --hosting-base-path option, and access your docs directly at the root of your site (/documentation/coolcode) if you'd like.

There's a little more information around GitHub Pages hosting in the Swift-DocC Plugin's documentation here. It is written specific to the plugin's usecase but should still be broadly applicable.

brianmichel · February 14, 2022, 5:41pm

Much appreciated Ethan, I figured I was doing something wrong. I'll give this a try!

gwendal.roue · February 14, 2022, 5:43pm

I didn't know it was possible today!

Joseph_Heck · February 14, 2022, 10:36pm

@brianmichel - Let me toss out three more:

I've been splatting a script into a docs-specific branch and rendering into the docs directory since that's a default that Github Pages supports easily.

One version uses the new Docc Plugin:

github.com

heckj/MeshGenerator/blob/main/docbuild.bash

#!/bin/bash

echo "Make sure you've rebased over the current HEAD branch:"
echo "git rebase -i origin/main docs"

set -e  # exit on a non-zero return code from a command
set -x  # print a trace of commands as they execute

# rm -rf .build
# mkdir -p .build/symbol-graphs

# $(xcrun --find swift) build --target MeshGenerator \
#     -Xswiftc -emit-symbol-graph \
#     -Xswiftc -emit-symbol-graph-dir -Xswiftc .build/symbol-graphs

# Enables deterministic output 
# - useful when you're committing the results to host on github pages
export DOCC_JSON_PRETTYPRINT=YES

# $(xcrun --find docc) convert Sources/MeshGenerator/MeshGenerator.docc \

This file has been truncated. show original

Another uses classic Docc convert:

github.com

nicklockwood/Euclid/blob/feature/documentation/docbuild.bash

#!/bin/bash

echo "Make sure you've rebased over the current HEAD branch:"
echo "git rebase -i origin/master docs"

set -e
set -x

rm -rf .build
mkdir -p .build/symbol-graphs

$(xcrun --find swift) build --target Euclid \
-Xswiftc -emit-symbol-graph \
-Xswiftc -emit-symbol-graph-dir -Xswiftc .build/symbol-graphs

# Enables deterministic output
# - useful when you're committing the results to host on github pages
export DOCC_JSON_PRETTYPRINT=YES

$(xcrun --find docc) convert Euclid.docc \

This file has been truncated. show original

With the pattern I've started using being to keep a docs branch "rebased" over the top of whatever current dev or release branch I'm documenting. I've written up a walk-through on my personal blog if you're so inclined.

yo1995 · September 14, 2022, 6:37pm

To my understanding, a "--hosting-base-path" must be specified if the static files are hosted in a subfolder, is that correct?

We generate the static files on a build server and eventually copy it to the release server to a different sub path.
It would be good if we can preview the generated files directly on the build server, but it seems that would require specifying different --hosting-base-path's for build and release.

Is it possible to just arrange the generated static files in its hierarchy, without depending on a relative base path? Thanks.