Support hosting DocC archives in static hosting environments

Hi all!

This post discusses an enhancement to Swift-DocC and Swift-DocC-Render that will allow developers to build DocC archives that can be hosted without custom routing. This is specifically designed to enable DocC to be used in additional static hosting environments, most notably GitHub Pages.

This change is meant as a quick solution to address a pressing need, and provides general goodness. But please know that we’ve heard the community’s feedback that they would love for Swift-DocC to directly emit static HTML, and this feature is high on the priority list.

The proposed implementation put together by Steve Scott (Scotty) and Dobromir Hristov is available here:

Proposed Solution

To produce a new DocC archive that does not require custom routing rules for hosting on a simple Python HTTP server, you run the following:

docc convert MyFramework.docc --transform-for-static-hosting \
    --output-path MyFramework.doccarchive

# Start a local Python server
cd MyFramework.doccarchive
python3 -m http.server

You can also now produce a DocC archive for hosting on the GitHub Pages environment at your my-framework-repo repository.

Assuming, you’ve configured the GitHub Pages site at my-username.github.io/my-framework-repo to render the documentation from the docs directory at the root of the repository, you would run the following from the repository root to publish documentation:

docc convert MyFramework.docc --transform-for-static-hosting \
    --static-hosting-base-path /my-framework-repo \
    --output-path docs

git add docs
git commit -m "Update documentation."
git push

Background

A DocC archive is a static, single-page web application. As a web app, it contains just a single html file and requires some configuration on the server in order to route all incoming requests to this html file.

For example, when trying to access the “Formatting Your Documentation Content Page” in DocC’s documentation on Swift.org, the server accepts a request at the documentation/docc/formatting-your-documentation-content path and routes it the base index.html file at the root of DocC’s own archive.

For more information on the custom routing rules required to host a DocC archive on a server, please see the documentation on Swift.org.

However, if a you are hosting a DocC archive on a server where you don’t have the ability to customize routing rules, the request to documentation/docc/formatting-your-documentation-content would fail because there is no file at that path.

Solution

The proposed solution will copy the index.html file at the root of a DocC archive to all of the documentation paths in the given archive. This removes the requirement for custom routing rules but, in all other ways, Swift-DocC-Render will continue to behave the same way.

For example, when transformed for a static hosting environment, DocC’s own archive would look something like this:

DocC.doccarchive
├ data
│  │ formatting-your-documentation-content.json
│  │ distributing-documentation-to-other-developers.json
│  └ ...
├ documentation
│  ├ formatting-your-documentation-content
│  │   └ index.html
│  ├ distributing-documentation-to-other-developers
│  │   └ index.html
│  └ ...
├ js
├ css
└ ...

Because the DocC archive now contains an index.html file at all documentation paths, the archive can be hosted without custom routing rules. This enables out-of-the-box support for simple file servers like Python’s HTTP server (python3 -m http.server), and hosts that don’t expose custom routing, like GitHub Pages.

This a less-than-optimal solution because of the duplication of the index.html file that is required. However, we feel that as we work towards emitting static HTML directly, this duplication is acceptable in the meantime as it supports a simpler hosting solution.

By default, Swift-DocC will continue emitting a single index.html file for space efficiency in server environments that support custom routing. To get the new GitHub Pages-compatible behavior you will need to pass the --transform-for-static-hosting flag, for now.

Base Path Configuration

This solution addresses the needs of developers who are hosting their documentation content at the root of their website (i.e. www.my-website.com/documentation/my-framework). However, many developers may wish to host their documentation at a specific sub-path on their website. This is also a requirement for many hosting their documentation on GitHub Pages where developers will likely be hosting their documentation at my-username.github.io/my-repository/documentation/my-framework. To support these scenarios, Swift-DocC-Render will add support for a configurable BASE_PATH argument. This path will be configurable by Swift-DocC when producing a DocC archive for static hosting environments.

Implementation

The docc convert command will add an additional flag and option:

  • --transform-for-static-hosting: Produce a DocC archive that supports a static hosting environment.
  • --static-hosting-base-path <static-hosting-base-path>: The base path your documentation website will be hosted at.
    • For example, to deploy your site to example.com/my_name/my_project/documentation instead of example.com/documentation, pass /my_name/my_project as the base path.

When the --transform-for-static-hosting flag is passed to a docc convert invocation, the produced DocC archive will include an index.html file at all documentation paths, making it compatible with static hosting environments.

In order to support base path customization, Swift-DocC-Render will vend a new file named template-index.html. This file will include {{BASE_PATH}} template placeholder tags that Swift-DocC will be able to replace with a user-provided path.

When a static hosting base path is provided to docc convert via the --static-hosting-base-path option, docc will replace all instances of {{BASE_PATH}} in the new template-index.html file vended by Swift-DocC-Render. Then it will copy this customized version of the index.html file to all documentation paths. The produced archive will be compatible with static hosting environments at the given base path.

In addition, a new command will be added to docc that allows for updating an existing DocC archive to make it compatible with a static hosting environment.

The usage for this command will be as follows:

OVERVIEW: Transform an existing DocC Archive into one that supports a static hosting environment.

USAGE: docc process-archive transform-for-static-hosting <source-archive-path> [--output-path <output-path>] [--static-hosting-base-path <static-hosting-base-path>]

ARGUMENTS:
  <source-archive-path>   `Path to the DocC Archive that should be processed.`

OPTIONS:
  --output-path <output-path>
                          The location where docc writes the transformed archive.
        If no output-path is provided, docc will perform an in-place transformation of the provided DocC Archive.
  --static-hosting-base-path <static-hosting-base-path>
                          The base path your documentation website will be hosted at.
        For example, to deploy your site to 'example.com/my_name/my_project/documentation' instead of 'example.com/documentation', pass '/my_name/my_project' as the base path.
  -h, --help              Show help information.

Alternatives Considered

We considered using Node.js to pre-render the content in the DocC archive out to static HTML. This format would inherently remove the requirement for custom routing rules. However, because this would add a run-time dependency on Node and have performance impacts on producing a compatible DocC archive, we decided to not take this route. When we do add support for static HTML in Swift-DocC, we’ll want to enhance the documentation compiler to emit HTML directly.

37 Likes

Maybe this idea is too hacky, but I've gotten my doc archives hosted on GHP by making the index.html the 404 page and updating the paths in the 404 page with the "static-hosting-base-path" you mention here. (I also had to update some generated JS, but I assume your --static-hosting-base-path flag would take care of this).

Yes, your assumption is correct. The —static-hosting-base-path flag should take care of this without needing to edit any generated JS in DocC-Render by hand. :)

While the described 404 page solution would be an alternative that solves this in a more minimal way with just the one file, one potential downside is that the site would lose the ability to properly distinguish real 404 and 200 responses based on the URL.

2 Likes

Very happy to see this pitched, and the approach looks good!

Thanks for not forcing developers to install node to generate swift docs :wink: That would have been very annoying :grinning_face_with_smiling_eyes:

I guess my only piece of additional feedback is that the option is very long :thinking: Have you considered just --static or --static-site (and --static-site-base-path) or something shorter? Not sure the "for" and "hosting" words add much context here. I might as well just want to generate the files to browse locally (and not have to install nodejs).


Question: What are the plans for cross-repo linking? Say Swift NIO hosts their docs on their repo, but I have my distributed actors library (and docs) in another repo, but want to link to NIO symbols. We should be able to construct well known links based on adding some "package NIO has base url https://apple.github.io/swift-nio/docs/current/<docc path to symbol>). This could totally be a follow up proposal, but is something I think we should address eventually.

12 Likes

@ktoso Very happy to hear that!

This is good feedback. Thank you! We chose --transform-for-static-hosting over something more simple to avoid any confusion that this is related to purely static HTML. I think my assumption seeing a --static-site flag would be that the DocC is emitting HTML directly which is not the case yet.

But the verbosity and command-line ergonomics of the proposed flag are a drawback and we're definitely open to feedback here.

We could also consider adding a second form for the flag. So we would still provide --transform-for-static-hosting for CI use cases where the clarity is more important than the typing ergonomics. But then also allow for --static-hosting (or something similar) for the command-line use case.

Great question! Currently the way we've been thinking about this is that DocC should allow for link resolution against other pre-built DocC archives. This is something I know @ronnqvist has been looking into and definitely something I'd be interested in discussing further in a separate thread. The point about providing a known base URL for a given repo is a really good one.

EDIT: We actually have an existing thread about cross-framework linking here: Linking between frameworks - #3 by Karl.

3 Likes

Thanks for the links. There does not seem to be a specific bug/radar for this yet so maybe I'll shoot one and make a separate thread on the forums to discuss it a bit. Other than the static hosting (this work), that'd be the only missing/blocking feature before we can move all server libraries to using docc, thanks!

2 Likes

Hmm... I see now, it's a bit hidden in the proposal text to be honest that this isn't just static html. I just re-read the text again and don't actually see it spelled out what exactly this will produce. Isn't it "static html" in your definition because there's still some JS libs involved or what is the difference between this proposal and the alluded to "static html"?

Perhaps this needs to become some form of --output-format=something instead then, which would allow doing that static-html in the future? Would the output today be e.g. --output-format=html and the "future" one you are alluding to html-nojs..?

The "static hosting" terminology is just something I find very weird somehow, having worked with many other documentation engines (rst, antora & asciidoctor, godoc, javadoc, paradox, etc.) It's all really just about some output format, and options to configure some base paths, but those are shared really between different formats even perhaps -- a base url is typical for all kinds of outputs, I don't think we need to associate it with "static hosting".

Not sure if this helps, thanks for being open to feedback!

3 Likes

This is pretty much the process that I use already for Vexil so it would be great to see it properly supported, and it works well for just shoving a bunch of doccarchives into a single bundle.

Some issues I have noticed doing this is with handling images and tutorials. Would they be equally supported under this proposal?

2 Likes

Exiting to see this moving forward! Making hosting easier will probably increase DocC adoption significantly.

One question though: instead of copying the index.html to all directories, was it also considered to use a url query or fragment (aka slug, anchor or #) instead of a path? Servers usually ignore the query for static files or can be at least configured to do so. A fragment is guaranteed to not be send to the server and we could read it locally on the client with some JavaScript.

1 Like

@ktoso Sorry to hear that- I definitely didn't intend for this to be misleading.

This work is designed to help support common hosting scenarios, like GitHub Pages, where Swift-DocC documentation hosting wasn't possible. It is explicitly not a rewrite of Swift-DocC's renderer. Swift-DocC-Render is a single page web application and continues to be even with this new hosting support.

This produces a DocC archive in the same format that is currently emitted. The only change is that we now duplicate the base index.html file that powers the Swift-DocC-Render SPA to all relevant file paths so that when a user navigates to a given path, we don't need custom routing to direct them to the single index.html at the root. You can see a copy of the current index.html here: Swift-DocC-Render-Artifact:index.html. That is the file that will be copied around.

This is currently described in the original post here but I agree that it could be more clear. There's some documentation on how DocC archives work on Swift.org here.

I wouldn't consider this "static HTML" because the actual page content isn't included in the HTML file. Regardless of how much JavaScript is used, I think when looking at a "static HTML" website, people would expect to be able to inspect a given HTML file and find the prose content for the page within.

This is not the case for Swift-DocC-Render today or with the static-hosting support described here. From Swift-DocC-Render's README:

Swift-DocC-Render is a web Single Page Application (SPA) powered by Vue.js for creating rich code documentation websites. Pages and content are generated using render JSON data from DocC. SPAs are web apps that render dynamically at runtime entirely in the browser, using JavaScript.

I think this is a good idea that we should explore when we do add support for emitting HTML directly. However, today, Swift-DocC only supports a single output format: RenderJSON. That JSON is then dynamically rendered by the Swift-DocC-Render SPA. So in the future, I could imagine us having both --output-format=json and --output-format=html.

I agree that "static hosting" terminology is unusual. But I think what we're doing here is a little unusual. We're not adding support for a new output format, we're transforming an existing one to enable new hosting scenarios.

This is a great point. I spoke with @marcus_ortiz and he agrees that the base-path option we're adding should not be specific to the static hosting transformation we're doing here. For example, there are users of Swift-DocC that would benefit from being able to customize the base path of Swift-DocC-Render but don't need the --transform-for-static-hosting support.

I think updating this flag from --static-hosting-base-path to just --hosting-base-path makes sense.

This definitely helps and is exactly the kind of feedback we're looking for. Thank you so much!

2 Likes

@bok1 They would be! Everything Swift-DocC-Render supports today will be supported here- just without the need for custom routing rules.

I think this is an interesting alternative solution where we could potentially use a URL system with paths appended as a fragment so that there truly only ever is one real HTML file needed.

Unfortunately, I think there are some downsides to this approach—the main one being that we would lose our current system where we have true URLs right now and don’t need to append the hash fragment for every single page since we would be bypassing the History web APIs. If we were to go with this approach, you may have to access pages like /documentation/example instead of /documentation/example.

I think this approach might also run into the issue I touched on earlier in the thread with giving the ability to serve proper 200 and 404 responses based on the URL path.

Also, the logic for linking to other pages could be impacted, and it might conflict with how we already support URL fragments in our current URL system (like /documentation/example#overview as an example) for same page linking.

1 Like

Thanks for the clarifications, that all makes much more sense now, thanks!

Thanks for the reminder, now I see why this isn't "static html".

I definitely hope to get "static html" at some point in the future -- it is nicer on search results (google and friends), than single page apps from my experience.

Right, I think that makes sense -- if/when we'd do real "static html" that's the time when such option could be introduced then.

That sounds good, ok :+1:

Thanks for following up here, I think this is heading into a good direction :+1:

2 Likes

This feature should be high on the priority list. Right now, not being able to host documentation online forced me to maintain two duplicates of the code in ARHeadsetUtil. I also had to give extensive instructions to users of ARHeadsetKit to get documentation to compile properly in Xcode, and having it already exist on GitHub pages would make much more people inclined to try (or at least skim) the tutorials. I took the entire SwiftUI course on Apple's website, not in the Xcode documentation viewer, and I suspect that other people feel the same way.

This would also allow the replacement of Google's Colab notebooks in the resurrection/successor to Swift for TensorFlow: Swift for TensorFlow Resurrection: Differentiation running on iOS

To be clear, you can host the documentation online, it's just in a (IMO) quirky format that requires you to also support specific pathing redirects in whatever you're using to host it.

like Konrad, I'd love to see a future variation that has an end result of flat, static HTML being generated in a directory - which is far more akin to what other tools provide (sphinx, jazzy, asciidoc, hugo, etc).

I can see some benefits to have the core information about what's in the docs in an intermediate format that can be re-used to multiple purposes (in-editor doc lookups being the obvious use case where it's exposed today). That said, I was admittedly a surprised when I first saw the internals that the end result of Docc is something that requires a javascript SPA (the docc-render app) to render the documentation.

3 Likes

I did a lot of digging trying to get my DocC documentation to work online, and I saw a workaround that made HTML pages. However, it was buggy and couldn't be automated to the extent I wanted.

Also, being able to link to other packages' documentation (as Apple does on their own website) would be a major benefit, as listed in the ARHeadsetUtil overview.

1 Like

Today, I'm making some changes to ARHeadsetUtil to polish it up before using it in MetalFFT. I have to synchronize changes with ARHeadsetKit, which is not a very productive workflow.

Thank you for all of the great feedback here! The proposed implementation has been merged to Swift-DocC's main branch. :partying_face:

https://github.com/apple/swift-docc/commit/4ee8b3a571185a038a553ad025752fbbf05d0bc3

6 Likes

Am I correct in thinking that "merged to main" means it'll be out with the next successful swift-nightly toolchain release? Or am I being too optimistic (I wasn't sure if there was another branch delay in there or such...)

You are correct! Now that Swift-DocC is part of the Swift toolchain, the repository's main branch is pulled every night for the nightly toolchain builds. So I would expect this to be included in the next successful build.

3 Likes