Support hosting DocC archives in static hosting environments

Exiting to see this moving forward! Making hosting easier will probably increase DocC adoption significantly.

One question though: instead of copying the index.html to all directories, was it also considered to use a url query or fragment (aka slug, anchor or #) instead of a path? Servers usually ignore the query for static files or can be at least configured to do so. A fragment is guaranteed to not be send to the server and we could read it locally on the client with some JavaScript.

1 Like

@ktoso Sorry to hear that- I definitely didn't intend for this to be misleading.

This work is designed to help support common hosting scenarios, like GitHub Pages, where Swift-DocC documentation hosting wasn't possible. It is explicitly not a rewrite of Swift-DocC's renderer. Swift-DocC-Render is a single page web application and continues to be even with this new hosting support.

This produces a DocC archive in the same format that is currently emitted. The only change is that we now duplicate the base index.html file that powers the Swift-DocC-Render SPA to all relevant file paths so that when a user navigates to a given path, we don't need custom routing to direct them to the single index.html at the root. You can see a copy of the current index.html here: Swift-DocC-Render-Artifact:index.html. That is the file that will be copied around.

This is currently described in the original post here but I agree that it could be more clear. There's some documentation on how DocC archives work on Swift.org here.

I wouldn't consider this "static HTML" because the actual page content isn't included in the HTML file. Regardless of how much JavaScript is used, I think when looking at a "static HTML" website, people would expect to be able to inspect a given HTML file and find the prose content for the page within.

This is not the case for Swift-DocC-Render today or with the static-hosting support described here. From Swift-DocC-Render's README:

Swift-DocC-Render is a web Single Page Application (SPA) powered by Vue.js for creating rich code documentation websites. Pages and content are generated using render JSON data from DocC. SPAs are web apps that render dynamically at runtime entirely in the browser, using JavaScript.

I think this is a good idea that we should explore when we do add support for emitting HTML directly. However, today, Swift-DocC only supports a single output format: RenderJSON. That JSON is then dynamically rendered by the Swift-DocC-Render SPA. So in the future, I could imagine us having both --output-format=json and --output-format=html.

I agree that "static hosting" terminology is unusual. But I think what we're doing here is a little unusual. We're not adding support for a new output format, we're transforming an existing one to enable new hosting scenarios.

This is a great point. I spoke with @marcus_ortiz and he agrees that the base-path option we're adding should not be specific to the static hosting transformation we're doing here. For example, there are users of Swift-DocC that would benefit from being able to customize the base path of Swift-DocC-Render but don't need the --transform-for-static-hosting support.

I think updating this flag from --static-hosting-base-path to just --hosting-base-path makes sense.

This definitely helps and is exactly the kind of feedback we're looking for. Thank you so much!

2 Likes

@bok1 They would be! Everything Swift-DocC-Render supports today will be supported here- just without the need for custom routing rules.

I think this is an interesting alternative solution where we could potentially use a URL system with paths appended as a fragment so that there truly only ever is one real HTML file needed.

Unfortunately, I think there are some downsides to this approach—the main one being that we would lose our current system where we have true URLs right now and don’t need to append the hash fragment for every single page since we would be bypassing the History web APIs. If we were to go with this approach, you may have to access pages like /#documentation/example instead of /documentation/example.

I think this approach might also run into the issue I touched on earlier in the thread with giving the ability to serve proper 200 and 404 responses based on the URL path.

Also, the logic for linking to other pages could be impacted, and it might conflict with how we already support URL fragments in our current URL system (like /documentation/example#overview as an example) for same page linking.

1 Like

Thanks for the clarifications, that all makes much more sense now, thanks!

Thanks for the reminder, now I see why this isn't "static html".

I definitely hope to get "static html" at some point in the future -- it is nicer on search results (google and friends), than single page apps from my experience.

Right, I think that makes sense -- if/when we'd do real "static html" that's the time when such option could be introduced then.

That sounds good, ok :+1:

Thanks for following up here, I think this is heading into a good direction :+1:

2 Likes

This feature should be high on the priority list. Right now, not being able to host documentation online forced me to maintain two duplicates of the code in ARHeadsetUtil. I also had to give extensive instructions to users of ARHeadsetKit to get documentation to compile properly in Xcode, and having it already exist on GitHub pages would make much more people inclined to try (or at least skim) the tutorials. I took the entire SwiftUI course on Apple's website, not in the Xcode documentation viewer, and I suspect that other people feel the same way.

This would also allow the replacement of Google's Colab notebooks in the resurrection/successor to Swift for TensorFlow: Swift for TensorFlow Resurrection: Differentiation running on iOS

To be clear, you can host the documentation online, it's just in a (IMO) quirky format that requires you to also support specific pathing redirects in whatever you're using to host it.

like Konrad, I'd love to see a future variation that has an end result of flat, static HTML being generated in a directory - which is far more akin to what other tools provide (sphinx, jazzy, asciidoc, hugo, etc).

I can see some benefits to have the core information about what's in the docs in an intermediate format that can be re-used to multiple purposes (in-editor doc lookups being the obvious use case where it's exposed today). That said, I was admittedly a surprised when I first saw the internals that the end result of Docc is something that requires a javascript SPA (the docc-render app) to render the documentation.

2 Likes

I did a lot of digging trying to get my DocC documentation to work online, and I saw a workaround that made HTML pages. However, it was buggy and couldn't be automated to the extent I wanted.

Also, being able to link to other packages' documentation (as Apple does on their own website) would be a major benefit, as listed in the ARHeadsetUtil overview.

1 Like

Today, I'm making some changes to ARHeadsetUtil to polish it up before using it in MetalFFT. I have to synchronize changes with ARHeadsetKit, which is not a very productive workflow.

Thank you for all of the great feedback here! The proposed implementation has been merged to Swift-DocC's main branch. :partying_face:

6 Likes

Am I correct in thinking that "merged to main" means it'll be out with the next successful swift-nightly toolchain release? Or am I being too optimistic (I wasn't sure if there was another branch delay in there or such...)

You are correct! Now that Swift-DocC is part of the Swift toolchain, the repository's main branch is pulled every night for the nightly toolchain builds. So I would expect this to be included in the next successful build.

3 Likes

Does this mean, for example, that I can compile ARHeadsetKit’s DocC catalog into HTML code tomorrow and host it on GitHub pages?

1 Like

Full support for Github hosting would be convenient for so many reasons, but there is one more. Granting the right to publish an update to the doc is as easy as granting the right to modify the code: just grant a push access to the repository that contains the generated documentation. This will be really helpful for teams.

I hope that Github hosting is planned to enter the end-to-end test suite of DocC.


Feature requests (just in case those use cases were not properly identified yet):

  • Ability to host several versions of the doc (so that users of an older version of a lib can still access the documentation for this version): http://author.github.com/repo/docs/1.0, http://author.github.com/repo/docs/2.1, etc.
  • The final lovely touch: document a way for the old versions of the doc to warn they are outdated, and to provide a link to the latest version, without regeneration. This would be really nice because as long as old versions of the doc are online, one can never tell which one Google will put in the results. Besides, immutable links to old versions will exist online (in online comments, forums posts, stack overflow answers, etc.). Telling the user he's looking at outdated documentation would be a nice touch. I agree that a well-designed robots.txt can help pruning old docs from the search engine indexes, but immutable links remain.
10 Likes

Huge +1 from me for supporting multiple versions. This is crucial for hosting documentation for any publicly available library. Although it is orthogonal to static hosting, I couldn't resist voicing my support for it here since you mentioned it :see_no_evil:

4 Likes

I hope that this is successful. I would like to have that process go easily, as I’m working on a different project right now and don’t want to spend a lot of time manually recompiling the ARHeadsetKit documentation.

You'll be able to use the next Swift.org nightly toolchain to produce DocC archives that are compatible with GitHub Pages. I’ll post in this thread with detailed instructions on how to use the new functionality when the first toolchain that includes these changes is released.

If you'd like to get a jump-start on things, you can pull Swift-DocC's main branch and use the latest version of docc to transform an existing DocC archive to make it compatible with GitHub pages. For example:

git clone https://github.com/apple/swift-docc.git
cd swift-docc
swift run docc transform-for-static-hosting <path-to-your-doccarchive> --hosting-base-path <github-repo-name>

This will transform an existing DocC archive to make it compatible with GitHub pages.

Again, once functionality this is available in a nightly toolchain, I’ll post instructions on how to create a new DocC archive that can be hosted on GitHub pages.

4 Likes

I’m thinking of using the new DocC to publish online documentation for my MetalFFT (GitHub - philipturner/MetalFFT: 1D, 2D, and 3D variations of Fast Fourier Transforms for a Metal S4TF backend) framework instead this weekend.

@gwendal.roue I agree! If there's anything you'd like to see beyond what is described in the original forums post that you would consider necessary for "Full support" of GitHub hosting, please let me know. This is a really important feature for Swift-DocC to get right.

Regarding your feature requests:

Proper versioning support in Swift-DocC is definitely something we'd like to tackle in the future but you may be able to achieve something like what you're describing with what's supported today.

I haven't tried this- but I think this should be possible today with the --hosting-base-path flag described in the original post. I can imagine configuring your CI to pass a different base path depending on the current branch. So maybe:

  • --hosting-base-path my-repo-name/main
  • --hosting-base-path my-repo-name/v1.0
  • --hosting-base-path my-repo-name/v1.5

And then storing the output in various root folders also depending on that branch like:

main
├ data
│  │ example.json
│  └ ...
├ documentation
│  ├ example
│  │   └ index.html
│  └ ...
├ js
├ css
└ ...
v1.0
├ data
│  └ ...
├ documentation
│  └ ...
└ ...
v1.5
└ ...

GitHub Pages would then make your documentation available at various base urls like:

This isn't a great user experience and there's a lot we can do to make this process easier when we address versioning more directly (and we're tracking some feedback like [SR-15541] Support hosting of multiple DocC archive(s) (versions) - Swift on this) but I'm excited about the possibilities already.

I agree this would be amazing and is exactly what I've been thinking of as "proper" versioning support in Swift-DocC. I don't believe we're currently tracking this as a feature request- if you'd like to post some details on how you'd like this to work on bugs.swift.org and start a dedicated thread about it, I'd love to chat more.

6 Likes

Thank you very much @ethankusters. I'm very glad the team behind DocC shows such focus on user experience, not only UX for doc readers, but UX for doc producers as well. This level of empathy is quite refreshing :-)

8 Likes