Future of the `transform-for-static-hosting` subcommand?

Development Swift-DocC
1

Hi all,

I want to ask the community if they're using the docc process-archive transform-for-static-hosting subcommand (and if so; how and why) and very informally float the idea of deprecating this subcommand.

The history of this subcommand

The Swift-DocC output—a '.doccarchive' directory—is a single page web application that contains a per-page JSON file which Swift-DocC Render turns into rich documentation web pages using JavaScript. The very first version of this output required custom routing rules on the web server and was incompatible with static hosting environments, like GitHub Pages.

Back in 2021, DocC added both a --transform-for-static-hosting flag to the main docc convert command—to add an identical per-page copy of the render template's "index.html" file—and a dedicated transform-for-static-hosting subcommand to make the same identical per-page copy of the render template's "index.html" file. Later in 2022, this transformation was enabled by default for docc convert, making DocC's output compatible with static hosting environments unless the developer explicitly opted out by passing --no-transform-for-static-hosting. At this point, the usefulness of the subcommand was severely diminished.

Ever since the introduction of this subcommand in 2021, it has been incompatible with DocC's support for custom page headers and page footers (--experimental-enable-custom-templates available since DocC's introductory release). This meant that if a developer built documentation with custom header and footer elements and then ran the transform-for-static-hosting subcommand it would destructively lose the custom header and footer information. More recently, DocC added minimal static content in each per-page HTML file, which the transform-for-static-hosting subcommand also doesn't support—destructively losing this content when run.

Since then, both of these issues have been reported (1, 2) and I was surprised to hear that anyone was still using this subcommand for anything at all.

Are people still using this subcommand?

It's at this point where I would like to ask people if they are still using this subcommand and—more importantly—what workflows they need this subcommand for and why the default version of these per-page HTML files (with custom footers and static content) isn't sufficient.

Possible futures of this subcommand?

Talking through this, thinking out loud. Given the context that there is an ongoing effort (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ...) to build on the already shipping feature to add static content to each page and use it to make DocC output fully static HTML pages as an alternative to the single page web application, I can't help but wonder if this subcommand has a future and if it's worth spending any time and effort to improve it.

I believe that when we get there, this subcommand will ultimately be useless. It will certainly be completely incompatible with the static HTML output but even for the single page web application output, I fail to imagine a use case for running the transform-for-static-hosting subcommand that wouldn't be better served with using the static HTML output from the beginning instead.

The transform-for-static-hosting subcommand has since the beginning been implemented as a basic file copying operation that takes a render template "index.html" file as input and makes identical copies of it for each page. This render template file could be different from what's already in the single page web application output. Redefining this subcommand to instead parse and individually merge the provided render template file with any existing per-page file would be some amount of work that has an opportunity cost to it—we could work on other things instead.

If we haven't done this redefining of this subcommand by the time that static HTML output is an option I don't think we would ever find it worthwhile. In the meantime, I'm still wondering if the opportunity cost is worth it.

Over the coming few years I'm picturing three different futures for this subcommand:

  • We can leave it as-is; a mildly destructive operation that loses custom headers/footers and loses static content. When static HTML output becomes an option we may update it to detect and warn about the complete loss of data if passed static HTML output as its input. We may alternatively choose to remove the subcommand.
  • We can redefine to individually merge and parse files if there is some strong use case that would make this a priority. When static HTML output becomes an option, those (currently unknown) use cases may no longer need this subcommand. At this point we may still consider the subcommand too useless to keep around.
  • We can deprecate the subcommand and communicate its upcoming removal, which could possibly be before static HTML output becomes an option considering that the default per-page template files already have more features/information.

Regardless of what we do in the short term, I personally have a hard time seeing a reason to keep this subcommand around for more than a couple of years. This leads me to wonder if it's better to deprecate it up front; to signal that it won't be further developed and to communicate its future removal. Depending on if there are workflows or use cases—that I'm not currently aware of—that aren't covered by the default per-page template files, that future removal could be anywhere from one or two minor releases (after 6.5 or 6.6) to some number of minor releases after static HTML output becomes an option.

2 Likes
2

I understand I'm a weirdo, but I browse the internet with JavaScript off by default, turned back on on a per-site basis. Sites that require JS just to see the content, no interactivity, lower my opinion of the creator of the page.

Historically search engines have also done better with separate pages for separate content. Whether that's still true I don't know.

4 Likes
3

I do personally agree that DocC's documentation output shouldn't require JavaScript to display—which is a problem that I'm gradually chipping away at. However, this isn't particularly related to the transform-for-static-hosting subcommand (other than the subcommand becoming obsolete once DocC supports fully static HTML output). Until then, we added a different feature in 6.3, as a stepping stone solution, that among other things allows you to read DocC documentation output with JavaScript off (although without custom style and layout). I'm proposing that we enable that feature by default which would increase the the amount of documentation pages that you can read with JavaScript off.

2 Likes
4

I use it after merging. My workflow generates per-target archives with --transform-for-static-hosting already on, then merges them with docc merge, but that command itself doesn't have a static hosting option, so I run transform-for-static-hosting again on the merged output to get it back. Right now that's the only way I know of to get a statically hostable merged archive, so wanted to flag it as a use case before this goes away.

1 Like
5

I don't know where that information came from but to the best of my knowledge that information is completely incorrect.

The merge command doesn't have any specific static hosting options because it doesn't need any. If the input archives contain index.html files the merge command copies those into the combine archive, just as it does with the json files, the assets, etc. DocC's own library/contributor documentation is a merged archive and it doesn't use the transform-for-static-hosting subcommand but still serves per-page index.html files (with minimal static content in each page from the --experimental-transform-for-static-hosting-with-content flag. This code/logic in the merge command hasn't changed in a couple of years but I still retested the few most recent releases and got the same correct behavior on all of them (the merge command copies the index.html files from each input archive into the combined output archive).

For as long as the merge command has existed I don't recall seeing any bug reports in the nature of the issues that you're describing. If this is an issue that reproduces for your content I would appreciate it if you could open a new "issue" on the Swift-DocC repository with steps to reproduce it so that we can fix it before removing the transform-for-static-hosting subcommand. If it doesn't reproduce for your content, running the transform-for-static-hosting subcommand on the combined archive is just an unnecessary destructive operation that I wouldn't consider a valid use case. If anything I'd see this anecdote as more evidence that the transform-for-static-hosting subcommand is easily misused and that removing it may stop people from unintentionally removing some information from the archive's index.html files.

6

Maybe i misunderstood the question i use transform-for-static-hosting a lot in my github actions

          xcodebuild docbuild -project "$PROJECT_PATH" \
          -scheme "$PRODUCT_NAME" \
          -destination "$DESTINATION" \
          CODE_SIGNING_ALLOWED=NO \
          CODE_SIGNING_REQUIRED=NO \
          CODE_SIGN_IDENTITY="" \
          OTHER_DOCC_FLAGS="--transform-for-static-hosting --hosting-base-path `basename ${{ github.repository }}` --output-path ${{ inputs.temppath }}"

snippet from: build-documentation/action.yml at main · 0xWDG/build-documentation · GitHub

7

When I tried merging archives where some were transformed for static hosting and others weren't, I ran into an error about mismatched static hosting support between the archives. I individually transformed the ones that were missing it, and added the post-merge step to make sure that the whole archive was transformed.

Fair point, and it doesn't seem like a huge loss if the goal is to get to a place where the transform isn't needed at all.

8

No worries. That would be using the --transform-for-static-hosting flag for the main docc convert command rather than using the docc process-archive transform-for-static-hosting subcommand which is what I'm asking about.

As mentioned above, the --transform-for-static-hosting flag has been enabled by default since 2022 so as long as you're using Xcode 14.0 or later you can safely stop passing that flag now.

1 Like
9

Thanks for that information!

10

I see. You did correct by transforming the input archives that didn't support static hosting environments but the post-merge step is completely unnecessary (and even slightly destructive).

1 Like
11

I use it constantly for the content I'm working with, and the biggest pain point with it that I've found is using it when working with merging multiple catalogs together.

Today, I'm building a number of catalogs independently, and without hosting base paths, then later combining the archives, setting a hosting base path to apply across the set, applying a header & footer HTML - that's the loose process that goes into assembling the content at docs.swift.org.

For my part, I pretty much always want it on, and I'm using enough of the experimental or augmented commands (enabling markdown output, the header/footer overlay bits, the embedded content in the HTML, etc) that I've definitely hit some speed bumps where I'd like to use combination of those options as it rolls forward when combining and setting a final hosting base path (in my case, representing the version of the suite of content).

I'm quite in favor with on-by-default-and-deprecate up front, with that caveat that I find myself really feeling like I'd like to make the hosting base path update/revision choice something that I can apply last minute, so that I can consolidate the build processing and merging of catalogs prior to that point without each needing to know the explicit base path that I want to use with that static content hosting.

Is there a way we can accommodate that? Support tweaking that specific "base path" value throughout all the nodes within a DocC archive, without messing with all the other content that may be dropped into place (such as the static HTML, embedded header/footer links, etc)?

12

I'm not sure that I understand the steps involved in that workflow. Could you elaborate on the steps a bit? Specifically; how are you applying the header and footer after merging the archives (if I'm understanding that right from the order that you listed them in the description)? As far as I'm aware the only command that can configure custom headers/footers is the main docc convert command.

Also, is there a particular reason why you're not supplying the base path and custom footers and header when creating the individual archives? That would be more aligned with the intended way to use these commands together.

Sorry. I don't follow what this is talking about. What do you want "on"? Are you talking about the --transform-for-static-hosting flag? I'm also struggling to understand how the rest of that paragraph relates to the transform-for-static-hosting subcommand.

If you want to modify very select portions of the HTML files without replacing the files with new copies of a—potentially different version of the—renderer template file, then it sounds to me like it might be easiest to do that with a short custom script. For example, if you only want to modify base path and none of the archives configured it when they were first built, then you could find all the "index.html" files and replace that substring using find and sed:

find path/to/YourDocs.doccarchive \
  -type f \
  -name "index.html" \
  -exec sed -i '' 's/<script>var baseUrl = "\/"<\/script>/<script>var baseUrl = "\/some\/base\/path"<\/script>/' {} \;

That said, I wouldn't recommend modifying the output after it's been built like that. Instead I would recommend that the script that builds and assembles the documentation specify the base path up front, at the same time that it configures the custom headers and footers (assuming that's done via docc convert).

13

In my use case (the combined swift docs stuff), we have a flow that builds the docs for the standard library independently from all the other content, just because it's so devilishly awkward to get the symbols for those modules collected together without building the whole Swift compiler. So today, that part of the build doesn't have the knowledge of where it's ultimately going to be located.

The other options (header/footer inclusion, rendering in markdown output, etc) I set uniformly across the various catalogs before I merge them into a single combined set, and a final step (so far) has been to use process-archive to set the location as a final step in that overall build process. At a high level, the flow looks like:

- download stdlib docc-archive
| ("in parallel sorta, but not really - it's serialized in a script")
- + - build archive 1
- + - build archive 2
- + - build archive 3 (...)
  | 
  | - merge the archives together
    | - set hosting base path for the whole kit

For each archive build, the script clones or updates to the appropriate version of the content, pre-flights by copying in common header/footer bits, and in the case of swift-book, does a smidge more pre-work to support generating the Swift language grammar content that goes into the DocC catalog prior to merge.

The heart of that pattern worked on the idea that a DocC archive holds all the information it needs within it to be re-rendered into another form, and I took that implication (which clearly isn't perfectly accurate) from the fact that the DocC driver commands include commands like:

docc process-archive [subcommand]

A couple of the flows and issues I've found is that this isn't the case today, and I'm not sure if that's a bug or a missing feature. For example, if you "process-archive" like I'm doing to change the hosting base path at the tail end of the whole process, you loose all the detail about the other options you may have used earlier - such as the header/footer HTML injection, rendering out the HTML content and not just a stub, and injecting the markdown variant of the whole thing into the final structure.

These are all pieces from experimental features, so I didn't have a super solid expectation on that front.

If we're intending that archives are only re-creatable if all the details are known up front, and a DocC archive isn't meant to be an intermediate format,
then I think we ought to dump that hierarchy of commands which consist of:

  • docc process-archive index
  • docc process-archive transform-for-static-hosting

If that's the case, then we'll need to push all the options that we need forward to the spot where we're building the catalogs, treating the archives as something to be read, and potentially merged, but which may not be able to be re-created.

For years we've been treading a loose line on if the catalog, and internally its RenderNode JSON data format, are a supported, well defined output that encapsulate all the details about how to present the content or not. It's been the interface between DocC and DocC-Render for years, but it's also not had a lot of public effort put into ensuring that it's well documented (or that the specs are correct, in some cases), or any sense of how to deal with evolving that schema, or what those display options are meant to present.

I think there's more value, and a clearer separation of concerns, if we do treat the DocC archive as an intermediate format that contains everything that's needed to present the content, and move forward to formalizing that interface, defining rules of evolution & migration for that schema, and supporting the idea of breaking how the content is displayed from the content itself.

But that also means considering how we can embed the detail of knowing if the HTML should have the content included or not, if markdown output is desired and how its presented, what the "version" is of this content (which currently really isn't encoded in an archive), and so on.

1 Like
14

Without knowing more about the rest of the build it sounds like this this part of the build could either be updated to:

  • Only be responsible for extracting the symbol graph files for the standard library.
    That would allow a later step that knows the "hosting base path" information to perform the easy step of passing those symbol graph files and a documentation catalog to docc convert.
  • Add a "hosting base pass" pass-through option—or a more general "other DocC flags" option—that this part of the build would pass when it call docc convert to build the documentation for the standard library.

Either way, this seems to me like a solvable problem in the calling script rather than a good use case for the transform-for-static-hosting subcommand.

I can see how it may seem that way but unfortunately that pattern is based on a faulty assumption; the JSON files that make up the DocC output for the single-page web app is only the data that DocC-Render needs to render the page in its current structure and layout. It is not suitable for general purpose transformations and it doesn't include information that's not needed for DocC-Render to render the page in its current structure and layout.

That is more or less what I'm trying to clean up here. The docc process-archive index subcommand is already deprecated and scheduled to be removed after the 6.6 release and I'm more and more leaning towards that we do the same for the docc process-archive transform-for-static-hosting subcommand.

I feel that this is straying rather far off topic from the transform-for-static-hosting subcommand.

The RenderNode JSON is heavily tied to specifics of DocC-Render and is not a suitable intermediate format. The RenderNode type in DocC is rather lossy compared to all the information DocC has in other model objects. Further, the encoding process loses even more information, so a decoded RenderNode value contains less information than the value before it was encoded. A strong indication that the RenderNode type isn't a suitable intermediate format—even before losing some information when encoded—is that DocC itself doesn't use it like that; it is strictly an output format that is structured according to what DocC-Render needs to make serialization more convenient. Further, the few pieces of DocC that are operating on RenderNode values are gradually moving away from that to instead recompute the information they need from source-of-truth values.

If DocC were to have a serializable "intermediate" format I suspect that it would look nothing like the JSON files that DocC outputs for the DocC-Render single-page web app today. I would want such a hypothetical format to not enforce a specific structure and organization of the pages, to support some sort of data querying across "pages", and to include information even if that information may not be displayed on a page. The JSON file output for the single-page web app is none of those things. Instead, I would probably suggest that such an "intermediate" format should be more akin to a SQLite database. As for the structure of that data; two key success criteria for me would be that such a structure would cover a breadth of output from "reports"—like statistics above documentation coverage per symbol—to interactive apps and rendered web pages to read the documentation and that the data would be flexible enough to support presenting different interpretations/facets/organization of those documentation pages.

1 Like