[Pitch] Enable minimal static content in each per-page HTML file by default

Development Swift-DocC
,
1

Hello Swift community,

Swift DocC 6.3 added an experimental feature to supplement DocC's output with minimal static content in each per-page HTML file.

This made the output more easily consumed by tools that don't evaluate JavaScript when visiting a page; for example a search engine crawler/indexer, a large language model fetching a page in response to a prompt, or a custom tool that reads or processes the documentation pages.

Evaluating the impact on SEO of these change has been a recurring topic (1, 2, 3, 4) in Documentation Tooling Workgroup meetings during the last year and @Joseph_Heck has done great work to measure this impact in a small scale experiment.

Even though that experiment was small scale and the data too noisy to be statistically conclusive, I feel that the improvements to social sharing alone—brought by the addition of per-page <title>...</title> and <meta name="description" ...> information—make this feature a worthwhile addition and I propose that Swift DocC 6.5 both:

  • Remove its experimental status.
  • Enable it by default.

Because I feel that this transform-for-static-hosting behavior is right for most people (and I'm proposing that we make it the default) I'm also proposing that we update the phrasing/spellings of the related command line flags so that this behavior has the shortest and most appealing name. Specifically, for these three transform-for-static-hosting behaviors:

  • No transform, meaning that DocC's output requires custom routing rules on the web server.
  • Transform with identical files (as described by this 2021 pitch), meaning that DocC's output can be hosted on static hosting environments (like GitHub Pages) but requires JavaScript to display any page information.
  • Transform with per-page content (as described by this feature's pitch), meaning that tools that don't evaluate JavaScript can access a minimal representation of the requested page's content.

I propose that we update the command line flag phrasing/spellings to use a "without" phrase for the middle behavior rather than a "with" phrase for the last behavior:

Behavior New spelling
No transform --no-transform-for-static-hosting (unchanged)
Transform with identical files --transform-for-static-hosting-without-content
Transform with per-page content --transform-for-static-hosting (default)

This DocC PR both renames these flags and enables this feature by default—if you're curious about the specifics.

If you hadn't heard about this feature want to try it in your documentation today you can pass the --experimental-transform-for-static-hosting-with-content flag to Swift-DocC 6.3 (which comes with Xcode 26.4) or later.

If, as proposed, we enable this feature by default with the proposed spellings of each behavior you will no longer need to pass a flag to enable it but you will be able to pass --transform-for-static-hosting-without-content to Swift 6.5 or later to opt out of this behavior and continue to transform the output with identical files. The --experimental-transform-for-static-hosting-with-content would continue to be supported (as a no-op) in Swift-DocC 6.5 but will be removed after that.

6 Likes
2

I've done a lot of work into this with Kiln. Some observations from my experience:

  • an SPA will always lose against a plain static site for SEO ranking
  • JSON-LD helps massively and also helps layout the results in Google for a more 'professional' result
  • Your correct that the metadata/head content helps, this makes the crawler actually categorise the pages as separate when scraping. Adding canonical URLs also helps
  • (For LLM usage, llms.txt and per-page markdown rel links help)
  • A real sitemap is needed

I think my opinion is pretty clear, but IMO with 7ish months until 6.5 (going off previous release cadence) I think just flipping a flag is too small a change. We should take the time and fundamentally change the rendered sites, rewrite the renderer, run in on a modern software stack etc etc

10 Likes
3

To the original pitch, yes - please. Especially based on what we've learned from enabling the capabilities.

And my perception of the output is leaning even more heavily towards Tim's suggestions - in particular, the lack of canonical URLs in the default output is critical anywhere that we publish more than one copy of some content - Swift Package Index, and now the docs.swift.org site. This steps beyond what you're pitching, but I think we need to allow a developer generating the DocC content to have a default presentation that allows them to be explicit about where they're providing a canonical representation.

Even more broadly, I think we should look at the guts of DocC and figure out how we can break apart the data processing that it provides - capturing data from the compiler, or other external data structures, and combine it with markdown overlays into a well-defined, strongly typed output that can be independent of presentation. I don't think we need scrap what's there, and improving a built-in default presentation is absolutely worth doing, but I see the reliance on a single page JS app to present the content as proving to limiting to the needs of presenting the content.

Stepping back to your pitch - "yes, and". At a very minimum, please let's make the embedded HTML content the default, and if we can squeeze in some metadata into the archive structure to support presenting canonical URLs consistently for all the content, that would push these even farther into a better zone with search indexers and crawlers.

2 Likes