[Pitch] Support for custom scripts in DocC

Pitch: Support for custom scripts in DocC

Introduction

We propose first-class support for adding custom scripts to a DocC-generated website. These may be local scripts, in which case the website will continue to work offline, or they may be external scripts at a user-specified URL. This support will be in the form of a custom-scripts.json file (spec), the scripting analog of theme-settings.json.

For a demo of this proposal and the accompanying implementation, see here and here. Notice that the LaTeX expressions in the documentation text are dynamically rendered into equations. In this pitch, we’ll show how you can use the proposed feature to achieve this in your documentation pages.

Motivation

The Single-Page Applications generated by DocC already come with a design and behavior well suited for most documentation websites. While theme-settings.json allows DocC users to customize the design, there is an unmet demand to customize the behavior.

There are at least two use cases with documented community demand for customizing the website behavior: rendering LaTeX and diagramming.

Rendering LaTeX

Documentation pages often need to include mathematical expressions. For example, Apple’s documentation pages for Accelerate (which are made with DocC) are enriched by equations and matrices that help clarify the documentation's text. But the current approaches for adding math to documentation websites — images and Unicode math — are insufficient.

When I pitched support for MathML in DocC as a potential solution — the preceding paragraph is adapted from, and elaborated in, that pitch — the main bit of feedback I received is that some (if not most) documentation writers would prefer to write their math in LaTeX, not MathML, rendered dynamically using a JavaScript library like MathJax or KaTeX.

There is community demand for MathJax/KaTeX support in DocC. And documentation authors would be more inclined to augment their documentation with explanatory math if it were as easy as importing a popular library and writing the math inline, in the standard language for doing so (LaTeX).

Diagramming

Similarly, an open issue in swift-docc-render requests support for rendering diagrams using a JS library like Mermaid or D2.

Diagrams are useful in documentation pages for clarifying the behavior and component relationships of a software package. But, as with math, the current solution for displaying diagrams — externally-generating and including an image — leads to a cumbersome workflow, having to keep multiple files in sync, and subpar results.

Playground

One could also use custom scripts to demonstrate the unique features of your package by embedding, into its documentation website, both:

  • A web-based Swift-to-Wasm compiler (if the documentation website is for a Swift package, analogously if otherwise).
  • An interactive coding playground (loaded with your package).

Then readers can easily try out your package as soon as they learn about it, or experiment with it while consulting its docs.

Whatever you want

The principal motivation for this proposed feature is so that documentation authors can tailor their documentation website to the specific needs of their package. For example, if your package is used for physics simulation, is a game engine, or does low-level rendering, you could add a 3D scene to your website dynamically rendered or simulated with your package (if the package supports the Wasm target). You could embed a WebXR experience, a drawable canvas, or any other technology that may be useful for explaining, demonstrating, or promoting your package.

You could add your own quality-of-life enhancements to the documentation website, or make changes to the stylesheet or page structure that are not yet directly supported. You could have your documentation website show confetti on its release anniversary or add fun easter eggs. Custom scripts remove the limitations on customization.

Proposed solution

Users can add a custom-scripts.json file to their documentation catalog. The file consists of an array of “custom script” objects that specify where the source code of each script is; and, optionally, how it should be loaded and when it should be run. The source code for each custom script can be in one of three locations:

  1. At an external URL specified by a url property.
  2. In a local file specified by a name property. While the file names of local scripts must have the “.js” extension, the extension may be omitted from the value of the name property.
  3. Inline with the declaration of the script; that is, in the custom-scripts.json file itself. Write inline scripts as a string in a custom script’s code property.

#1 (scripts at an external URL) are easiest to add, so they’re suitable for developers who are ok with their documentation website requiring an internet connection.

However, some developers will want their documentation website to continue to work offline even if they add custom scripts. This is useful for previewing the documentation website while developing the package without an internet connection, and it’s essential for developers who want to distribute a documentation archive that can be opened offline. So, in this scenario, the best way to add custom scripts is via #2 (local script files).

For one-line scripts that don’t warrant their own file, inline scripts (#3) are a convenient alternative to #2.

Example usage: adding MathJax

Before jumping into the detailed design: as an illustrative example, below is a custom-scripts.json (with an accompanying local configuration script) that adds the MathJax library to your documentation website.

// custom-scripts.json
[
    {
        "name": "mathjax-config", 
        "defer": true
    },     
    {
        "url": "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-svg.js",
        "defer": true,
        "integrity": "sha384-KKWa9jJ1MZvssLeOoXG6FiOAZfAgmzsIIfw8BXwI9+kYm0lPCbC6yTQPBC00F1/L"
    },
    {
        "code": "MathJax.typeset();",
        "run": "on-navigate"
    }  
]  
// mathjax-config.js
MathJax = {
  tex: { inlineMath: [['$', '$']] }
};

The first object in the example custom-scripts.json above tells the website to run a local script called mathjax-config.js. This script, which can be placed anywhere in the documentation catalog, sets a global configuration variable that the MathJax library will read immediately after it’s loaded. We could have also written "mathjax-config.js" — that is, including the “.js” extension — as the script’s name.

The second object in the example tells the documentation website to load the MathJax library from a CDN. (We could have instead downloaded the library to the documentation catalog and included it as a local script.) Note that we’ve marked both scripts with defer so that they run in order, without blocking the rest of the page while the files are being downloaded.

With these first two scripts, MathJax will render all LaTeX on the page immediately after the library is loaded. But if the reader navigates to a different subpage, any LaTeX there will not be rendered. The solution is to call MathJax.typeset, a function that re-runs the LaTeX renderer, after each navigation. This is what the third “custom script” object does: it’s an inline MathJax.typeset(); script that runs when the reader navigates to a subpage (but not on the initial page load) since its run property is set to "on-navigate". The first two scripts, which do run on the initial page load, have an implicit run of "on-load". If we wanted to run a script on the initial page load and on each navigation, which is not the case in this example, we would set the script’s run property to "on-load-and-navigate".

Detailed design

Spec

The OpenAPI spec for custom-scripts.json is here.

As described in Proposed solution above, each “custom script” object in the array must have exactly one of the following string properties:

  • url for external scripts
  • name for local scripts
  • code for inline scripts

The type, integrity, async, and defer properties — the last two are booleans, not strings — are all optional, and correspond to the identically-named attributes of the HTML script element.

  • The type property of custom scripts, like its HTML namesake, behaves as if it were set to "text/javascript" when omitted. Setting this property is rarely necessary. One use case is importing MathJax 2, for which the configuration script must sometimes have a type of "text/x-mathjax-config", but this is never necessary in MathJax 3.

  • The integrity string is a hash that the browser will use to verify that the script has not been unexpectedly manipulated. It can be used on both external and local scripts.

    • The custom-scripts.json schema intentionally does not have an explicit “cross-origin” (or similar) property. If the integrity property is set and the custom script is external, our implementation will automatically use CORS when loading the script.
  • Inline scripts cannot have the integrity, async, or defer properties.

All three kinds of custom scripts may have the run property, which is an enum string admitting the following values:

  • "on-load", which runs the script when the website is first loaded and the DOM is ready — following the usual rules for async and defer scripts — even if the first loaded page is not the landing page.

  • "on-navigate", which runs the script each time the reader navigates in any way (including via the browser’s forward/backward button) within the documentation website. on-navigate scripts don’t run on the initial page load, even if the first loaded page is not the landing page.

  • "on-load-and-navigate", which runs the script when the website is first loaded and on each navigation.

Technically, on-load scripts are not run immediately after the initial page load, but only after the dynamic HTML for the current route has been added to the DOM. This applies similarly to on-navigate and on-load-and-navigate scripts. That is, any custom scripts that should be run for the current routing event — either the initial page load or a subsequent navigation — are only run after the HTML for the current documentation topic (or landing page, tutorial, etc.) is available. This allows custom scripts, including on-load scripts, to customize the dynamic content of documentation pages.

Folder structure in the catalog

DocC is intentionally agnostic to the folder structure of the documentation catalog.

As a result, the custom-scripts.json file and all local script files can be added anywhere in the documentation catalog. If you group all local scripts under the same parent subdirectory of the catalog, which is not required, you can name that directory whatever you want.

But to keep things consistent within the DocC community, I recommend keeping all local custom scripts under one directory called CustomScripts. This name communicates the directory’s connection to custom-scripts.json while following the convention of catalog subdirectories having upper-case names.

Here’s an example documentation catalog containing custom scripts:

MyModule.docc/
    theme-settings.json
    custom-scripts.json
    CustomScripts/
        script1.js
        script2.js
        script3.js

The file names of local scripts, like all files in a documentation catalog, must be unique per catalog. If two scripts have the same name in the catalog, which one will be copied to the documentation archive is undefined.

Folder structure in the archive

All files in the catalog with the “.js” extension will be copied to a subdirectory of the documentation archive called custom-scripts.

This is intentionally separate from the existing js subdirectory of the archive, which contains the scripts common to all swift-docc-render-based websites (like the ones for syntax highlighting). Separating custom scripts from standard DocC scripts matches how custom images are separated from DocC’s “built-in” images:

Archive subdirectory created by swift-docc containing custom files Archive subdirectory copied from the prebuilt Vue website, common to all swift-docc-render websites
Images images/ img/
Scripts custom-scripts/ js/

The separation also makes it impossible for the filenames of custom scripts to collide with those of built-in scripts (though the latter are suffixed with hashes, so collisions would be highly unlikely regardless).

Privacy and security

In the accompanying implementation, custom scripts can make arbitrary network calls and changes to the DOM — by design. So custom scripts could be used to ping trackers, show ads, or otherwise harass the reader, possibly without the knowledge of the documentation authors who added those scripts.

This raises two questions:

  1. Does this proposed feature make it meaningfully easier for malicious documentation authors to embed malware into documentation websites?
  2. With this proposed feature, could non-malicious documentation authors unintentionally expose their readers to security risks?

The answer to #1 is no: it is already possible, and easy, for a malicious documentation author to embed malware into their documentation website by directly adding a malicious script to the index.html files in the archive. This proposal does not make that meaningfully easier. And, ultimately, authors are responsible for using DocC ethically.

The answer to #2 is also no — if the proposed feature is used responsibly. Only add scripts written by you, your team, or a third party that’s reputable and trusted. If you import scripts from an external URL, take the additional precaution of computing the script’s hash (using this tool, for example) and setting that as the value of the integrity property. Finally, in the unlikely event that you do unintentionally add a malicious script to your documentation website, it’s unlikely that serious harm will be done to your readers given the security protections of modern browsers. Still, if that happens, remove the library and re-deploy your documentation website.

Alternatives considered

First-party, out-of-the-box support for rendering LaTeX, diagrams, etc.

Instead of having users import JS libraries for the rendering needs of their package, we could go through each popular rendering need and add out-of-the-box support for it in DocC. But this idea is not mutually exclusive with our proposed “custom scripts” feature, and implementing the former would not completely remove the demand for the latter.

It may still be a good idea to add first-party support for rendering math and diagrams, and for any other popular use cases of custom scripts. And as new features are added to DocC, the need for custom scripts will decrease.

But DocC can’t have first-party support for the behavior customization needs of every single documentation website. This is chiefly because there are simply too many different packages with different documentation needs, so there will always be some demand for deeper customization of the documentation website’s behavior. And adding first-party support for more and more increasingly-niche rendering features could lead to unmanageable complexity in DocC if we implement those rendering features ourselves, or dependency creep if we use existing libraries instead.

Finally, different documentation authors may have incompatible preferences for how a given rendering feature should work. For example, suppose that some developers prefer Mermaid over D2 for rendering diagrams, and vice-versa. Instead of forcing all documentation authors to use the same library, custom scripts would allow developers to use whichever one they prefer.

Future directions

Import local scripts automatically, without having to add them to custom-scripts.json

In the proposed design, adding a “.js” file to the documentation catalog is not sufficient for adding a local script to a documentation website: the documentation author must also require the script by its name in custom-scripts.json. This aspect of the design is an artificial restriction; however:

  • It greatly simplifies the implementation. The documentation website only has to look in one place (custom-scripts.json) for scripts, as opposed to also having to look in the custom-scripts directory. And client-side JavaScript cannot access file systems, so iterating through the files in the custom-scripts directory may not be trivial.
  • Even if local scripts were imported automatically, adding them to custom-scripts.json would still be necessary when execution order matters, or when the local script’s run property should be anything other than "on-load".
  • Adding local scripts to custom-scripts.json is a good idea regardless because it’s useful to have a centralized view of all scripts being included in the documentation website, whether they’re external, local, or inline.

Still, if there is community demand for omitting local scripts from custom-scripts.json, this could be a future direction for the feature.

8 Likes

Appendix: a good-to-know implementation detail

In the accompanying implementation:

  • Running on-load and on-load-and-navigate scripts on page load is accomplished by dynamically adding HTML script elements (with the appropriate attributes) to the DOM, in the order in which they appear in custom-scripts.json. Since script elements are executed in the order in which they appear in the DOM — modulo the rules for async and defer — this makes it easy for documentation authors to control the order in which custom scripts are executed on page load.
  • Running on-navigate and on-load-and-navigate scripts on each navigation is accomplished with the fetch and eval APIs. This avoids repeatedly inserting script elements on each navigation, but has the downside that execution order cannot be controlled.

This means that, if run is "on-navigate", then async and defer have no effect. If run is "on-run-and-navigate", async and defer only apply to the first execution.

If the community so desires, the implementation can be changed so that all scripts, including navigation scripts, are executed by creating script elements. But adding more and more script elements to the DOM each time the reader navigates could have performance implications if the scripts are sufficiently large.

5 Likes

I haven't had time to dig into the actual implementation details yet (very cool that there already are PRs!), but I just wanted to say that I really like the overall direction of this pitch at a high level as described here.

Specifically, I like the idea of allowing for custom behavior as a general tool to integrate specific third party JS libraries as opposed to having the renderer having to pick a canonical library to implement support for custom kinds of things that developers have expressed interest in, like LaTex, MathJax, etc, etc. This allows the renderer to be more small and focused on the common scenarios while also being flexible to handle more specific needs like math, diagrams, charts, etc if developers opt-in.

I generally agree with your thoughts on the security questions that this brings up, although I would want to make sure that this kind of solution can't easily be abused my malicious actors in the scenario where the docs are not being self-hosted by the developer (thinking about the swiftpackageindex.com as one practical example).

I've also thought a bit about a solution very similar to this that also allows for custom DocC directives that could be associated with custom renderer components as a way of integrating this kind of custom behavior more in the markdown content itself--I think that kind of potential feature could just expand on the foundation you're proposing here though, so it's very cool to see this post. Maybe I'll have to think some more on how the two ideas could work together in some future world if others like the direction of this proposal.

Thanks for sharing this pitch @Lucca-mito! I'll likely be digging into the details and actual PRs more soon.

2 Likes

Hey Lucca,

I don't know anything about the internals of the docs-render side of things to comment on the JavaScript specifics, but at a high level I love the pitch, PRs, etc! I think this would be a huge addition to the DocC system overall. Thank you for pitching it and doing this work!

A massive +1 from me!

1 Like

This topic was discussed a bit at the most recent Documentation Workgroup meeting. I think there was general agreement about the fact that we would want to make this an opt-in feature (command line flag?) instead of having it just be the default behavior. The main reasoning for this is that it would make it easier for non-self-hosted environments like the Swift Package Index website to intentionally either disallow this feature (due to the security implications) or possibly allow a sanitized version of this feature where only popular, vetted scripts could be utilized in a scenario like that where the DocC user does not manage the actual hosting infrastructure.

2 Likes

Hey @marcus_ortiz, I’m glad to hear that my proposal was discussed during the Documentation Workgroup meeting!

And I agree with the security feedback raised during the meeting. My security analysis in the pitch assumed that it’s already possible for authors to add arbitrary JS to documentation websites anyway, but this holds only when the author is the one building the documentation website, which is not the case in (e.g.) SPI. So I’ll add a flag to DocC for controlling whether custom scripts should be copied from the catalog to the archive; no changes should be needed on swift-docc-render. I believe that the DocC flag should be to opt out of custom scripts (that is, --disable-custom-scripts), not opt in (--enable-custom-scripts), since the number of people using DocC directly greatly exceeds the number of Swift package/documentation hosting services like SPI and swiftinit.

(Also — and this is unrelated — the reason I’ve been taking a while to implement the feedback you gave in the swift-docc-render PR, and to respond to @ronnqvist’s feedback in the swift-docc PR, is just that I’ve been busy with some other stuff; I’ll get to them!)

2 Likes

Hey Lucca,

I appreciate your idea of integrating third-party JavaScript scripts into DocC. However, I have some reservations. Implementing third-party scripts could potentially provide a workaround for certain features or directives that have not yet been fully implemented in DocC. For instance, developers might use custom scripts to simulate the functionality of directives like @Diagram for rendering diagrams or @Formula for displaying equations.

While the ability to include these scripts can enhance documentation capabilities, it raises the question of whether it encourages reliance on external solutions rather than addressing the core functionality directly within DocC. This could lead to a fragmented experience where essential features, such as standardized diagramming or equation rendering, remain unimplemented or inconsistently supported across different projects.

Additionally, the use of third-party scripts could introduce compatibility issues, performance concerns, or security vulnerabilities, which would detract from the overall stability and reliability of the documentation.

Therefore, while the flexibility of integrating external scripts is appealing, we should carefully consider the implications it may have on the completeness and integrity of DocC as a documentation tool. It might be more beneficial to prioritize the development of native directives that can handle these requirements consistently and securely.

If I'm mistaken, please feel free to correct me. I’d also love to hear your thoughts on this matter.

Whishing you a great day!
NH

1 Like

Hey, @N3v1! Thank you for the comment; I agree that custom scripts are a tricky problem and that we should be careful with how exactly we support them. However, I’m not quite following.

That’s the idea. These directives you listed (or any directives like them) don’t exist yet, so custom scripts are a useful workaround until they do. And custom scripts are not just a temporarily fallback: since we can’t have a directive for every possible rendering feature (for the reasons outlined in the pitch’s Alternatives Considered section), there will always be some demand to customize the behavior of documentation pages.


Is this arguing that DocC contributors will be less motivated to add features to DocC if there’s a good enough fallback? If so,

  1. I don’t think this is true. DocC has a growing community of motivated and passionate contributors, some of which work on DocC full-time, and I can think of at least one large company with a vested interest in DocC’s continued improvement. I trust that the proposed feature won’t cause the DocC community to start phoning it in.
  2. Does this concern over the drive of DocC contributors outweigh the usefulness of custom scripts?

re: “remain unimplemented”, see above.

re: “fragmented experience” and “inconsistently supported”, it’s true that custom scripts would allow different authors to choose different libraries for their rendering needs. But I see this as a feature, not a bug. Quoting from the pitch:

Leaving the choice of rendering libraries to authors benefits not only the authors, but the docc-render project itself. Quoting @marcus_ortiz,

And quoting again from the pitch,


re: security vulnerabilities, there are two cases.

  1. The documentation author builds (using DocC) and hosts (e.g. on GitHub Pages) the documentation website themselves.
  2. The entity building and hosting the documentation website is not the same as (and therefore should not necessarily trust) the documentation author. This is the SPI case discussed in the comments above.

Case #1 is already addressed in the pitch’s Privacy and Security section. Case #2 is solved by simply disallowing custom scripts using a --disable-custom-scripts flag, which I will implement. (As a future direction, we could have a “sanitized” version of the custom scripts feature instead of having to disallow it entirely on Swift package hosting services.) The fact that custom scripts would have to be disallowed in certain cases for security purposes does not detract from the feature’s usefulness in the common “build the documentation website yourself” case.

re: performance concerns, running custom scripts is as performant as the libraries you choose and the scripts you write. And the option to use custom scripts does not create any overhead if you don’t use it, so authors who don’t wish to slow down their documentation website with custom scripts can simply not add any.

re: compatibility issues, I’m not sure what you mean. Are you referring to JS libraries with poor cross-browser support? If so, you don’t have to (and probably shouldn’t) use those libraries.


The proposed feature increases completeness by allowing authors to “fill in the gaps” of docc-render with rendering features that are missing. As for “integrity”, are you referring to the “fragmented experience” concern? If so, see above.


I agree with the general sentiment here: first-party, out-of-the-box support for something will almost always be “better” than a DIY approach. Quoting from the pitch,

But I don’t think the notion of “prioritizing” is applicable in this situation. Someone implementing a custom scripts feature does not get in the way of someone else implementing first-party support for rendering math or diagrams.

1 Like

Hey @Lucca-mito,

Thanks for your thoughtful reply! I appreciate your explanation and the nuanced approach you're taking with custom scripts.

I completely understand the points you're making about custom scripts serving as both a useful workaround and a permanent feature for flexibility. As you said, "these directives […] don’t exist yet, so custom scripts are a useful workaround until they do." And I totally agree that custom scripts will always have a role in customization, as you mentioned, "there will always be some demand to customize the behavior of documentation pages." Offering developers more freedom in areas like diagram rendering, where preferences vary (e.g., Mermaid vs. D2), is a great point.

One of my concerns was about ensuring that core directives and essential features—such as those for diagramming, formulas, or other common elements—don’t end up being deprioritized or left unimplemented due to the availability of these custom scripts. While I fully understand that custom scripts are necessary for flexibility, there's a slight worry that it might give teams an excuse to rely on external solutions for math libraries or diagram rendering instead of addressing those needs natively within DocC. I mention this with respect, and I trust the DocC community to continue improving the framework, but it’s something I wanted to express as a long-term concern.

Regarding the --disable-custom-scripts flag you mentioned, I realize I had initially overlooked that point, so thank you for highlighting it! It's a very practical solution for managing security vulnerabilities, especially when considering platforms like Swift Package Index. It strikes a good balance, ensuring that self-hosted documentation can still leverage this flexibility without compromising on security. (Maybe a safety check for scripts could be a good addition?)

On the compatibility front, yes, my main concern was related to cross-browser performance issues that might arise from using certain JavaScript libraries. However, as you rightly pointed out, "running custom scripts is as performant as the libraries you choose and the scripts you write." I see now that it ultimately falls on the author’s responsibility to make informed decisions regarding the libraries they choose. Additionally, I also wanted to reiterate my concern from above: by allowing custom scripts, we need to ensure that directives like those you mentioned—@Diagram, @Formula, etc.—don’t end up remaining unimplemented in DocC, as it could lead to the perception that custom solutions are a substitute for native features.

Re: "DocC contributors will be less motivated [...] there’s a good enough fallback [...]"

In response to your question about whether custom scripts would demotivate DocC contributors from adding new features, I think it’s important to look at the bigger picture. DocC has a strong and growing community of passionate contributors, many of whom are deeply invested in its continuous improvement (as you said). Some of these contributors, as you pointed out, even work full-time on DocC. This level of commitment shows that custom scripts aren't likely to reduce their motivation to build out native features.

Rather, the introduction of custom scripts can complement the ongoing development efforts. It offers developers an immediate solution for use cases not yet covered by built-in features. However, this doesn’t mean contributors will stop prioritizing important directives like @Diagram or @Formula. Custom scripts are a tool for flexibility, but they’re not a permanent substitute for widely-needed native features. In fact, as DocC matures, we’ll likely see native support for the most commonly-used custom scripts, reducing the need for external workarounds.

So while I understand the concern about relying too much on custom scripts, I don’t believe it would hinder the natural progression of DocC’s feature set. The motivation to improve core functionality and expand native features remains strong, especially given the community's track record and vested interests of key stakeholders. It’s more a matter of balancing flexibility with robust out-of-the-box capabilities, and I believe the two can coexist without detracting from one another (like you said in):

Whishing you a great day!
NH

Has there been any additional movement on this topic? I’d love to see this implemented.

1 Like

Hey, @creativecat! Unfortunately I’ve been dealing with several unexpected situations in the 3 months since I shared this pitch, but rest assured that the proposal and the accompanying PRs are not abandoned and will not be abandoned. I will get back to implementing the feedback I received in the PRs as soon as I can, which will be this month.

Apologies to the PR reviewers for the delay in my responses, and to the members of the DocC community who are also interested in seeing this merged.

2 Likes

Status update (quoted from here):

(Probably should’ve sent a status update sooner.)

1 Like

Resurrecting this thread since I rebased both PRs, re-re-requested review, then received the (correct) suggestion that it’d be good to revisit the discussion to see how custom scripts in DocC would fit in with static HTML output.

But first

A lightning-round of user-facing changes since the last update, mostly from review feedback and the Documentation Workgroup’s security discussion:

  • Added --disable-custom-scripts so hosting services (e.g. SPI) that build untrusted docs can turn them off.
  • Removed the “inline scripts” feature (code inlined into custom-scripts.json itself): use a local .js file instead. This keeps custom scripts working under a strict Content Security Policy (no unsafe-inline).
  • Removed the defer option from the custom-scripts.json schema, which corresponded to the defer HTML attribute, because browsers ignore that attribute on dynamically added scripts. Scripts run in the order they’re listed in custom-scripts.json (unless you use the async option).
  • Navigation scripts now work just like page-load scripts: they’re re-run with a fresh script element (not via eval or new Function) that gets auto-removed. This means that the awkwardness in this pitch’s appendix is superseded.
  • Navigation scripts now run when the documentation content changes, rather than on every route change. So they still run when switching between Swift and Objective-C, but no longer run on same-page (“hash”) navigation events, like when you click on a heading.
  • One failing script no longer stops the others. Any failures are logged in the console.
  • For docc merge: the combined archive uses the first archive’s custom scripts, like theme-settings.json and the favicon.
  • Fixes: local scripts now support docc preview, documentation websites without a custom-scripts.json no longer log a 404, and custom-scripts.json is now only fetched once.

This means that the pitch is slightly outdated and my “appendix” comment should be ignored. (I’d update them, but the pitch is older than the edit limit for members on Discourse.) Worry not: the public-facing documentation of custom scripts, which I’ll add in a follow-up PR once the feature has been tried out by early adopters, will of course describe the latest version of the feature. I’ve already updated the OpenAPI schema of custom-scripts.json in the swift-docc PR to reflect these changes.

How will custom scripts in DocC fit in with static HTML output?

Nothing much changes.

With static output (--output-format experimental-html-for-development), the distinction between page-load and navigation events disappears: every navigation loads a new page of static HTML. So there’s no difference between "run": "on-load", "run": "on-navigate", and "run": "on-load-and-navigate" scripts.

This means that, at documentation build time, DocC:

  • Ignores any run properties set in custom-scripts.json. (Recall from the pitch that run has always been optional.)
  • Writes the custom scripts (as <script> tags) to the static HTML pages in the order in which they were requested in custom-scripts.json, preserving async, integrity, and type if requested. This is identical to how custom scripts already work in the default (dynamically rendered SPA) output mode, just at build time instead of at render time. The scripts are inserted at the end of the <body>, not the <head>, so that the documentation content exists (and can be manipulated by the scripts) by the time they run.

In other words: in static-output mode, every script acts like a "run": "on-load-and-navigate" script would in the default (SPA) mode. This is not a problem:

  • The main reason to have an on-load script in SPA mode is to do one-time setup that you don’t need to redo on navigation. In the MathJax example from the pitch, that’s running MathJax = {tex: {inlineMath: [['$', '$']]}}. But in static-output mode, you do need to redo that setup on navigation since each navigation loads a fresh page, so you do want your page-load scripts to also run on navigation.
  • The main reason to have an on-navigate (as opposed to on-load-and-navigate) script in SPA mode is to redo work that an on-load script already did on the initial page load. In the MathJax example from the pitch, that’s running MathJax.typeset() on navigation: the on-load script that sets up MathJax also runs typeset as a hidden side effect, so we need to explicitly call MathJax.typeset() on navigation but not on the initial page load. You don’t need this in static-output mode since each navigation loads a fresh page, so you would simply remove the unneeded on-navigate script. [1]

There could be use cases for scripts that should only run on the initial page load even in static-output mode, e.g. if you want to animate some confetti (to celebrate a release milestone) when a reader first visits your documentation website: if the confetti rained down on every navigation, it would get annoying. This is still doable in static-output mode; documentation authors would just need to get a bit creative, e.g. by using sessionStorage.

I’ve already updated the PR with support for static HTML output, and the PRs now link to a second live demo built using static HTML output. (The static-output demo webpage is missing some stuff, like the sidebar, but this is a limitation of the experimental static-output mode.)

If static output becomes the only output mode, we can deprecate then remove the run property.

Also: --disable-custom-scripts works in static-output mode too.


  1. Or, if you build documentation in both modes, you can choose to simply keep the on-navigate script: repeating typeset() is harmless. For scripts that aren’t idempotent, like adding a banner, use a single on-load-and-navigate script in both modes. ↩︎

2 Likes

Thanks for reviving this conversation.

I'm wondering what value the "custom-scripts.json" file adds to this feature and if there exists a simpler solution that doesn't introduce a custom JSON manifest file.

This pitch suggests the full solution that's designed to cover everything as the initial solution and defers to a future direction to simplify the common use case. I would like to see if that can be turned around; start with a simple solution for the most common problem but have a future design direction in mind that can grow to cover more advanced features as well.

To that end, do you know how much of the proposed functionality would be possible with only local JavaScript files within the catalog that are automatically added to the page's head?

2 Likes

Thanks @ronnqvist.

I'm wondering what value the "custom-scripts.json" file adds to this feature

  • Without custom-scripts.json, users can’t request libraries by URL, which is much more user-friendly than authors needing to download the library and manually embed it into their catalog. Requesting libraries by URL also reduces exposure to vulnerabilities: if you request the latest version (or the latest patch of a minor version) of a library, your documentation website will get dynamic security patches for free.
  • And, for documentation websites built using the default (SPA) mode, authors need to be able to control whether a script runs on page load, on navigation, or both. This is done via the run property of custom-scripts.json.
  • Finally, custom-scripts.json is needed to control the execution order of the scripts (if needed) or to opt in to async execution (if desired).

While my original pitch did propose “Import local scripts automatically, without having to add them to custom-scripts.json” as a future direction, on second thought I’m not entirely sure that would be a good idea. It’s perfectly doable: if custom-scripts.json is omitted, the documentation website would run all custom scripts in an arbitrary order (say, lexicographic) with the default run (currently on-load, but we can change this) and default attributes (e.g. no async). My main concern with this is: which .js files in the documentation catalog count as custom scripts to be executed like this? It would have to be all .js files anywhere in the catalog: since DocC is intentionally agnostic to the folder structure of the catalog, we can’t scope it to (say) a CustomScripts folder in the catalog (as noted in the pitch, grouping custom scripts into a CustomScripts folder would be a convention only). So any .js file deep in the folder structure of a documentation catalog, including tutorial code listings or downloadable sample files that happen to be .js, would be automatically executed in the documentation website. This would surprise documentation authors and could be a safety or correctness concern.

So I think the approach in the current PRs is the safer, more ergonomic (for the common use case of loading libraries), and more flexible solution. And custom-scripts.json doesn’t make customizing documentation pages any more complex than the current paradigm, it just follows the existing precedent: theme-settings.json is a JSON manifest for custom styles whose schema mirrors CSS, and custom-scripts.json is a JSON manifest for custom scripts whose schema mirrors the attributes of <script> tags. But I am, of course, open to feedback: if you (or other members of the community) feel that a manifest-less mode should exist, let me know and I can easily add support for that to the PRs.

I personally don't feel like theme-settings.json is a particularly good replacement for CSS and I'm not convinced that the HTML output would support it; with one possible alternative being using actual CSS files for theme customization instead. It's possible that other features following the same design precedent could have a similar fate.


From what I've understood on the JavaScript front; ES6 modules is considered "baseline" today and support import statements with both relative and absolute URLs. The highlight.js README has an example of this that I tested and it seems to work fine as an ES6 module imported in the page's head:

import hljs from 'https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.12.0/build/es/highlight.min.js';

import swift from 'https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.12.0/build/es/languages/swift.min.js';
hljs.registerLanguage('swift', swift);

document.addEventListener('DOMContentLoaded', (event) => {
  document.querySelectorAll('.language-swift').forEach((el) => {
    hljs.highlightElement(el);
  });
});

Similarly, it looks like your MathJax example could either be set up as two JavaScript files:

// mathjax-config.js
window.MathJax = {
  tex: { inlineMath: [['$', '$']] }
};
// custom-script.js
import './mathjax-config.js';
import 'https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-svg.js';

or combined into one, following this example from the MathJax documentation:

window.MathJax = {
  tex: { inlineMath: [['$', '$']] }
};

(function () {
  var script = document.createElement('script');
  script.src = 'https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-svg.js';
  script.defer = true;
  document.head.appendChild(script);
})();

Unless I'm missing something it looks to me like a JavaScript module could solve the issues of both linking to external libraries and of importing files in the right order without needing to define a bespoke JSON file.

3 Likes

True, import from absolute URLs (or requiring authors to insert their own <script> tag) solves the library-from-URL problem, and your example also solves the execution order problem.

Combining all custom scripts into one could also mitigate another problem with removing custom-scripts.json: without a manifest of custom scripts, we'd have to treat every .js asset in the catalog as a custom script, which could be problematic for the reasons in my previous comment. So, instead of running every .js asset in the catalog as a custom script, we could require that authors provide custom scripts as a single file called (say) script.js. Forcing authors to combine their scripts wouldn't be an ideal user experience, and a script.js file anywhere in the catalog could still end up leaking into the website, but this would work as a solution.

The remaining problem with removing custom-scripts.json, though, is that your example doesn't work in SPA mode when the reader navigates.

Suppose we removed custom-scripts.json. In SPA mode, should scripts run on page load only, or on both page load and on navigation?

a. If on page load only: then scripts meant to customize documentation content won't run again when that content changes. For the MathJax example, it means that your example script will typeset LaTeX on page load but not on navigation. So when the reader navigates to a different page that has LaTeX, that LaTeX won't be typeset.

b. If on both page load and navigation: then your example script will append a new MathJax-loading <script> each time the user navigates, and the script tags will accumulate. A second (and third, etc.) copy of the library can throw or misbehave: for the specific case of MathJax this is harmless, but I would imagine that not every library loads and bootstraps idempotently.

Note: option b does work in static output mode (since every navigation is also a fresh page load) and is what the PR implementation does.

Since option b would make custom scripts in SPA mode behave the same as in static HTML mode, let's try to make it work. Authors could use a global variable in their scripts to determine whether it has already run. So your MathJax example would have to become:

if (window.MathJax) {
  MathJax.typeset();
} else {
  window.MathJax = { tex: { inlineMath: [['$', '$']] } };
  const script = document.createElement('script');
  script.src = 'https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-svg.js';
  document.head.append(script);
}

(Your example also had script.defer = true, but browsers ignore defer on scripts added from code, so I removed it.)

But there's another problem: if a reader navigates before MathJax loads, window.MathJax exists but MathJax.typeset doesn't (because MathJax isn't done loading), so we get a TypeError in the if branch. This is a footgun. The author can try replacing the if (window.MathJax) with if (window.MathJax && MathJax.typeset). But now, if the reader navigates before MathJax loads, then MathJax loads twice (the else branch runs again) and the second copy throws a TypeError (I confirmed this experimentally). This is footgun #2. Then the author can either:

  • Handle this third branch explicitly.
  • Keep the original if (window.MathJax), but add script.async = false on the MathJax-loading script. [1] This is a more elegant solution, but it's non-obvious.

My point is: without custom-scripts.json, authors wouldn't be able to control whether their scripts run on page load, navigation, or both, and in SPA mode this would lead to footguns and nontrivial workarounds.

With all of that said, I recognize that requiring these workarounds might still be preferable to having custom-scripts.json. So my two questions to the DocC community are:

  • Should we keep custom-scripts.json to let authors control when their scripts run? Or should we remove it, do option b (scripts always run on page load and navigation), and let authors figure out how to get their desired result?
  • If we remove custom-scripts.json, should we solve the ".js asset leakage" problem by requiring custom scripts to be combined into a single script.js file (instead of DocC running every .js asset)?

  1. This works because dynamically added scripts are async by default, and removing async from the MathJax-loading script forces the second run of the author's custom script (the one with the if/else) to wait until the MathJax-loading script is done. ↩︎