Documentation archive validation

Swift Documentation
1

This is a follow-up to our recent Documentation Workgroup discussion.

We are hosting DocC documentation for more than 1,000 open source Swift packages on the Swift Package Index. By default, we build documentation with the latest version of Swift. When we updated to Swift 6.1 we encountered a subtle regression in documentation rendering. The problem in this case was that the sidebar content was incorrect.

We've also had issues in the past where sidebar content was missing or documentation generation failed and led to missing docs.

Of all these issues, we can only reliably detect when docs are missing. To that end, we run regular checks if known documentation pages return 404s.

Obviously, this check cannot detect incorrect content on documentation pages.

This is a concern, given how many pages we host. It is a concern in particular, because doc archives often consist of thousands of files, which we upload to and serve from S3. At 173GB of docs across 30M files, there are almost guaranteed to be upload errors or missing files.

As you can imagine, problems like these are impossible to detect for us. Even if the doc archive app (i.e. the Vue.js hosting app) had some way of enumerating all archive pages, we wouldn't be able to crawl them all to verify them.

However, if we don't have a verification method, we risk breaking documentation on Swift version updates or if we mess something up in our build system. We do have an integration test that ensures documentation generation generates output on all supported platforms. However, this process cannot detect issues like the one I described at the start.

A while ago, Chris, a Swift Mentorship Program mentee, prototyped a solution that loads the Vue.js app in order to run deeper tests than just HTTP status code inspection (https://github.com/msuzoagu/spiDocSmokeTest, finestructure / spi-doctest · GitLab). However, this is quite a slow process and would only allow spot checks.

This is probably still worthwhile to pursue but we hope there could be a more comprehensive solution.

We've been thinking about this for a while and believe two features as part of the Vue.js application would allow for better testing:

  • a health check endpoint that can be queried via HTTP which runs a "structural integrity" check
  • a checksum mechanism covering the many files doc archives contain allowing for a quick way to verify an archive in its entirety is intact

A "structural integrity check" is obviously a fuzzy term but what this should ensure is that the app is up and running correctly. This should include issues like missing data. Sometimes, you can "see" that a doc archive isn't behaving correctly via errors in the JS console but this isn't surfaced in HTTP error codes. The health check should surface these same errors and allow them to be inspected via a simple HTTP call.

Regarding the checksum mechanism, this would be a great help allowing us to verify uploads. It's probably not something we'd use across all repositories (because it would mean re-downloading all files from S3) but it would be helpful to verify an archive is intact on its own if we have to.

What are people's thoughts around this? How are others testing their hosted documentation archives? Is there something else we could/should be doing to ensure documentation pages are displaying correctly?

5 Likes
2

My first thought when this came up was that maybe it was time for a public "integration test" target for DocC, where (with the fancy new GitHub actions) we could create an action that built a well-known, non-trivial DocC catalog with symbols, host the content temporarily, and use something akin to Selenium, Cypress, or Playwright to interact with the single-page JS app (DocC-render) and verify that the results of the whole process end up as expected.

Since docc, the swift-docc-plugin, and docs-render are all independent repositories, my first thought was to make another end-to-end test repository in swiftlang that housed one (maybe more) swift packages within it, and the GH action to do this end-to-end testing, with a first cut of testing the a main or release-... branch of Swift.

The obvious downside of this kind of testing is that it's potentially fragile, since some of what it's depending upon is the output generated by swift-docc-render. On the plus side, I've set up this kind of testing before, and could help in putting something in place.

The idea of a javascript "health check" endpoint in a single-page app that verifies the data integrity of what's expected in the catalog data structures is interesting, but other than verifying that the schema exists as hoped for, I'm not sure we have sufficient information to validate the overall rendering is resulting in what we hope to see.

A checksum or some way to track and tackle "Do I have expected JS files for the rendering" could definitely guard against corruption or mismatched file checking, but I'm not sure that would provide much of a head's up that something is awry for bugs that are missed in the release process and included in a release.

@marcus_ortiz do you have any insights or advice, from your deeper knowledge of the renderer?

3

I generally agree that this type of "integration test" would be useful as a sanity check if it were automated and had some basic but reasonable assertions. Maybe the test fixtures in the swift-docc repo could even be used or expanded on for example content (or maybe the catalog(s) for the DocC documentation itself even).

I'm not sure that checking for specific JS files is more useful than just verifying that the page renders and doesn't seem broken. Any kind of in-depth testing can be more fragile since the tests may need to depend on CSS selectors that might change without warning, etc. There's probably a good balance though of useful things to check that don't involve being overly specific with the resulting HTML so that the tests don't have to constantly be rewritten.

Since the renderer doesn't include its own HTTP server, I don't think that testing the HTTP responses will provide any value since it would just be testing the server at that point (although that may be useful for specific servers that vend DocC content of course).

1 Like
4

One way you could make it easier to fix regressions in the renderer web app specifically would be to remove the HTML/CSS/JS from the DocC archives you host and replace them with a single copy used for the entire site. Assuming the JSON data files remain forwards-compatible, you could quickly update that single copy of the renderer to address any issues without having to rebuild any package documentation.

5

I think that would be very useful. We have some regression test we can contribute and my hope would be that over time we build an early warning system.

One of the biggest problems with us detecting issues at the end of the chain is that fixes might take a longer time to land and be available and that it can be difficult to identify all the doc archives that we generated with bad output and fix them up.

Just to stress that point again: the problem is that often they don't seem broken :slightly_smiling_face:

If I understand correctly, this would mean we'd have to keep around a renderer version per Swift version, because we have packages that render their docs with older Swift versions. (This might be because that's the Swift version that was the latest at the time of their latest release or because they pin their doc gen to an older Swift version.)

I guess that'd be possible but I think it would also mean we'd have to fiddle with the references to those files per Swift version, which would introduce quite a bit of complexity :thinking:

6

My hope was that you’d be able to use the newest version of the renderer for everything (and as a result get all the new features) but if there are breaking changes to the render JSON format over time then you are probably right that it isn’t worth the effort to centralize.

7

At Marcus' prompting, I spelunked through the DocC test suites. There's two test targets that have test fixtures and test bundles included: SwiftDocCTests and SwiftDocCUtilitiesTests.

After scanning the tests themselves, it looks like the first is more focused on internal-to-docc white-box style tests, and the Utilities set is more oriented around functional and integration tests. The bundles used for testing include one labelled LegacyBundle_DoNotUseInNewTests.docc - so I'm guessing :smiley: that using that bundle wouldn't be great.

My first thought in terms of a bundle was to find and grab the SlothCreator demo bundle that exists and is referenced in some of the documentation, I think it's a pretty complete coverage of most of DocC features for rendering, but that's without any in depth analysis. Would anyone from the @swift-documentation-workgroup have different suggestions there? Better sample catalogs that include symbols, etc?

I thought I might assemble something in my fork of swift-docc, maybe as a sub-package, just to see if I can get something useful running with GH actions on my own and prototype out getting something operational. I'm not sure if it would be more beneficial having it in the swift-docc repo, or external to it, given swift-docc-render is external as well, but however we'd like to wrangle it - getting something operational first and vetting that it might do what we hope to see seems like the first step.