[Pitch] Compressed DocC Archives

Development Swift-DocC
1

Hi everyone,

I'd like to propose a new, compressed, output format for Swift-DocC.

Motivation

Swift-DocC likes to generate filenames that contain characters like :s that, while fine on UNIX-like platforms, are fundamentally incompatible with CP/M-family operating systems like Windows.

Additionally, while macOS has features that allow directory trees to look
like and be treated like files, this is not the case on other platforms and
dealing with a tree of files is harder than dealing with a single file.

Proposed Solution

The idea here is to make Swift-DocC write out a .doccarchive file that is, under the covers, a zip file. There is plenty of precedent for this kind of behavior — Java jar and war files, Microsoft Office documents like docx, xlsx and pptx, Pages, Numbers and Keynote documents are all zip archives.

This fixes the filename problem, since the only restriction on filenames in zip files is that the / character is reserved as a directory separator (this is essentially the same restriction as UNIX has).

It also results in less disk usage and more convenient manipulation of .doccarchives on Windows and Linux as they are now a single file, rather than a directory hierarchy.

Additionally, for use-cases where it is desirable to have a directory tree, standard tools can be used to expand the zipped archive.

Changes to DocC command line

A new flag, --[no-]compress will be added for the convert and merge actions. This flag will be on by default for Windows.

Impact on DocC's dependencies

This change does mean that DocC will need to be able to generate compressed archives. We would prefer to keep the set of dependencies require to build DocC as small as possible, and ideally to avoid anything that isn't already a dependency of some other Swift component.

To that end, we will only add a dependency on Zlib.

Preview support

The preview server within DocC will be updated such that it is able to directly serve files from a zipped archive, in addition to its existing support for directory trees.

Alternatives Considered

Changing DocC to not output special characters in filenames

The downside of this is that it will change the URLs, at least for the static file serving case. We could have made this change just for Windows, but if we did that then the URLs would differ depending on whether the documentation archive was built on Windows or on some other platform, which is itself undesirable.

It is possible to work around the URL issue in the web server, but the details depend strongly on the web server that is in use, and explaining to users how to configure that would be tricky, as well as leaving them with a more fragile web server configuration.

Making compressed output the default everywhere

This might break existing build processes. Users building documentation on Windows need to use compressed output because not doing so simply doesn't work, so making it the default there seems the right choice.

If we want to make it the default everywhere, we can do that in a future release, giving people plenty of notice that it's going to change and that they'll need to add --no-compress to maintain the present behavior.

Using some other container format besides zip

We could have used tar.gz/tar.bz2/tar.xz, or one of Microsoft's various multi-stream formats, or done our own thing.

Zip has the advantage that it is standardised (as ISO/IEC 21320-1:2015), well documented and widely supported across multiple platforms. Unlike some of the other choices, it also compresses files individually and has a central index, which makes serving content directly from it relatively efficient.

More technically adept users are also used to the notion that some modern file formats are really zip files internally.

Using a new file extension rather than doccarchive

DocC itself doesn't really care what name you use for your output anyway, but it seems entirely reasonable to treat a doccarchive file as a zip, and a doccarchive directory as a directory tree.

There doesn't seem to be a good reason to e.g. add an x on the end or to use doccarchive.zip or anything of that nature.

7 Likes
2

-1. I am opposed to investing in any more proprietary formats before finishing the format that should have been DocC’s original all along: a directory full of flat HTML files that can be served from any static file host, with no JavaScript or URL rewriting dependencies.

DocC has not been successful with its unique delivery format so far, and it will continue to be unsuccessful by doubling down on a second proprietary format.

4 Likes
3

I view the topic of filenames and the topic of output formats (static HTML or a web app) as related but still different.

To give a more specific example; if DocC output a static HTML file at SomeModule/SomeClass/someMethod(with:and:).html (or SomeModule/SomeClass/someMethod(with:and:)/index.html) then we would still have compatibility issues with Windows because of the “:” characters in the file path.

A zip archive output can solve that filename issue regardless of whether the archive represents a directory of JSON files or a directory of HTML files.

4

A ZIP archive only solves the issue if you serve it from the DocC server. You can’t generate the ZIP file from your Windows machine, upload it to your webhost, and unzip it.

5

Yes, you can.

1 Like
6

To be clear, all this proposal is suggesting is adding (optional) compression support to DocC, which (a) fixes existing functionality on Windows, (b) lays the groundwork for support for other compressed formats in the future (e.g. ePub), and (c) is generally more convenient to work with than the directory-of-files approach in many cases.

This is not adding any new formats, proprietary or otherwise, and the compression feature will work with both the JSON and the experimental static HTML formats out of the box.

4 Likes
7

I assume this would apply equally to tutorials, since they are bundled in the same doccarchive files. Is that right?

Today macOS interprets a doccarchive file type to conform to public.directory:

% mdls -name kMDItemContentTypeTree  empty.doccarchive 
kMDItemContentTypeTree = (
    "com.apple.documentation.doccarchive",
    "public.composite-content",
    "public.content",
    "com.apple.package",
    "public.directory",
    "public.item"
)

This pitch proposes that some archives will conform to public.directory and others will conform to public.zip-archive. Is that going to cause confusion on the mac?

8

Overall, I like the idea: it solves two problems at once.

The only thing that I am a little unhappy about is the additional dependency on zlib. We already have a copy of zlib in the toolchain, so it shouldn't be too big of a deal, but if we can avoid another static copy of it being embedded into tools, that is preferable. I suppose that we can start dynamically linking zlib across all the products.

1 Like
9

This pitch does not propose either of those things. Exactly what the doccarchive content type is registered as is up to Xcode, and as I noted, DocC doesn't really care what extension you use.

However, the switch from a directory to a zip archive is not without precedent; Pages, Numbers and Keynote have all done this in the past; the current file type for Pages is:

$ mdls -name kMDItemContentTypeTree empty.pages
kMDItemContentTypeTree = (
    "com.apple.iwork.pages.sffpages",
    "public.item",
    "public.data",
    "public.composite-content",
    "public.content"
)

which I note doesn't say it's a public.zip-archive (it just says it is public.composite-content). I suspect the type registration could be adjusted to simply remove public.directory.

1 Like
10

My reading is that this is an additional (preferred) format, not strictly a switch. If I use --no-compresswill that write out a bundle with a doccarchive extension?

A counter-example is rtf/rtfd where there are two related but different formats expected to coexist.

11

the format that should have been DocC’s original all along: a directory full of flat HTML files that can be served from any static file host, with no JavaScript or URL rewriting dependencies

That would be brilliant. Anyone can work with a directory of html/css, either locally or hosting it.

12

That's correct, yes. Though currently I'm proposing that, at least initially, --compress will be the default only on Windows in order to maintain current behaviour, and if you were to use --no-compress there, DocC may fail to build the output (depending on the filenames it generates).

13

Can you expand a bit on the choice of zlib?

If we're going to compress at all, not using one of the several widely-supported formats that achieve both better compression ratios and energy usage than zlib seems somewhat hard to defend in this day and age. Given that we have (relatively speaking) carte blanche to pick a compression algorithm without substantial baggage, why aren't we using one of the clearly-better alternatives?

2 Likes
14

My thinking here is:

  1. zlib is already a dependency of various things in the Swift ecosystem; we won't be adding a new dependency here (overall; it is new to DocC, of course).
  2. Using zip as a container format is a well-established path, and deflate (the algorithm zlib uses) is the most widely supported and therefore most compatible choice for compression in zip files.
  3. Deflate is the only compression algorithm supported by ISO/IEC 21320 (the ISO spec for a reduced complexity version of zip, which is commonly used as a container format by other things).

I agree that there are better choices if the main point here was compression rather than the ability to use a zip file container.