This would be really nice to do anyway; NIO takes a long time to build (probably more than DocC itself).
It’s overkill for the preview server IMO.
1 Like
Yeap, that was what I was doing to test the generated output from DocC on Windows and it was working well. Although it requires having python installed, you need that for debugging Swift code as LLDB depends on it, so its not too bad.
It seems strange that a Swift tool would require a working Python installation to be functional. It brings all the Python-specific complexity of virtual environments and site-specific configuration into the Swift documentation workflow.
In my opinion, this is just further evidence that the design of DocC has made some very unfortunate dependency decisions on web technologies. If DocC had been designed to emit a plain bundle of HTML files like many documentation systems preceding it, there problems could all have been avoided.
4 Likes
I think that this is misrepresenting the situation. The module http.server is part of the core Python distribution. LLDB already requires a working python installation to use the bindings for Swift. As a result, you already have all the necessary pieces installed on your host if you are debugging any Swift code. This is not really adding any complexity due to virtual environments. In fact, the complexity here is due to LLDB and not DocC - LLDB uses APIs that are not version agnostic, so it is tied to the specific version of Python it is built against.
3 Likes
The Python that lldb invokes is not necessarily the same one that is invoked when you type python -m on the command line. For example, on macOS it is ideally invoking the one in the toolchain, falling back to the one provided by the OS. But a developer who also does Python development will have configured their PATH or aliases such that python invokes a version that they installed and have probably set up virtual environments within.
Doesn’t this undermine the use of lldb as a good justification?
Particularly on Windows, this is equivalent to having changed DYLD_LIBRARY_PATH - at which point, you are responsible for both pieces. Path on Windows also serves as the library lookup order, and so, if you change that (directly or through venv) then that directly impacts the Python that LLDB uses. I'm not suggesting that this is ideal, but this is the current reality.
Currently, LLDB is more or less required to properly debug Swift code. You are welcome to help improve the story with WinDBG, but, so far, LLDB has been the recommended way to debug, even on Windows, except in very certain circumstances.
1 Like
I was chatting about this with folks (@al45tair, @compnerd, @daniel-grumberg), and there's perhaps a different approach we could explore.
We could introduce a new portable output format to DocC by leveraging the ZIP format, bypassing the need to actually write files with colon characters to disk. There is good library support for clients to read and write to ZIP archives without decompressing them, meaning that:
- DocC can use a library to write files into a ZIP archive directly.
- You can configure a web server (or even a browser) to serve files from inside the archive.
It could look something like this:
swift package generate-documentation --output-format zip
The benefits are:
- We can keep DocC URLs as-is, with colons.
- Documentation can be generated on all Swift platforms.
- On all Swift platforms, documentation in this format can be hosted using a simple dynamic web server that implements a routing rule to read files from a zip archive (e.g., via libarchive or the Python zipfile library).
- On Linux and macOS, you can host this documentation to a static hosting solution by unzipping the archive.
- The output is portable across platforms, meaning you can build on one platform and host on another.
In addition, we could explore:
- Supporting publishing to GitHub Pages on Windows by developing a tool (maybe packaged as a custom git subcommand), that reads files from a zip file without decompressing it and pushes them to a git remote in memory using libgit2.
- Adding support to DocC Render to read render files from the a ZIP blob directly. This would allow the server to simply serve the ZIP blob and leave the routing to the client. There are SEO, caching, and performance implications to this, though.
I don't have much time right now to go into a more detailed design, but I thought I'd put it out there in case someone wants to explore this idea further and create a forum pitch.
4 Likes
I think a path to cross-platform file names would be good, but I also think the idea of ZIP files is interesting in itself. It reminds me of the help files used with JavaHelp (as JAR files) and CHM files (and the successor format) on Windows (which actually are ZIP files respectively another compression format). Would be nice to have more information about the idea.
It then would be nice to have a small http server for viewing these ZIP files shipped with the Swift toolchain that is also distributable independently (on all platforms). This might even be a specialized application for viewing these files (some kind of “help viewer”, but relying on using a web browser). Neither the “big” NIO nor recommending Python seems to be a perfect solution.
BTW including good ZIP functionalities in the Swift libraries would be a good idea anyway.
FYI, I have a branch of DocC that essentially implements what's been discussed here; I've got it generating compressed doccarchive files rather than directory trees (in my branch, this is optional, but the default on Windows for compatibility reasons), and I've also got a new built-in preview server, removing the need for NIO on Windows, which means you can preview your changes. The preview server can serve directly from the compressed file, or from a directory tree.
The code in the branch removes NIO and replaces it with a small web server written entirely in Swift; and for the compression, all it needs is zlib — it doesn't add any additional dependencies that we aren't already using.
4 Likes
Especially in the age of widespread LLM use, I am even more resolute that the only sensible output format for DocC is a directory full of static HTML or Markdown files.
You can upload them to GitHub Pages. You can FTP SCP them to any web host since GeoCities. You can index them with Spotlight. You can grep them. And nowadays, you can point your LLM at them.
1 Like
I don't think that's true. The zipped equivalent (ala Jar, War, Docx, Xlsx and so on) is a better choice in 99% of cases. You can always expand it if you want the directory tree (for hosting, for instance), and in fact if you are going to deploy it on a web host you'd probably FTP over the archive file and expand it on the host side. It's also pretty trivial to host things directly from a zip file in many cases without even having to expand it.
(Plus I bet a lot of the hosting sites you're thinking about here actually accept zipped files as uploads anyway.)
As for Spotlight, sure, we'd presumably need to do something to tell Spotlight that it's a zip file full of documentation. But we presumably already have all the code for that for the various other zip-based containers.
1 Like
@Franklin @ronnqvist Does this need to be pitched somewhere? (I don't know what the process is for DocC changes like this — does this need Swift Evolution, perhaps through the ESG?)
I'm happy to write up a pitch, or to collaborate on one, or whatever works for you folks. I just want to get to the point where DocC works on Windows just like it does everywhere else.
3 Likes
I think that it would be good to have pitch for this; both to review some details of the solution and to see how that solution fits with other changes that have come up in the two years since this was last discussed. For example: DocC has gained both an experimental static HTML output and an experimental Markdown output. It would be good to ensure that this solution works well for those output formats as well.
DocC uses a less strict, and more loosely defined, evolution process compared to Swift itself. We haven't formally written it down but this post from a year ago outlines the process fairly well and links to several examples of previous pitches to draw inspiration from. Feel free to reach out if you have any questions about writing a pitch for this.
fwiw, I'd love to see a pitch for this, I added some insight here if that helps at all for your pitch.