Some time ago now I did some experimentation around adding literate programming support to Swift. The original patches no longer work because the ecosystem has moved on (indeed, part of the reason this didn't go anywhere at the time is that immediately after the patches were created, Swift Syntax landed, making them obsolete).
Anyway, I have updated my patches and added new ones for Swift Build and Swift Syntax.
I like the idea, particularly as part of a multi-source family for Swift.
Java got a second life (and a strong incentive to innovate) from on the JVM-compatible languages (AspectJ, Scala, Kotlin, Groovy, etc), where the same bytecode was generated from multiple source kinds. The key was source-location annotations, so debuggers and IDE's could trace back to the source without themselves parsing it or making assumptions (like one file per type).
Swift doesn't have bytecode, but source could work as the canonical representation, if Swift supported (optional) source-location annotations [2] with the line-control statement. Then anyone could support any format, generate the appropriate swift with source-location annotations, and tooling should mostly just work. It does mean updating the tooling to depend on source-location from the binaries and AST, but that's done only once, and then people can experiment with various forms.
As an example of why not clutter the compiler with specific formats, a niche (but intensely loved) variant of literate programming (mostly in python) has been the Leo editor [1], and their evolution/history can be illustrative: they migrated to a scatter/gather model with include links between files, in part because the same code is often relevant from multiple perspectives (and outlines), so text-stripping didn't work. The model in your evolution proposal is mostly 1:1, in order (just extracting code blocks from markdown/workbook form), but literate code writing might be a bit more like multiple paths through the same garden, depending on the reader's interests and concerns.
(A problem I see for Swift is that people might want both locations, the original source and the intermediate generated Swift source, and it's unclear to me if LLVM binaries support that or whether that would be the work of the tooling to walk back; if the latter, then the proposal has no binary shadow, but the tooling gets more complicated.)
For the source-tooling-oriented folks, can you describe a bit more about how the swift-syntax parser handles mixed content when parsing in a different mode? I briefly scanned the PR but it didn't immediately jump out at me.
My initial thinking is that it would make sense for non-code content to be treated as trivia associated with the nearest actual Swift token; e.g.
# Heading
Some text
```swift
func f() {}
```
## Heading 2
Some more text
func g() {
print("hello")
}
I could see one possible design being that there's a new trivia piece type introduced for external content that should be preserved verbatim. The func in func f(), being the first Swift token in the file, would have all of the preceding Markdown text as its leading trivia. For indented code blocks like the second one (or even indented fenced blocks), each line of Swift code would need to distinguish the whitespace that came from "external content" (the four leading spaces of the second code block) from whitespace that's actually part of the Swift code (the two spaces before print).
But there are probably other possible designs as well, so I'd love if the proposal could go into a bit more detail in its design/implementation section about this part.
This leads to really elegant tooling integrations. For example, swift-format could easily lint just the Swift code inside the file without any other changes. Formatting would likewise be straightforward: you could run a Markdown formatter on the code to format the Markdown parts, and then you could run swift-format on the same file and it would format the Swift snippets inside it, preserving the rest of the surrounding Markdown content as-is.
This is an interesting point. I haven’t added support for “split” trivia like this, but maybe it’d be a good idea to do so, since as you say that would mean swift-format would be able to do its thing without interfering with the indentation required by the document format.
Just so I remember: I need to make sure that I update the pitch with details of any API changes to Swift Syntax (in particular), because there are a couple of things that will be publicly visible here. (I also want to look at the split trivia thing before updating it, to see how easy that is and/or what might be involved.)
Literate programming would be neat! I would personally love to author blog posts and example programs as literate programs. I've written (and have drafted) posts in other languages that are a big markdown file with code blocks, and then hand maintained a git history of compilable project commits.[1] The way Swift allows defining types in separate decls / files, adding more overloads later, feels especially suited. For source compatibility, it's convenient that
let ``` = 3 // error: expected pattern
infix operator ```: // error: '`' is not allowed in operator names
are both rejected.
Is it really the case that an additional program is too much burden? I've never found 'extra build system' arguments compelling (maybe some people have workflows where they run each command by hand, I guess? I find it hard to believe but I'm sure someone does it) and so it feels like your build tooling can just invoke a preprocessor first.
It seems to me also that Swift compiler support would never be sufficient for all the features you (I?) could want here;
say I want to mix JSON, Swift, Python, and C++ for some reason, editor support for each code block, maybe even generating interop types that are exposed to each, with a build process that packs all of that up into a single binary
say I want to describe the evolution of a program, and so I want to name / version code blocks so that I can render the change as a diff, and create versioned binaries that represent different points at time within one literate program
naive version, optimized version but only render the code diff, then render the benchmark
adding support for some new thing, and showing how it worked before and after
say I want to show the result of running some block of code, both in my editor (like a notebook) and in the 'readable' pdf / html product. maybe even an interactive result, compiled to a swiftui based binary!
say I want to write about library evolution, so I want to mix multiple modules in one file
say I want to write about Swift compiler evolution, showing code that can't compile before / can compile now, or how code from different compiler versions interacts
Many of these want very complicated supporting "stuff", like a build system that is very aware of this process with deep editor integration. I'm not sure such a setup would even bother to take advantage of compiler support for "basic" prose and Swift code mixing. Seems to me that it would be better served by a format that represents the literate files in a more structured fashion than flat text or LaTeX, even with an editor / lsp that creates buffers for each code block and relays them to appropriate lsp. I think a format that is a series of "blocks", each mapped to buffers and handed to lsps after running arbitrary transforms (so that LaTeX can also go to an LSP) might be ideal... Many of these look to me like things that go beyond the specced lsp protocol and want editor extensions or fully custom editors.
From a practicality perspective, this seems like it would add one more thing to worry about every time we change the syntax of the language, when it would be more reasonable as a concern outside the compiler and parser. It should be possible to implement an ergonomic LiterateSwiftSyntax parser which parses Markdown, and hands the Swift code blocks into Swift Syntax? Then it would be free to go further and even associate markdown footnotes with parts of the Swift Syntax tree via comments, or many other kinds of exciting features which might not have compiler appetite.
Just to note, Markdown is technically a superset of HTML.
CoffeeScript did something like this, though it was pretty crude. If I remember correctly, they adopted Markdown files as source files, and simply treated every indented block as executable CoffeeScript, and every onside block as prose. The CoffeeScript blocks were then just concatenated together to get the source, and you could render the files however you wished (it's just Markdown).
They had a hack for HTML too, where HTML blocks had to be wrapped in a container element, which began (with its opening tag) at the start of an onside line, and ended (with its closing tag) at the end of an onside line (or something similar).
Personally, in practice, I still preferred to just add comprehensive docstrings to everything substantial, with an occasional inline comment. The real problems (for me) were editing prose inside indented, multiline comments (and dealing with the line breaks) and the amount of space the docstrings took up in the source. I really just wanted a way to move the docstrings into their own files (one for API docs, and one for implementation details), with a way to reliably link them to the source they document.
Observationally, people are using literate programming in Haskell (where .lhs support is built-in), and in LaTeX (where, again, support is built into the tooling). Outside of that, there is not much adoption.
The fact is that literate programming is available with external tooling for any language you please, but people are not using it. I think it really does boil down to it not being a built-in feature, which creates a couple of problems:
Resistance from people who know the language but are sceptical about or unwilling to learn about literate programming.
Lack of build system support — you can obviously get CMake or Make, or even for that matter IDEs like Xcode to do the right thing, but it requires knowledge of the workings of the build system and a willingness to do the work to integrate the extra steps that are necessary when using external tooling.
Building support directly into the compiler (and therefore making it an actual language feature) deals with both of these issues; people can no longer argue that it's "weird" or "non-standard" and that people who know the language won't understand it, and also you don't need to do much build system work (beyond, perhaps, telling the build system that e.g. Markdown files should be passed to the compiler, which is usually considerably simpler to do than setting up rules to use tangle/weave to get the source code and documentation respectively).
It is true that if you are planning on writing a mixed-language literate program, you are going to need to do some extra work. Most other languages require ordering of declarations and definitions, so might want to do some of the re-ordering that the traditional tangle tools support. In principle it might be possible to support this kind of thing in the lexer in Swift also (by creating additional buffers and handing them off for compilation), but because of the language design it's largely unnecessary. A similar thing could be done in Clang's lexer to add direct support there as an extension.
This kind of thing is already possible — the implementation I've linked to has a nocompile attribute that you could use to suppress the old versions. We could obviously do something fancier if it made sense.
Some of the other items would, as you say, be more complex to implement, though note that there's nothing preventing you from using external tooling for e.g. C/C++ on the same input file that you feed directly to the Swift compiler to build the Swift code.
Sort of. Markdown doesn't generally require that characters in code blocks be escaped. Even in an HTML <pre>, you may have to escape some things, and ordinary HTML (as opposed to XHTML) doesn't support the <![CDATA[ ... ]]> syntax that allows you to mostly avoid that.
Supporting HTML code blocks would be complicated, even for external tooling — consider, for instance, the problem of identifying line and column numbers, given that the code must be un-escaped before being processed.
How does this interact with DocC? It would be nice if there was some way to, for example, validate that a particular code sample compiles, runs, and produces the output shown in the documentation.
Sometimes, code like that will require additional setup (for example, access to a type defined on another documentation page, some setup code that isn’t useful to the main point of the page, or wrapping boilerplate like @Test func foo() {}). Could there be a way to hide code from the person viewing DocC output while still running it?
Is each code block required to be independently parseable, or would it be ok for one code block to end if someCondition { and then have the next one pick up with the statement’s body?
It does not. You can, of course, feed DocC a Markdown file that is also a valid source file, but there's no specific interaction with DocC.
Agreed, that's a nice thing to have, though I think it's a slightly different use-case, and maybe something we'd want to build into DocC itself?
For (La)TeX and reStructuredText, I think this kind of thing is pretty straightforward; you can do what you want in TeX, and reStructuredText has the :hidden: attribute. I don't think Markdown really has an equivalent, however; the closest you can get is probably using an HTML comment to hide some text.
There is indeed no requirement that individual code blocks be independently parseable, so yes, it's permissible to open a brace in one and then close it in a subsequent block.
This already exists under the name Snippets. We have been using them in a few places to move sample code in documentation articles into snippets which are verified in CI.