Surfacing educational notes for compiler errors

@Joseph_Heck had some good questions about surfacing educational notes in another thread. I thought I'd answer them in a new discussion thread because it's a slightly different topic from the original one.

The educational notes are currently included as markdown files in the Swift toolchain, but no tools are consuming and rendering them. The goal is to render them via multiple different avenues. For example:

Each of these has a slightly different workflow experience. An educational notes index would primarily be for browsing and searching. In the VSCode Swift extension or at the command line, you'd navigate to an educational note directly from an error message that you got while building code.

Some of the code blocks are not labeled swift because they are command-line output with error messages. I followed the same style that's already used in the Swift migration guide, where the original source example uses swift code blocks and the diagnostics use plain triple backticks to render as plain text. This seems to work okay for rendering via DocC (for example: Documentation), but there might also be a better way to display compiler errors in documentation like this.

7 Likes

Interesting! Is this something that has been there for a while? I don't think that we have included any such files in the Windows toolchain distribution in the past, and I want to ensure that if this is meant to be distributed, we try to include them in the 6.1 release cycle if they are present.

4 Likes

The educational note mechanism has been around for a while. Note that the -print-educational-notes flag mentioned in the PR description stopped working when the new Swift diagnostic formatter was enabled by default for command-line diagnostics. However, this flag dumped the full educational note content for every diagnostic emitted, and I don't think that's the ideal command-line experience.

Up until now, I think we've had a "chicken and egg" problem - no tools were consuming the educational notes because there were so few of them, and more educational notes haven't been added because they're not surfaced anywhere. I've been working on fixing the coverage problem, because I've been writing explanations for concurrency errors anyway when answering questions. There are still very few of them on release/6.1, but I plan to add a lot more on main!

4 Likes

Okay, sounds like it might not be worth it for release/6.1 then? We should get them added on main to prepare for the next rebranch though. Is there an install component that will install these or is that infrastructure that needs to be stood up? The Windows packaging tries to avoid pulling anything from the source tree and has been trying to build up a staged toolchain install to use to package up the toolchain distribution.

1 Like

The install rule is in userdocs/CMakeLists.txt at the moment, build-script doesn't have anything special for them:

swift_install_in_component(DIRECTORY diagnostics
                           DESTINATION "share/doc/swift"
                           COMPONENT compiler)

Thanks @xedin, I had figured that out. That actually is the special thing that we need for windows. There is the COMPONENT that we list so that we build the minimal amount of code.

This is now packaged up and should be in the newer installers as they come out.

2 Likes

Hey all,

SE-0443 introduced diagnostic groups, which have many similarities with educational notes: they tag specific diagnostics produced by the compiler and associate documentation with them. I've been aligning the implementation scheme for these features so that the documentation for a diagnostic group is available in the same way as an educational note.

I've also been thinking about the user experience on the command line: how can we help users find the documentation we write for diagnostics? The compiler currently has -print-diagnostic-groups, which prints the name of a diagnostic group in square brackets at the end of the message, e.g.,

t.swift:2:7: warning: expression uses unsafe constructs but is not marked with 'unsafe' [StrictMemorySafety]

That's useful, but you need to know that [StrictMemorySafety] means it's a diagnostic group, and then go hunting in the toolchain for the corresponding strict-memory-safety.md.

The compiler also has a hidden -print-educational-notes, which renders the Markdown for educational notes into the terminal:

t.swift:2:1: error: non-nominal type 'Crap' (aka '() -> ()') cannot be extended
extension Crap {}
^         ~~~~
Nominal Types

In Swift, a type is considered a nominal type if it has been explicitly named by a declaration somewhere in code. Examples of nominal types include classes, structures and enumerations. Nominal types are an important concept in Swift because they may conform to protocols, be extended, and have values created using the initializer syntax 'MyType()'.

In contrast, non-nominal types do not have these capabilities. Many are obtained by composing other types. Examples include function types like '(Int) -> (String)', tuple types like '(Int, String)', metatypes like 'Int.Type', and special types like 'Any' and 'AnyObject'.

Since a protocol is named by a declaration in code, it may conform to (in other words, refine) other protocols and it may be extended. However, when written as the type of a constant or variable such as 'let value: MyProtocol', the name refers to a distinct, non-nominal existential type that provides a “box” for a value of any concrete type that conforms to the protocol. The existential type itself does not conform to any protocols and cannot be extended, and a value cannot be created using the initializer syntax 'MyProtocol()'.

For more on using existential types, see [Protocols as Types](https://docs.swift.org/swift-book/LanguageGuide/Protocols.html#ID275) in The Swift Programming Language.

This provides the documentation, but: it's quite verbose, the terminal isn't necessarily the best place to read this documentation, and you likely don't want to see this most of the time.

In both cases, you have to find the flag to turn it on. I think we can do a lot better for our users here!

I suggest that, by default, we print the diagnostic group name or educational note name in brackets with a leading # to imply that it's a reference. Then, at the end of the compile, provide links to the documentation for each of the diagnostic groups or educational notes that were referenced in any diagnostic above. The result could look like this:

t.swift:2:7: warning: expression uses unsafe constructs but is not marked with 'unsafe' [#StrictMemorySafety]
1 | func somethingUnsafe(x: Int) {
2 |   _ = UnsafeRawPointer(bitPattern: x)
  |       |- warning: expression uses unsafe constructs but is not marked with 'unsafe' [^StrictMemorySafety]
  |       |- note: reference to unsafe type 'UnsafeRawPointer'
  |       |- note: reference to initializer 'init(bitPattern:)' involves unsafe type 'UnsafeRawPointer'
  |       `- note: @unsafe conformance of 'UnsafeRawPointer' to protocol '_Pointer' involves unsafe code
3 | }
4 | 

[#StrictMemorySafety]: /path/to/swift/toolchain/share/doc/swift/diagnostics/strict-memory-safety.md

For terminals that support it, we could make the [#StrictMemorySafety] a hyperlink to the markdown file and elide the footnotes at the end.

We could also consider rendering the markdown into HTML in the toolchain (as part of the install process), so we're not dependent on the user having a nice way to view markdown files by default.

What do y'all think? Does this hit the right balance between discoverability and verbosity? What could we do better?

Doug

9 Likes

That looks really neat!

The first thought that comes to mind is that it would be great to make sure this works well in CI/distributed environments, too. When a build is happening remotely, a path into the toolchain may not be relevant to the end user:

  • I could start a build (say with Bazel) on my physical workstation, but the compile actions are happening on a remote build farm somewhere, and the path that gets relayed back to me on the terminal would be to the remote machine's toolchain, which might be a different path than where I have it installed.
  • Likewise, if I'm viewing a build log after the fact in a CI environment, chances are I'm reading it in a browser, not on the terminal. Having the note still be navigable there would be a win.

Not immediately sure what the answer here is without introducing too much complexity. A flag that lets us replace the toolchain's install dir prefix with an arbitrary string could be interesting, because we could replace it with a URL prefix and then host /share/doc in a location reachable for our users.

It might be even nicer if swift.org could include docs/educational notes like this as part of automated hosting, like how the LLVM/Clang docs work, so that they're available in a persistent location accessible to everyone on the web, for each version of the toolchain that's released. On a semi-related note, I always find myself wanting to point someone to the description of a swiftc flag, and while the equivalent Clang command line argument reference page isn't the most user-friendly resource, it does have one major advantage: it exists.

3 Likes

Yes, I could use some help accounting for the cases here.

We already have -diagnostic-documentation-path <path>, which is what we prefix to the Markdown file name when showing the path. I think this can also be a URL. Is that sufficient for your uses?

I agree that we should have this, but I'm not sure we always want to point at this URL vs. something that's on the user's local machine. I could certainly see wanting to set -diagnostic-documentation-path https://swift.org/diagnostics/ or similar for CI builds.

Doug

1 Like

I must have missed that one! This is why we need that up-to-date hosted page of flags :grin: I think that would work very well for our use cases.

And totally agree that it's reasonable to not make it the default. But if we had the docs uploaded to some reliable versioned hosting, I could definitely see it becoming common in CI workflows to provide a different value there, for convenience.

2 Likes

Another idea in that space… you could use the DocC that (IIRC) ships in the toolchain to render and serve the markdown.

1 Like

I really like how this lays out the information, and most importantly I love the idea of providing a useful, easy to follow reference to where to find more information, a deeper explanation, etc. I think that kind of detail built into the compiler itself (and supported by the language's infrastructure) provides a better developer experience.

I don't know if there's a formal limit to a compiler error message, but I'd guess at "a simple sentence", as you're trying to keep something inline with error output, presumably exposed underneath or overlaid into the code itself for easiest marking. More information is frequently useful, especially for new features or for new-to-swift developers.

How to wrangle that link is an interesting question - simply because there's tradeoffs with choices, and potentially challenges. In my "ideal world" (meaning I don't really have a clue for how much work I'm suggesting), I'd love to see swift.org directly host such content for the compilers, with stable URLs that were practically an "API contract" for location, so that the Swift compiler - on any platform - could reference it usefully. Maybe that's a lot better built into the toolchain that comes with Swift, and perhaps it should be there as a formal "source of truth". Doing that certainly helps nail down issues relating to versioning and forward and backward compatibility.

I'd love to see even the "more detail" sections here reference the formal documentation for Swift - The Swift Programming Language "book", the standard library documentation, and even the libraries in swiftlang that extend the standard library (collections, algorithms, async-algorithms, etc). I think the more we can weave it (loosely!) together, at least pointing to where more information resides, the better it'll be to see a holistic view of how the details connect.

It's a little beyond the scope of what you're asking here - but I'd love to see that content, even embedded within the toolchain, in static files at a known (that is, guaranteed for external developer tools to use) location. Maybe sourced in Markdown, or maybe rendered in simple html that developer tools cloud opt into including and displaying alongside the code. The tricky point is the logical extension - as Markdown can start to break down with more advanced formatting features (tables, term lists, and what - if anything - is supported for embedded blocks). But if we're talking about "a couple of sentences, and maybe an HTML hyperlink" we're well beneath the threshold of complexity where different markdown rendering tech starts to make a notable difference.

Doing all this for compiler messages as well as educational notes is a great capability win. Today a developer has to search for it, and frequently that's a search engine query. It'd be great to be able to help direct that search a bit to make things easier for the developer to understand what's happening.

1 Like

I see a two challenges with using DocC - although the docc renderer has lovely output that I'd love to see us take advantage of and build on the doc experience already in Swift.

The first challenge is the URL construction (the location of the file) is relatively hard to control - it's always under /documentation/ to support the single-page JS application that's presenting the information with the final URL location pinned to the filename for article-like content.

The second is that today it (docc rendering) requires the JavaScript app to be running to see the content, since it's rendering on the fly from the archive. That makes consuming this from arbitrary developer tooling (Xcode to visual studio code to NeoVIM or emacs) far more challenging.

That said, if this is an argument in favor of getting static-html-output support into DocC tooling to make it easy to create single or sets of static HTML pages with content, count me in on that!

1 Like

A few miscellaneous thoughts in no particular order:

The local vs CI/remote builds discrepancy is largely why I was originally thinking of exposing a swift-explain executable back in the day - assuming identifiers are stable and the user has a toolchain at least as new as the one which produced a log, it would allow users to just copy and paste a command to access the docs. This is basically a copy of Rust's approach, which has worked out well IMO.

If we did that, I think the printed output could look something like:

[#StrictMemorySafety]: swift explain #StrictMemorySafety

and perhaps also add the path if attached to a tty

[#StrictMemorySafety]: swift explain #StrictMemorySafety (/path/to/swift/toolchain/share/doc/swift/diagnostics/strict-memory-safety.md)

VSCode support: this was added in Expose diagnostic educational notes as diagnostic codes by owenv · Pull Request #345 · swiftlang/sourcekit-lsp · GitHub using the DiagnosticCode api in VSCode. These show up as hyperlinks at the end of the diagnostic text which can be opened in the editor.


It might be worth moving the notes to a docc catalog even if it's a little different from the traditional use cases. They're loose markdown files right now just because the feature predates docc. Doing so would allow:

  • Potentially simplifying the Windows installer rules
  • Adding an optional mode to the hypothetical swift-explain to spin up a local web server instead of printing docs to the terminal

I don't think it makes sense to ship HTML artifacts in the toolchain itself - IMO it's useful to keep the content as human-readable as possible in its raw source form

4 Likes

[#StrictMemorySafety]: swift explain #StrictMemorySafety

Note that # is the comment leader in some Unix shells (e.g. zsh, bash) and that would make this difficult to use as you would need to do swift explain '#StrictMemorySafety'.

2 Likes

I think it's reasonable to have swift explain <identifier> for any of these names, which could do the Markdown rendering. That said, I still like the "one click" approach of having hyperlinks to the documentation files, but perhaps that makes the most sense in an IDE vs. on the command line.

This feels like it would get really verbose if there were more than 2 or so, but I suppose that's okay.

Some structure could be useful for publishing on swift.org as well. I don't have strong opinions on DocC catalog vs. something simpler and purpose-built.

Looking at the educational note infrastructure, I think I'd like to simplify down to say that we can have at most one document associated with a given diagnostic. That way, we don't end up with a pile of references for a given diagnostic, an everything can be simplified down to "zero or one category with its optional documentation."

Doug

2 Likes

I certainly interpret this as an argument for static-html-output support for DocC tooling. We don't need a lot for this documentation: static HTML with some decent Swift syntax highlighting that we can bundle in the toolchain. We could probably build something like this on top of swift-markdown fairly easily, but I'd rather not duplicate effort if static-html-output support is on the table for DocC.

Doug

1 Like