[Pitch] Extended Error Messages + Diagnostic Naming

As a follow up from Future of Diagnostics UX a couple weeks back, I wanted to further explore the idea of "extended diagnostic messages" and see if they would be a good fit for Swift. What follows is a rough pitch for what that might look like. I'm not sure if this will/should result in an evolution proposal, but I think writing this up using the pitch format makes it easy to evaluate and discuss alternatives.

I'm interested in any and all feedback people have on whether they'd find this useful! I'm especially interested in hearing from anyone who has worked with Rust/Elm in the past and can comment on the benefits & drawbacks of their approaches to diagnostics.

Extended Diagnostic Messages

Introduction

This document proposes adding support for a new form of extended diagnostic message to the compiler which can be exposed through the swift explain command, or an IDE. These new messages will supplement (not replace) the existing diagnostics by providing additional context, examples, and references for beginners learning Swift for the first time and more experienced users when debugging unfamiliar issues. This approach is heavily inspired by that of the Rust and Elm programming languages.

Swift-evolution thread: Discussion thread topic for that proposal

Recommended Reading

Motivation

The existing Swift diagnostics style[1], "aim[s] to fit a terse but clear description of an issue into a single line, ideally suitable for display both on the command line and in IDEs." Top level diagnostics (warnings and errors) are sometimes accompanied by additional notes and fix-its which provide additional information. This focus on precision and brevity has generally worked out well in practice from a user experience point of view. Most of the time, an experienced Swift programmer is able to quickly scan a diagnostic and identify the issue in their code.

However, there are a few cases where the existing diagnostics style may be insufficient. Most of these occur when a user, beginner or otherwise, encounters a new issue or language feature for the first time. For example, consider the following code which might be written by someone unfamiliar with protocol existentials:

protocol P {
  var bar: Self { get }
}

func baz(p: P) {}

This code will result in the following compiler error:

protocol 'P' can only be used as a generic constraint because it has Self or associated type requirements

To be clear, this is a good error message and it follows the existing style guide well. It clearly points out the issue and briefly explains why the code was rejected. While an experienced Swift programmer may be able to fix this quite quickly, however, to a beginner this message can be frustratingly opaque. It assumes the programmer is familiar with both 'generic constraints' and 'associated type requirements', concepts a programmer might not have needed to learn before writing the above code snippet. Today, users frequently end up relying on Google, Stack Overflow, and other resources to understand unfamiliar error messages like this one (a search of Stack Overflow reveals over 200 similar questions about this error message). In the spirit of progressive disclosure, we should instead point users toward authoritative sources when they encounter errors.

Proposed Solution

To address the issues described above, I propose supplementing the existing diagnostics with 'extended error messages'(to borrow terminology from Rust). These are longer form explanations of errors and warnings which provide more-in depth information. The style of these messages and how they are accessed is described below.

Detailed Design

Accessing Extended Diagnostic Messages

Extended diagnostic messages may be accessed through a new command, swift explain, followed by the name of an error or warning. For example, to access the extended error information for the error above, one could run swift explain unsupported_existential_type.

Additionally, if the compiler is not in editor/IDE mode, when a compile finishes, it will emit a new remark which tells the user the name of any emitted diagnostics which have extended explanations:

remark: unsupported_existential_type has an additional explanation available; run `swift explain unsupported_existential_type` to view

This is very similar to Rust's existing model (rustc --explain EXXXX).

TODO: How should these be exposed to IDEs/LSP?

Which Diagnostics Should Have Extended Messages?

Extended diagnostic messages are limited to errors and warnings only. Detailed descriptions of notes should be included in the extended message of their corresponding error or warning if desired.

As a general rule, all errors and warnings should provide an extended message if possible. However, exceptions may be made if a diagnostic's meaning is clear and obvious. Extended error messages should not be required when contributing new diagnostics to the compiler in order to preserve development velocity. However, if this is the case a bug should be filed to record the eventual need for an extended message.

Extended Diagnostic Message Style

  • Extended messages should be written in third-person, unabbreviated English. Like the regular diagnostic messages, they should focus on language rules which were violated as opposed to compiler failures.

  • Extended messages should be written in plain text. Code blocks should be denoted Markdown-style using triple backticks.

  • Code blocks included in messages should only use standard library and user-defined types if possible. Exceptions include, for example, using Foundation types in messages related to Objective-C bridging.

  • Code samples included in diagnostic messages should avoid assigning meaningless names like 'foo' and 'bar' to types and functions, as this can make them more difficult to understand. Instead, code samples should demonstrate realistic, if simplified, uses of language features. As a general rule of thumb, they should avoid exceeding 8-10 lines.

The following is a proposed outline for extended diagnostic messages, heavily inspired by Rust RFC #1567. Deviations from the established format should have some kind of compelling justification.

  • The name of the diagnostic, enclosed in square brackets, followed by the standard diagnostic message restated in unabbreviated English on the same line. If the diagnostic is no longer emitted by the current compiler version, this should be noted, but the message should not be removed.
  • A paragraph explaining the error/warning in greater detail. This longer explanation should describe the language rule which was violated and may suggest ways in which it could be resolved.
  • An example of code which triggers the diagnostic, following the code sample guidelines above.
  • The specific standard diagnostic message output that is emitted when compiling the provided code sample.
  • A 1-2 sentence explanation of how the code sample might be fixed, followed by a fixed code sample. If there are multiple valid ways of fixing the example code which warrant individual explanations, they may be listed one after another.
  • References to sections of The Swift Programming Language which explain concepts relevant to the diagnostic.

Using this style, the extended error message for the existential conformance issue above might be written as:


[unsupported_existential_type] A protocol with Self or associated type requirements can only be used as a generic constraint.

If a protocol 'P' has a requirement which references either Self or an associated type, then the type spelled 'P' does not conform to the protocol 'P' because it is unable to satisfy that requirement. As a result, 'P' cannot be used as the type of a variable, argument, return value, etc. It may be used to constrain the type of a generic parameter 'T' which represents a specific type.

Consider the following incorrect code:

protocol Copyable {
  init(copying: Self)
}

func cacheCopy(of copyable: Copyable) { /* ... */ }

Which results in the following compiler error:

5:30:error: protocol 'Copyable' can only be used as a generic constraint because it has Self or associated type requirements

This function should probably be rewritten to instead accept a constrained generic argument. Consider the following working code:

protocol Copyable {
  init(copying: Self)
}

func cacheCopy<T: Copyable>(of copyable: T) { /* ... */ }

To learn more about generic constraints and protocols with associated types, see the Generics section of The Swift Programming Language.


Alternatives considered

Do Nothing

Writing extended error messages for all of the existing diagnostics would be a large undertaking, and shouldn't be done lightly. It also creates more work each time a new diagnostic is added. We should consider whether time spent improving diagnostics would be better focused on other areas.

More Detailed 'Standard' Diagnostic Messages

One alternative to having both 'standard' and 'extended' messages is to instead adopt a new, more verbose style for existing messages. This would avoid needing to maintain two messages for each error and warning, and is roughly the model used by Elm[3]. However, this wouldn't necessarily be a good fit for Swift. It's unclear how they would affect the experience of IDE users, and potentially loses some of the benefits of the existing style as described in [1]. Additionally, the implementation complexity required to maintain correct messages in all edge cases is significantly higher due to the additional specificity compared to the proposed approach.

Future Directions

Swift Error Index

Rust has an online compiler error index which makes it easy to look up all of compiler's error codes. It also provides a link from each code sample to an online playground where user's can try out the erroneous and fixed code for themselves. If Swift adopts a similar extended message style, it would be nice to set up our own index as well.

Localized Extended Diagnostic Messages

If this feature is adopted, adding a --language option or similar to the swift explain command might be a good way to provide open-source reference material in languages other than English.

24 Likes

I like it. Anything that helps with errors, especially errors using vocabulary or jargon specific to particular areas of programming (like generics) would be great. I do think it would be better for the diagnostic names to follow Swift lowerCamelCase convention. e.g. swift explain unsupportedExistentialType.

Ideally, IDE integration would allow users to click on the diagnostic names to bring up the explanation inline, or perhaps even as a hover action. Better IDE integration with the Swift language documentation has been needed for years, so perhaps this could help drive that.

It could also be neat if, when errors have specific type information in them, for the IDE versions of the explanations to use those types. I'm not sure how applicable or possible that would be. It would certainly help put issues in context outside of the broken code.

1 Like

I think the intention is that the diagnostic names match the ones found in the compiler, which are the underscored versions.

1 Like

That's why I wrote the names using underscores above, but I don't see a problem with adopting a better convention If one can be agreed upon. Some of the existing diagnostic names are pretty user-unfriendly and would need to be changed anyway.


Customizing extended messages using specific types, etc. would be nice and is roughly what Elm does, but I'm not sure it's a good use of resources. Because we're providing so much more information in an extended message, it becomes very difficult to ensure the customized message is correct in all possible circumstances. I believe Rust tried to transition from extended messages similar to what I'm pitching to more customized ones, but it ultimately didn't work out (I could be wrong on this though). That's the primary reason I'm framing extended errors as supplementary information as opposed to a wholesale replacement.

1 Like

Correct me if I'm wrong, but unlike clang which uses the warning/error name in pragma and command line arguments (making them public API), swift diagnostics names are not use anywhere but internally by the compiler. So changing them to use anything should not be an issue, isn't it ?

That's correct, none of the names are public right now so changing or aliasing them wouldn't be a problem.

Anything which makes Swift’s error messages less obtuse is sorely needed and much appreciated.

Many of the existing error messages would be far more convenient if just condensed to a unique arbitrary word mashup, e.g. “potato-nail-fart”, since to make any use of them you have to [on the first encounter] look them up on StackOverflow, and then [on all subsequent encounters] try to keep memorised what that arbitrary string actually means.

Tip-toeing for now around the premise that the error messages are as clear as they really can be, I do wonder if having this as a separate command is superior to having there simply be controls on the error message brevity. e.g. a ‘—brief-diagnostics’ flag or somesuch to maintain the current behaviour, or a converse. It seems natural to me that you’d want to see the useful information as & where the error actually comes up, rather than having to do extra work, probably lose track of things off the top of your screen in your terminal, etc etc.

Adopting a “git”-style “one line summary + thesis” approach might maintain the necessary integration with IDEs etc, where you first need a terse, single-line identifier to indicate the “error code” inline with the code, where space is at a premium, but can easily expose more useful & verbose details through many UI means (e.g. the venerable disclosure triangle).

This could also be helped by adding a lexicon for Swift users, similar to:

Terms from the lexicon could be:

  • hyperlinked in IDE error messages;

  • highlighted somehow in REPL error messages;

  • printed with a swift explain 'associated type' command.

4 Likes

I'm a strong +1 re: conveying more information about errors at the site of error discovery.

Most of s-c-f 5.1's new error messages are longer and attempt to be more detailed in deference to that ideal. I'd love if we could codify clarity and disclosure throughout the stack.

1 Like

This looks good - more documentation about how the language works will surely be useful for some people, and localising this information would be nice. We might also think about localising our standard error messages; while on the one hand it might hurt Google-ability, error messages typically contain project-specific information like type names, so they're not universal anyway and you often need to strip those names when Googling. If the errors all have unique, standardised names, those names can be used instead to search for help.

One more thing: since this is strictly a compiler/QoI feature rather than a change to the language itself, I don't think it needs to go through swift-evolution. It can just be implemented IMO.

Only improving error messages may probably be considered as compiler/QoI only, but if we design a system that exposes some error unique identifiers, they become part of the public API (some IDE and other tools might start using them), so I think this is normal to go though swift-evolution.

That's why I wrote the names using underscores above, but I don't see a problem with adopting a better convention If one can be agreed upon. Some of the existing diagnostic names are pretty user-unfriendly and would need to be changed anyway.

Naming is hard, and the whole premise here is that even the displayed error message is insufficient sometimes, so what hope do we have to make the much more abbreviated name more user friendly?

I think making this even a goal would be a distraction from the overall effort to explain these errors better. If a user needs an error explained, they’re likely no more enlightened by the name “unsupported_existential_type” as they’d be by a nonsense string such as “zxcvb”.

Why not just number these messages, or if we want some additional context, adopt a per-category numbering, where the category would be a broad area or subject such as “type system,” “collection indexing,” etc.?

3 Likes

While for an IDE I think it makes a lot of sense to keep extra information right next to the standard error message behind a disclosure arrow or similar, I'm not convinced it's a good idea on the command line. There are plenty of times the user might not need the additional info, and if there are, say, a dozen errors, showing extended info for all of them might cause them to get lost in the noise.


IDE integration is definitely something which deserves additional consideration, but is somewhat complicated by the fact that the most popular IDE is closed source. I think a reasonable goal in this area might be to aim for great command line and SourceKit-LSP support, while maintaining backwards compatibility of the .dia file format, which I believe is what Xcode consumes.

That's another option, and you bring up a good point regarding the difficulty of naming. Rust and a few other languages have adopted numbered error codes, so there is some prior art as well.

Sure, but in that case why would they not just use '--brief-diagnostics' or whatever (or conversely not use '--detailed-diagnostics', if the default is chosen to be the current brief form)?

Having an explain command or any variation thereof removes any potential enhancements to the output from having the specific error context available. It's unclear to me how that's better than a StackOverflow thread or a formal reference guide - à la the existing language guides - of compiler error messages?

As I noted earlier, I think we should avoid trying to customize extended diagnostic info to its context:

It's arguably true that what I'm suggesting here isn't much more than a reference guide. However, I'd argue that:

  • We should have an authoritative source of knowledge that users can look to before trying Stack Overflow et. al.
  • Ease of access to these resources through compiler integration significantly increases their visibility
  • The combination of a short, specific standard message and a more general, educational message is preferable to long form specific messages which may not always be 100% correct in edge cases and come at the cost of high implementation complexity and slow adoption.

I'm not a fan of the swift explain ... extra step. Why can't error messages have the form:

A brief summary here

Some more detail here,  lorem ipsum dolor sit amet,
consectetur adipiscing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad
minim veniam, quis nostrud exercitation ullamco laboris
nisi ut aliquip ex ea commodo consequat. Duis aute irure
dolor in reprehenderit in voluptate velit esse cillum
dolore eu fugiat nulla pariatur. Excepteur sint occaecat
cupidatat non proident, sunt in culpa qui officia deserunt
mollit anim id est laborum.

For more information, please see: www.swift.org/help/something

The link to out-of-band info (web link, swift explain foo, whatever) is nice, but I don't see why we can't include a few paragraphs directly inline with the error.

Also, if we can format our syntax errors like Rust's amazing psuedo-graphic representations, that'd be awesome.

Related bug, if you're interested in picking that up. Whatever form this takes, I hope along the way we can link a particular error with a unique, searchable identifier.

2 Likes

Agreed, I think this is an important prerequisite. Whatever naming/error code scheme we decide on should be stable across Swift versions, and would be useful on its own as an incremental step towards better diagnostics. Thanks for linking that bug!

@AlexanderM's suggestion:

would also be a great incremental improvement IMO. We're actually pretty close to being able to support a nicer combined presentation of errors and associated notes like Rust's, there's just a little bit more work needed to prepare the diagnostics emission infrastructure.

One reason to use name is that if you ask on a forum "what is error unsupported_existential_type", the experienced users will reply immediately, but if you ask "what is error 102547", everybody will have first to lookup what this error is before being able to reply.

2 Likes
Terms of Service

Privacy Policy

Cookie Policy