SE-0217 - The "Unwrap or Die" operator


#204

They do. Just as some other care about the fact that the error message is embedded in eventual crash reports even in distributed builds.

But this file/line problem does not stop them from exploring the topic and feed the community creativity. If the conclusion is that operators should be able to handle #file and #line, so be it.


(Pavol Vaskovic) #205

Have you read the proposal?

Unfortunately, there is no direct way at this time to emit the #file name and #line number with the above code. We hope the dev team can somehow work around this limitation to produce that information at the !! site. The !!-alternative design that uses a () -> Never closure in the "Alternative Designs" section provides that information today.

Next section is literally called “Future Directions, Calling Context”. This is one of the reasons why the proposal seeks inclusion in the standards library, so that this shortcoming will be addressed.


(Pavol Vaskovic) #206

My impression from rereading Chris' objections is that we still have a misunderstanding of the proposal's motivation, especially given the side-points about try! and as! and the proposal for .forceUnwrap("reason here"). So I'll try to restate the motivation as I understand it:

Unwrapping Precondition Operator

This proposal seeks to establish the community’s best practice of documenting the precondition for forced unwrapping optionals with the !! operator as an easy, convenient and preferred alternative to using the plain forced unwrapping with !, in order to reduce its prevalent misuse.

let value = wrappedValue !! <# Document the forced-unwrapping precondition #>

Motivation

Swift language has adopted a type-safe solution to the billion-dollar mistake of null references in the form of Optional type. It went even further and made use of optionals in the language mostly opaque by introducing a lot of syntactic sugar around them: from the type system’s T? as a type alias for Optional<T>, to easing everyday use with optional chaining.

These facilities should in theory support progressive disclosure of language features, offering new learners a gradual on-ramp. But in practice, Swift users are confronted with optionals long before they fully understand the concept, because of their prevalence in Cocoa APIs. Aided by the compiler fixit suggestion to make their code compile, they quickly learn the anti-pattern of mindlessly slapping ! after optional values. Survey of questions on developer forums reveals a bleak picture of forced unwrapping misuse, often combined with unnecessary propagation of optional values — all signs of general misunderstanding of optionals and their proper use. The scale of the problem can be estimated by looking at the Stack Overflow question about “fatal error: unexpectedly found nil while unwrapping an Optional value” in Swift and the question about the prevalent Null Pointer Exceptions in Java. Comparing these two data points, it looks like the misuse of ! is responsible for negating most of the type-safety wins promised by optionals.

“A programming language designer should be responsible for the mistakes made by programmers using the language.” — Tony Hoare

Proper handling of optionals with multi-line constructs like optional binding (if-let, guard-let) is comparatively discouraged when the alternative (readily offered by compiler fixit) is to add a single ! after the optional. This has been readily recognized by the introduction of nil-coalescing operator ?? which offers an easy to use syntactic sugar for the case, where the missing optional can be substituted by a default value.

This proposal seeks to add complementary operator !! for a case when there is no reasonable default value. This is conceptual extension of the proper use of the forced unwrapping with !, which forces the user to explicitly document the precondition. By stating the reason for why it is safe to forcefully unwrap, it serves to communicate the underlying assumptions in code, aiding clarity at the point of use. It can also be used to provide better diagnostic message, when the precondition is violated at runtime. The right hand side of the unwrapping precondition operator can use lazily evaluated string interpolation, to further help in debugging the programmer error.

With introduction of this operator, we propose the compiler fixit that offers to Insert ! be replaced with two fixits that encourage the use of ?? and !!.

The unwrap precondition operator is notably less convenient than pure !, because it includes the precondition. We consider this a feature, nudging users to find a better solution, like early unwrap when getting optional URL from failable initializer. This would help avoiding accidental optional chains, where it’s required to wrap !! application in parenthesis. It still provides a compiler fixit that makes the code compile, but it makes it more apparent, when it is just a mindless misuse of forced unwrapping.

On the other hand, !! is more convenient syntactic sugar for a multiline guard-let with precondition, making it much easier to use properly. It removes a lot of unnecessary ceremony when forced unwrapping is the right thing to do. The operator form composes well in variable assignments and in method calls where one of the parameters needs to be unwrapped. The precondition with the operator can be neatly tucked away on a new line, when it doesn’t fit. It says: “watch out, I’m forcefully unwrapping here (!!), but I believe it’s safe because of this reason”.

let indices = Dictionary(uniqueKeysWithValues:
  zip(knownTests.map{$0.name}, 1...))
let namesOrIndices = Set(filters)

tests = knownTests.filter { test in
  if namesOrIndices.isEmpty {
    return test.tags.isSuperset(of: _tags) &&
      test.tags.isDisjoint(with: _skipTags)
  } else {
    return namesOrIndices.contains(test.name) ||
      namesOrIndices.contains(String(indices[test.name]
        !! "All knownTests have been assigned an index"))
  }
}

This leaves forced unwrapping with plain ! as a power tool, to be used in cases where it is obvious from the surrounding context why the unwrapping cannot fail. When in doubt, documenting the precondition with !! should be the default choice.

We believe that stating the unwrap precondition using the operator in proposed form strikes the right balance between discouraging the anti-pattern of ! misuse and promoting best-practice of documenting the precondition for a situation where force unwrapping is the right thing to do.


(Brent Royal-Gordon) #207

I'm a little irritated that you put it this way, because we do care about it—the proposal specifically calls it out as a shortcoming that would require language support to fix.

But in the spirit of "don't get mad, get even", I decided to implement that support. This implementation is very quick and dirty and needs some clean-up, but it works fine. It breaks two tests that we reject invalid operators; they'll be easy to fix.

So I'd appreciate it if we could argue about some other aspect of this proposal. Now, if you'll excuse me, the sun came up while I was doing this, so I should probably get some sleep.


#208

:heart:


(Xiaodi Wu) #209

I have to say that this is an excellent write-up which lays out very clearly what I, too, understand to be the motivation of the proposal.

One of the benefits of having such a good write-up is that, with clarity as to the underlying argument, contrary viewpoints can also be made more clearly. So here goes:

That users have trouble with unexpected nil both in Swift and in other languages such as Java points to the underlying issue: users expected something and got nothing. This is a logic error independent of what language is used. It must be remembered that Swift's syntax does nothing to cause that, only to reveal it. Ideally, such logic errors would be revealed at compile time; but better at runtime than never.

If this topic is overrepresented only for some languages on Stack Overflow, I would take that as a good sign that those languages help to surface this logic error in some way, as opposed to other languages that carry on silently in the face of such an error. Therefore, I do not consider it a good litmus test or goal to minimize the posts that arise on Stack Overflow.


Where can the language further help users to avoid such logic errors, ideally at compile time? It can help in three main ways:

  1. by not promoting the presumption that there is something where there might be nothing;
  2. by helping users to figure out when and where they got nothing instead of something; and
  3. by helping the user to substitute something or take an alternative action when there is nothing.

How can we improve in each of these respects?

For (1), we can get rid of the ill-advised ! fix-it.

For (2), we can improve diagnostics; for example, it would be nice if we could easily see not only where the user attempted to unwrap nil, but where a function call (for example) originally returned that nil.

For (3), we already have the compiler offering ?? as a fix-it; we can also offer guard let forms where appropriate.

These are all improvements that can be made which require no language change, and I would heartily support all of them.


Does the proposed operator advance any of these areas? In my view, no:

By proposing !! as a fix-it, we continue to promote making the presumption that there will be something when there might be nothing.

User-supplied diagnostics have some possibility of improving (2), but they are necessarily limited by user insight into their own logic error. This might be helpful for the advanced user, but as you point out, your motivation is centered on users who do not understand optionals, for which this is of limited use. For reasons I pointed out earlier, the overwhelmingly common scenario will likely be a user-supplied diagnostic that simply recites the meaning of unwrapping.

By offering to trap instead of performing some other routine in a guard body, the fix-it would discourage rather than help the user to consider alternative ways to address an unexpected nil value. For example, I would wager that, in a callback for a button in an app, many cases of nil would be appropriately handled by guard ... else { return }. A user prompted to contemplate what goes into the else block would easily arrive at that conclusion; a user encouraged to insert a runtime trap would not.

Comparison is made to the usefulness of the nil-coalescing operator. However, the expected-something-got-nothing problem of ! is to be distinguished from ??, which allows something to be substituted instead of getting nothing, a task which fundamentally means the user's logic is correct rather than erroneous.


(Matt Rips) #210

Right. Also, often it is not a logic error; it is an intentional act by the user: "I know this value could be nil, and I am using ! and accept that the program will crash, because I haven't yet learned how to handle this situation properly."

Perhaps fix-it could explicitly teach the correct patterns as follows (wording is conceptual, not according to style guide):

  1. If possible, provide a default value using ??
  2. If a default value is not appropriate, provide an alternative logic path using guard let ... else
  3. If the value is logically guaranteed to not be nil, use .forceUnwrapped(willNotBeNilBecause:)

In this fashion and perhaps with a few more words than usual, the language can teach users the patterns that Swift intends.


(David Waite) #211

I'm -1, pending real-world usage.

A small part of this is due to the use of the operator in a 'non-homogenous' way, but my main concern about real world usage is based on this line:

let value = wrappedValue !! <# "Explanation why lhs cannot be nil." #>

This explanation has two audiences:

  • The persons reviewing the code to why force unwrapping was done, e.g. why there can't be a default value on nil, or why they assumed the data could never be nil
  • The persons diagnosing an issue about why the failure occurred.

For the line:

let forums = URL("https://forums.swift.org") !! "swift makes us unwrap these values"

would a fatal error of 'swift makes us unwrap these values' in any way be appropriate for a customer to see, or help the developer diagnose the issue faster (other than potentially having a string to search for across the code base?)

For this reason, I'd prefer to see something like this adopted as a library and in coding standards and actually used before being added as part of the language. I simply don't trust the ergonomics because of this ambiguity

Absolutely for both, if the ergonomics are correct.

None

quick reading


(Howard Lovatt) #212

For what its worth Kotlin allows:

val name = node.getName() ?: throw IllegalArgumentException("name expected")

Which is well liked in the Kotlin world. The above works in Kotlin like ?? that accepts a Never on the rhs would in Swift.


(Hooman Mehr) #213

That is cool! Sorry for keeping you up last night.

Now we need to decide if enhancing operators like that is a good idea or not. !! proposal already provides an example of how it could be used. For the moment, I am undecided.

I still don't think adding !! operator is a good idea, even with file/line included. Lack of file/line is not my reason for opposing it.


(Braden Scothern) #214

While I'm not a fan of the proposed !! operator, I understand peoples desire for it to be sugar added to the language. I would rather it not be added right now (@Chris_Lattner3 has a lot of good points about the future of Swift and the dangers of sugar), but possibly in the future as a shortcut for an unwrap(because:) function (which is a function I am willing to support).

I think that the discussion above of backing operators with functions is a very valuable notion since they chain much better, especially if we can get functions like unwrap(because:) to work without an extra set of ()s around optional values.

Slight tangent and potential new can of worms here...
That said, if we do end up adding a function like unwrap(because:) should we add something along the lines of coalesce(to:) to back up the ?? operator?


(Paul Ossenbruggen) #215

val name = node.getName() ?: throw IllegalArgumentException("name expected")

I really think we need something like this, this is more important to me than !! this would eliminate the need for most !s, and having a single line solution is important. In fact, I would like the ?: better than ?! for a similar feature because ! means danger, crashes ahead whereas, ?: means it is a safe operation, no !. (I know, adding operators are frowned upon, but these seem to be extensions of existing operators)

I am not objecting to !! though. I think Pavol Vaskovic had a very good description The ! or !! I believe are better than .unwrap or .unwrap(reason) because you can tell people ! means danger, a potential crash. Just like the triangle signs with a ! on the edge of a cliff. The consistency of this is what is appealing to me and unwrap does not indicate it could fail (unless I am misunderstanding the unwrap suggestion). Either way, people need to feel that danger awaits and the !! says that loudly.


(Paul Ossenbruggen) #216

I get this but the way, with newer programmers I have seen ! use is just to get the code to compile, they are not thinking about, "they expected something and got nothing". I did a survey of a fairly big app at work, and by far, the number one culprit for crashes was implicitly unwrapped and force unwraps. And, we found that they can lurk for a long time before they cause crashes. People will also put them on classes when they don't need to. It is an issue of not understanding, but it is often hard to tell whether the person did it on purpose or they were in a hurry to get it to compile. !! makes it clearer when it is on purpose.


#217

I'm not sure if introducing !! will help with this problem. If they are not thinking about what they expect, this operator won't make any change, exclamation marks will change more likely to let y = x !! "Xcode force me". If we are talking about Swift newcomers, better solution just for this would be replace ! fixit with ??.


(Jon Hull) #218

@Chris_Lattner3 I am curious what you think of the option/idea involving commenting syntax //!:

let x = optX!  //! This comment would appear in the console/log if it traps

In my mind, the advantages are:

  • It also works for as!, try!, subscripts, etc...
  • It is very lightweight: just as compact as !! "Message"
  • The "messages" from all of these options/ideas really are comments in a sense
  • It can apply to an entire chain if desired (or you can split a chain across multiple lines and comment each one if desired)

(Brent Royal-Gordon) #219

Which to a more experienced programmer (possibly the same person a few months later) will make them stand out as likely bugs, while let y = x !! "while loop should have found a value" will pass muster.


(Chris Lattner) #220

The core team met to discuss this. I had to leave a few minutes early, so I don't know what the conclusion of that discussion was, but I'd suggest waiting to see what advice/thoughts/opinion comes out of that.

-Chris


(Marc Palmer) #221

I like this proposal in terms of what it tries to achieve, but would strongly favour something that is not !!.

My rationale is one of accessibility - I don't know if any of the hundreds of other comments has mentioned this - but I'm 45, and my vision is OK, I don't need glasses. But depending on size and font it is very easy to misread:

let a = x !! "Woah"
let b = x || "Woah"

This is IMO a serious concern for readability.


(Joe Groff) #222

Proposal rejected

The core team has decided to reject this proposal as written. However, the core team concurs that the motivating problems posed by the proposal are important to solve, as did an overwhelming majority of commenters who participated in the public review. The fact that the only fixit the compiler offers to unwrap an Optional is to use the ! operator is an unfortunate legacy of the Swift 1.0 days, before anything in the SDKs Swift was designed to work with had been audited for nullability, so force-unwrapping was far more of a necessity. Nowadays this legacy is actively harmful, and encourages bad habits in new Swift programmers, as the proposal and review discussion highlight extensively. It is clear too that ! giving inadequate runtime feedback is a major problem, since a large contingent of the Swift community follows style guides that flat-out ban it, and the guard ... else { fatalError("message") } idiom is widespread as a way of more thoughtfully crashing on nil with an actionable error message.

All that said, there isn't a clear case that adding a new operator to the standard library is the best way to address these concerns. Many issues with the proposed design were raised in the review, including:

  • Adding operators, in particular operators without term of art precedent, in general has a high bar. Operators, while compact and convenient to use once learned, are harder to learn, being more difficult to search for, talk about in speech, or query in IDEs than named functions.
  • !! suggests an expansion of the taxonomy of ?/! operations that doesn't pay off—if one thinks of it as an extension of ! with a message, it suggests the existence of try!! and as!! variants with messages too.
  • Syntactically, !! doesn't work as fluently as !. Being an infix operator, it would require parens to chain additional method calls onto the unwrapped value, as in (x !! "message").foo().
  • !! is visually similar to ||.

The core team believes that this entire discussion would be much different if the tooling around the language were improved independent of language evolution. It is clear that the ! fixit is harmful and counterproductive; that must be fixed in the compiler independent of the language design. The compiler shall instead offer a choice of fixits, including at least the possibilities of x ?? <#default value#>, x? (where that is possible), and x! (with a note to use it only when certain the value can't be nil in the fixit note). Doug is working on implementing this improvement. Changing the fixit will make it no longer possible for "fix all in scope" to thoughtlessly pepper code with force-unwraps, and it will make it plain to the user that they must think about what they need to do in response to the nil value at that point. The core team believes this will go far in mitigating the misuse and bad practices promoted by the fixit as identified by the proposal.

Meanwhile, elsewhere in swift-evolution, the core team has also accepted SE-0215 — Conform Never to Equatable and Hashable, and came out of that discussion strongly in favor of investigating making Never be a subtype of every other type. This requires its own design cycle, but it fills in a general gap in the type system, and would enable the idiom x ?? fatalError("message") without any additions to the standard library. This pairing of ?? with Never was also discussed thoroughly in review, and in turn has its own share of criticisms:

  • It has some of the same drawbacks as !! would: there's the same lack of syntactic fluency, requiring parens to chain with further method calls. Also like !!, there's no corresponding try?? or as?? operator.
  • It raises valid concerns for teachability. Although someone familiar with the language could reason about it from first principles, when taken as an idiom by a beginner, it appears to imbue ?? with an alternative meaning "crash with message", which is at odds with the general taxonomy of ?/! operations. Furthermore, since ?? would allow an arbitrary Never-returning expression on the right-hand side, it also presents a crisis of choice on what function to put on the right-hand side that is its own candidate for confusion—fatalError? preconditionFailure?
  • Although more compact than guard let ... else { fatalError("...") }, x ?? fatalError("...") does not reach the terseness possible from a dedicated unwrap-with-message operator or method.

Nonetheless, similar idioms are well precedented in other languages, such as Perl and Ruby's x or die, Rust's x.unwrap_or_else(|| return) (which was recently sugared as Rust's postfix ?), and Kotlin's x ?: throw, and the expressivity of the idiom is well-liked in those communities. The review discussion raised the possibility of adding an overload ??<T>(_: T?, _: Never) -> T as a stopgap, but in experimenting with this solution, we found that overloading had unacceptable impact on type-checking performance, and adding the overload presents a non-zero risk that, if making Never be a universal subtype does not end up happening for whatever reason, then the overload on ?? would be left as a strange special case in the standard library.

The core team does not think this has to be the last word on this subject, but we would like to see how usage of the language in practice changes once the steps of improving the optional unwrapping fixits and implementing implicit conversions for Never are taken. If there's still widespread confusion and misuse of optionals, we are willing to reevaluate.

Thank you to everyone who participated in the review!


Another try at allowing optional iteration
[Rejected] SE-0217: The Unwrap or Die operator
(Ben Cohen) #223