Optionals cause more problems than they solve

Evolution Discussion
1

I've used a lot of different languages over the years and I've never seen such a fundamental language feature cause so many problems. Just search this forum for optionals to see the convoluted discussions people have about handling complex data structures with optionals and consider people working on projects having to deal with this.

I understand the motivation to try and make it harder to for people to accidentally put in nil values but the priority of a good development language is to make easily readable, easily maintainable and easily portable code. Putting question marks and exclamation marks (which are typically used for other purposes) all over a codebase with conditional assignments is a complete mess.

I just tried to upgrade a basic project in Swift after using some forced optional unwraps to avoid having to deal with as many cascading optional handlers and now get errors like the following:

asyncProgressHandler:((HTTPProgress!) -> Void)? = nil
Using '!' is not allowed here; perhaps '?' was intended?

It's torture having a compiler constantly barking at you about these things that aren't important for most projects. I deal with nil types all the time in other languages. Here's how it's done:

let obj = JSON.decode(json);
if (obj.value != nil) {
doSomethingWith(obj.value);
}

Clear for everyone, no confusing syntax, no having to look up what ?? or !? means. Maybe enterprise development benefits from all the clutter but I'm not seeing the value in it at all. Quite frankly, at this point, I'd rather develop an entire app in Typescript and have it transpiled to Swift if needed.

Are other developers actually seeing the benefit from optionals or are you also having to read through endless blog articles or forums on how to solve basic problems with them that are no issue at all in pretty much every other language?

It's such a waste of time trying to deal with this.

If people do see the benefit, that's great but can we at least get something like a strict or non-strict mode so the rest of us don't have to deal with it? I don't want to have to deal with it at all, I want to write code quickly without this distraction.

A non-strict mode would allow writing code the 'old-fashioned' way as in, not forcing people to use optional declarations or checks. Yes this means this software will potentially be unstable and have to be fixed by running it and checking it (the horror) but it also means you can copy/paste code between languages much more quickly and develop projects more quickly.

If people feel like optionals have been beneficial can you provide some stats? Has it reduced production bugs for you and at what rate? Has it come at the expense of production speed vs other languages? Has it noticeably increased code bloat or not? Do you feel like you'd be worse or better off developing the typical way without them?

1 Like
2

Optional/Maybe types are one of the best programming language innovations to enter the mainstream in the last decade, and Swift is the first language I’ve used that made them truly ergonomic and fluent.

It’s worth keeping in mind that familiarity bias accounts for almost all of a human being’s initial reaction to a language feature. “Confusing” is more a function of a developer’s past experience than of a language’s inherent virtue. (Java’s smartest marketing decision was using curly braces so that it look like C and gave people warm familiarity fuzzies.)

I recommend working through the familiarity bias against Optionals, and embracing the language’s idioms. I’m glad I did. Yes, they catch bugs; they also have helped me understand APIs I use and write clearer code. I now miss them when working in other languages.

57 Likes
3

I feel that Optional actually requires a lot less thinking once one gets used to it. I can just let the compiler tell me you forgot the nil case, and I just go ahhh, thanks.

It is part of the Swift direction not to be dialectal, where the code can compile in one mode but not in another. I'd say it's very unlikely.

19 Likes
4

Thanks for the discussion! FWIW, you compared two very different things between your compiler-error code snippet and your short JSON parsing snippet.

You said other languages deal with nil easily this way:

let obj = JSON.decode(json);
if (obj.value != nil) {
    doSomethingWith(obj.value);
}

Given a JSON parsing library that is just as simple as the above (no error handling, only returns nil on failure) then Swift is no less terse:

let obj = JSON.decode(json)
if let value = obj.value {
    doSomethingWith(value)
}

Of course once error handling and other cases are added in, things get more complex. I just wanted to point out that the example given in argument for simplicity is quite similar in Swift. IMO Swift does shine quite nicely once the problems get more complex than the above.

As to the compiler error itself...

var asyncProgressHandler: ((HTTPProgress!) -> Void)? = nil
// compiler error: Using '!' is not allowed here; perhaps '?' was intended?

When using !, you are telling the compiler to assume that the value will never be nil (and trap if so). It makes sense for properties or local variables that can't be populated immediately at the moment they are declared, but you are assuring the compiler that they will be populated with a non-nil value by the time they are read from.

Using ! on a type in a function parameter doesn't make sense because if you wanted to tell the compiler that the value of that type will not be nil, then just use the type name. The compiler thinks that you meant ? presumably because offering an optional parameter is a common thing to do. Granted, that error diagnostic isn't very helpful. Since the compiler already knows that a type specified in a function signature doesn't need the !, it could definitely offer a better explaination.

Regardless, if you write it this way...

var asyncProgressHandler: ((HTTPProgress) -> Void)? = nil

Then the compiler knows for a fact that whatever asyncProgressHandler comes along must be called with a non-nil HTTPProgress value. Callers attempting to do otherwise wouldn't even compile. It would be completely redundant to check for nil inside these handlers. I even assume the compiler would optimize it out!

14 Likes
5

I found Optionals confusing at first. I came to the realization that what they do for me is force me to write the value checking code my professors used to insist was needed in "professional code".

Where they started to shine was when I created a large struct which is meant to reflect a backing store in an SQL database. Suddenly I was thinking about SQL schema, and the need for the NULL/NOT NULL distinction leapt out at me.

1 Like
6

Thanks for all your replies.

You're right that it takes time to get used to new ways of doing things when you have done them a certain way for a long time. The comparison with Java and C code formatting is a good example and Apple did something similar with Objective-C switching the square brackets. This familiarity is important to productivity, code maintenance and portability and likely contributed a lot to Java's popularity. I'd like to see Swift become as popular a language as Java.

Having these safety features are useful when they are needed but hinder productivity when they are always required. Code linters are the same. They are nice to maintain neat code but when you need to meet a deadline, it's quicker to turn them off, get the job finished and do the cleanup after.

I agree it can be helpful on occasions where there's a clear problem and a clear fix, usually on single variables. The main problem I've seen is when dealing with 3rd party code and trying to access nested properties where the compiler will say that I can't access one property as it's not unwrapped and recommend a '?' and then I think hey that's nice but when I add that, it then goes 'ah but now that's an optional, you also have to do this other one' and then I fix it and then it does the same thing somewhere else and it just cascades through the codebase and it does this constantly, interrupting my workflow as I work on a project.

When you are in brainstorming mode and trying to get a project off the ground and get familiar with the language, it creates a roadblock to learning the language and productivity. You end up thinking about language design instead of the project. That's why languages like Typescript, Python and Javascript are so popular because they get out of the way and just let you work.

Thanks for the detailed reply. That code used to compile in Swift, I think it maybe doesn't now due to removal of implicit unwrapping in some places?

The main issue I have with optionals is not that it requires handling nil types, it's that it requires me to keep track of when something is optional and when it's not, that takes a lot of work on a big codebase because I have to trace back through the path that led to that call. There is an example on this page:

https://docs.swift.org/swift-book/LanguageGuide/OptionalChaining.html

if let beginsWithThe =
john.residence?.address?.buildingIdentifier()?.hasPrefix("The") {
if beginsWithThe {
    print("John's building identifier begins with \"The\".")
} else {
    print("John's building identifier does not begin with \"The\".")
}}

When I work with things like JSON objects (or databases as @Hacksaw mentioned), the entire data object is untyped and has to be mapped onto a type-checked structure at runtime.

Wouldn't it be easier in code to just assume that everything can be nil? Why write the following:

if let beginsWithThe = john.residence?.address?.buildingIdentifier()?.hasPrefix("The") {

instead of:

let beginsWithThe = john.residence.address.buildingIdentifier().hasPrefix("The")?

and have the compiler warn that this is potentially a nil value and that it should be checked with a conditional or even require a default assignment to a non-nil type?

This would be similar to try/catch error handling. When I develop projects, I normally either assume that everything can be nil at some point or have a confidence that something will always be non-nil and if not, something went wrong.

Swift's strictness takes away my control over that and while it might make the code safer, it slows me down and it encourages me to force unwrap things just to get the project to build.

That site links to a StackOverflow search for Swift crashes that has 60 pages of results.

I guess you could say that's evidence for why it's important to write the syntax correctly to avoid those crashes but the reason people do that is because it's not intuitive nor easy to maintain.

Examples of non-intuitive things are optional optionals:

"because the user is optional it uses optional chaining, and because getMessages() can throw, it uses try? to convert the throwing method into an optional, so we end up with a nested optional. In Swift 4.2 and earlier this would make messages a String?? – an optional optional string – but in Swift 5.0 and later try? won’t wrap values in an optional if they are already optional, so messages will just be a String?."

It's good that the language is evolving to simplify these things but why did it take 5 revisions to do that? The fact that someone ever thought that nested optionals was a good idea shows that nil-checking is being overthought. The language created this problem in the first place by having such strict adherence to nil-checking and is now trying to fix it in retrospect and breaking code.

It would have been better to start with no strict adherence to nil-checking and add the strictness where it was needed. Given that most languages don't offer this and most developers don't use it suggests it's not that big of a deal.

That's why I think having a strict/non-strict mode (like in Javascript) would make things a lot easier, especially in mixed language environments. Enterprise developers who absolutely need the strictness can require it in their code. Indie developers or multi-platform developers who need to keep code synced or kids just learning to code in school can turn it off. All this would do is treat everything as implicitly unwrapped and it can warn of mistakes instead of fail to build and run and can be done on individual classes, lines or files.

7

In your use case, does everything actually have to be optional, or is there some domain boundaries that you could use to remove the optionality so that the rest of the code doesn’t need optionals?

Can you simplify any of the long chains by using guard to bail out early?

Swifts optionals and explicit handling are part of what give me a lot of confidence as a swift user. Things that can potentially fail or cause a crash are visually very explicitly marked. I personally would not want to see a strict/non-strict mode as I think it would weaken one of the core strengths of the language.

13 Likes
8

It's hard to do this with nested and untyped runtime data that comes from things like JSON messaging and databases. This data is all held as strings and dictionaries. Any part of the hierarchy can be nil or even the wrong type.

I quite like that there's something to help keep code more stable but I think Swift's implementation causes a lot of complexity. I'd prefer a simpler implementation like a flattened/single-level nil-check for everything.

9

Swift values safety over convenience. In the long run, Optionals guide developers to write better code. Even if it's just because you have to decide what it means for something to be optional and whether it's really necessary in any given case.

A common example is an array property: empty versus absent. If your logic can treat them the same, why allow an absent (nil) array? It's better to initialize to an empty array and simplify the algorithm. A lack of special cases increases code maintainability. You have to think whether absence is semantically-distinct from empty.

Consider Java's approach. Accessing a nil reference is a runtime error. That requires you to litter your code with if-not-nil checks before using the reference. Any place you forget that check is a crash waiting to happen.

In Swift, it's also a runtime error, but Swift makes it harder to write code where that might happen by accident. You have to opt in to it by using implicit unwrapping or forced unwrapping at the call site. This makes Swift safer by design, and more bugs are avoided at compile time. If you don't use ! in your code, it's almost impossible to have a runtime crash related to Optionals. (Bridging to ObjC is a notable exception.)

1 Like
10

JSON processing would be a good example of some friction when dealing with untyped object in a typed language. The more restricted the language is, the more friction there would be. A Swift answer to that has been JSONDecoder, which admittedly does not apply to arbitrarily/unstructured JSON.

That aside, there are distinctions between a?.b?c and a?.b!.c, but users usually use a?.b?.c without much thought. Not that the consequence is devastating (about as much as missing ! opportunity), but it's something to think about.

11

Maybe it’s because I work mostly in the realm of iOS application development, but I haven’t seen the same friction. Using something like Codable to actually decode the response that you’re expecting from your json. On the other hand, it’s possible to decode arbitrary JSON with something similar to this.

Something from a database may come back as nil, but after you’ve loaded it before you persist it again, you should be able to have guarantees about which values are actually nullable and which are not.

Is there a more fleshed out code example that can illustrate the problem more clearly?

12

The part bolded is the main point. What you're saying is this is by design at the priority of safety over convenience. I get that but in practise, this isn't a big problem. On projects I work with, we put them through QA processes and get bug reports. The number of times we get reports about nil types causing crashes is a tiny fraction of the reports. It's usually the cause of 1-2 in 100 bugs.

I'm not finding the complexity caused by Swift's implementation of nil-checking (especially on nested types and untyped data) to justify the added safety that only fixes a couple of bugs.

What you're saying is correct about having to add nil checks in places in other languages but that's what Swift is doing too. The problems are caused due to this being required in all variable and function definitions and nested throughout the entire code.

Usually nil checks only need to be used when you do a value lookup. The developer is making the decision whether something can or can't be nil so they would know when to check for it and the compiler can warn them otherwise.

An example alternative implementation for nil-checking would be that any time a value is looked up, it is required to be declared as an optional or non-optional type and handled accordingly. That means adding a single different syntax vs other languages.

let stringObject:String
let arrayObject:Array

myFunction(parameter:String):ReturnType

These might be what they say, they might all be nil. It doesn't matter until someone tries to use them so the only requirement is a check at the point they are used not when they are defined and I don't feel like it should be a requirement for a successful build.

The compiler can warn and say that a lookup result might be a nil value and there's no conditional check. A developer is ultimately always responsible for the stability of their code in every language they use.

13

Swift does not require you to decide anything at the time of declaration other than "can this be nil?". If you don't know, why are you writing code in the first place? At the point of use, adding a ? to indicate that it's optional (good for the reader to know it might be nil) and that you don't care if it is, is hardly a burden, compared to the clarity it provides.

In Java, the answer to "may it be nil" is always yes. In Swift, the answer is "it depends". The fact that you can rule out a "yes" just by declaring a non-optional (the default) is a boon to understanding code. It is encoding an invariant into the type system, and that's good for writing correct code.

14 Likes
14

I’m not sure that the argument that your particular team has a good QA processes is a particularly strong arguments for weakening the invariants in the system. Swift catches these errors so that a QA team doesn’t have to.

14 Likes
15

This sounds good when you talk about simple cases. Just one variable, declare it optional, do a lookup. The problem comes when you scale up the complexity to class types and these are getting nested. Then you run into confusing scenarios where the 'correct code' is messy and hard to understand:

"With a doubly wrapped optional, the value held by the variable could be one of 3 things: Optional(Optional("some string")), Optional(nil) if the inner optional is nil, or nil if the outer optional is nil. So a ?? nil unwraps the outer optional. If the outer optional is nil, then ?? replaces it with the default value of nil. If a is Optional(nil), then ?? will unwrap the outer optional leaving nil. At this point you will have a String? that is nil if either the inner or outer optional is nil. If there is a String inside, you get Optional("some string").

Finally, the optional binding (if let) unwraps Optional("some string") to get "some string" or the optional binding fails if either of the optionals is nil and skips the block."

They even said there that casting flattens the optional chain so in some scenarios, this is giving people a false sense of security? The more complex the solution to something is, the more likely people use it incorrectly and then it causes its own bugs.

True but it's more to do with how Swift implements this and its requirement on everything than the benefits of nil-checking or not. Wouldn't it be easier if all nil-checking was done on a single level than adding question marks to every separate statement?

The entire design of optionals is to fix one issue:

let variable = something <- is this nil or not

All the syntax added to fix that simple issue seems to me like it's been over-designed and causes unneeded complexity as in the above example.

Most of the Swift code I've seen that is moderately complex is messy and hard to read compared to other languages.

16

Optionals are a fundamental part of Swift. The official “About Swift” page specifically calls them out:

Another safety feature is that by default Swift objects can never be nil , and trying to make or use a nil object will results in a compile-time error. This makes writing code much cleaner and safer, and prevents a common cause of runtime crashes. However, there are cases where nil is appropriate, and for these situations Swift has an innovative feature known as optionals . An optional may contain nil , but Swift syntax forces you to safely deal with it using ? to indicate to the compiler you understand the behavior and will handle it safely.

I don't know how to say it more delicately, but if you simply cannot abide the way Swift handles nil, you probably need to use a different programming language.

12 Likes
17

The ergonomics of nested Optionals has improved, and could be further improved. However, there aught to be a reason, at each level, for the Optional. In the vast majority of cases you don't care, and you just use if let to collapse them. When you do care, however, you have the option to check just what you need.

Would you rather write this:

if a != nil && a.b != nil && a.b.c != nil && a.b.c.d != nil {
   a.b.c.d.e()
}

Or this?

a?.b?.c?.d?.e()

This?

if a != nil && a.b != nil && a.b.c != nil && a.b.c.d != nil {
   let f = a.b.c.d.e // Is f nil or not?
}

Or this?

if let f = a?.b?.c?.d?.e {
  // f is not nil
}
8 Likes
18

Speaking of other programming languages, why haven't other language developers been rushing to implement the same thing?

In that example, I would rather write:

a.b.c.d.e()?

but it's not this part that's the problem as much as things like a.b.c.d != nil. The compiler will tell you things like you can't write a.b.c.d because b is an optional type so you have to write a.b?.c.d and then it will say but b hasn't been unwrapped so... and it does this all the time when you are writing code.

If I could just write a.b.c.d.e()? and then check if it's nil, I'd be perfectly happy with that.

19

Plenty of other languages have. Rust's pointers are always valid, and you need to use the Option type to represent a pointer which may be null. Even C++ added the std::optional type in C++17.

11 Likes
22

It's refreshing when somebody actually criticizes Swift here ;-), and there are more than enough fans to defend every aspect. But ...

That happened, and is happening: Kotlin, Ceylon, Julia... all modern (created recently) languages that come to my mind have optionals.
It's not always modeled in exactly the same way as Swift (there are designs which don't have the issue of nesting), but null pointer bugs are definitely considered to be harmful enough to warrant constructs to avoid them.

By the way, I wouldn't say that Optionals are a new feature — the innovation are objects that are not optional (guaranteed not to be nil).

9 Likes