SE-0231 — Optional iteration

[dennisvennink]

@Ling_Wang

I only see one returning Array.

Range conforms to BidirectionalCollection which contains a default implementation as well.

[Ling_Wang]

I get the ubiquitousness and usefulness of Array. But any Sequence can be easily converted to Array. Anyway I believe that's not the reason Sequence.reversed() can't return Self. One reason I can think of is some Sequences are not reversable, for example, lowerBound must be less that upperBound for Range so Range is irreversable as a Sequence. Is that the only reason though?

[Ben_Cohen]

@endofunk

That trap seems very specific to the combination of .? and ?? []

Yes, it's pretty specific to Swift, and how type checking works in conjunction with overloaded functions returning different types. It's not completely specific to this case though. There are similar unfortunate overload choice situations without involving optionals or array literals.

I feel like we should probably take some of these conversations about these specifics to one of the user or dev forums, rather than clutter the review thread. For review purposes, it's enough to establish the there's a significant downside to the ?? [] solution, that several alternative solutions (both language and library level) don't suffer from.

[gregtitus]

@Ben_Cohen

For review purposes, it's enough to establish the there's a significant downside to the ?? [] solution, that several alternative solutions (both language and library level) don't suffer from.

Note, though, I offered a potential ?? overload that removes the downside (leaves lazy sequences lazy). I only repeat this because this thread has been so busy today, it’s easy to miss things.

[Ben_Cohen]

@gregtitus

Note, though, I offered a potential ?? overload that removes the downside (leaves lazy sequences lazy). I only repeat this because this thread has been so busy today, it’s easy to miss things.

Overloading ?? is a non-starter. A recent prototype overloading it for the Never case showed adding a third overload to that operator can have catastrophic consequences for type checker performance. These kinds of overload are also really problematic as they lead to unexpected consequences from edge cases that can cause things like user-inexplicable type checker diagnostic when they make a typo or type error in their code.

At several points in this thread there seems to be an underlying assumption that solutions achieved in the library are unambiguously better than solutions by changing the compiler. While that's often the case, this is not a universal rule. Over the life of Swift, we have had occasions in the past where something has been done in the standard library, often with overloading tricks, in lieu of a neater solution in the compiler. This frequently turns out to be a mistake that we need to unwind. And bear in mind that, after ABI stability, things in the standard library will be harder to unwind.

[xwu (Xiaodi Wu)]

@gwendal.roue

What started as a humble convenience-oriented proposal has unexpectedly turned into many expressions of frustrations about optionals (for loops, optional chaining, now x?.count + 1 , etc).

If some people have concerns with optionals, have gathered pet peeves as time has passed, have untold long-term plans that were on hold, it's time to speak out loud and clear, for the benefit of the whole community.

I'm now waiting for some kind of "Optional Fluidity Manifesto" that would provide a high-level view of future expected changes in the language and the standard library, and make sure that they do improve the language, and that they are internally consistent.

My 10,000-foot view of the topic is that, fundamentally, one major reason that optional types help us to write better code is that they provide a speed bump for users to think about the nil case. This necessarily introduces, like real-life speed bumps, interruptions that decrease fluidity.

My hunch is this, that there will never be a point at which optionals will not prompt users to complain about lack of fluidity; that is, a perfectly fluid design for optionals is one in which there are no optionals at all, only null pointers.

Therefore, I believe there needs to be some sort of consensus that in Swift there will be so much fluidity and no more. Otherwise, what we will have here is a true slippery slope where each successive idea proposed about adding convenience to the use of optionals erodes further their role in helping to prompt readers and writers of code to slow down and think about the nil case.

---

Edit to expand: Consider the common critique that too many novice users use the fix-it to add '!' without truly understanding the implications of unwrapping or considering the nil case. When I see the principle enunciated by some that "wherever I use '!' I should be able to use '?'," I fear that if adopted the takeaway for some users (certainly not those on this forum, of course) would be, "I should be able to get all the convenience of not having to think about unwrapping provided by '!' with none of the crashiness." And that sounds an awful lot like null pointers.

What seems palatable about optional chaining is that it's optional in, optional out. Yes, it is true that when using forEach, a method that's not pure, one can avoid considering how to deal with the nil case by using optional chaining. But that's rather a specific use of a general tool. By contrast, this proposal is solely about allowing a single character operator to cause a for loop to take an optional in without giving an optional out.

[Chris_Lattner3 (Chris Lattner)]

@Ben_Cohen

I did a quick check of the compatibility suite . It was a simple regex + eyeball so I think it avoids false positives but will probably be off +- a few.

I could see 20 examples of for with ?? [] and 55 examples of if let immediately followed by use in a for . Overall 24 projects used one of these patterns, so just over a quarter of the projects.

Out of curiosity, how many lines of code is the compatibility suite in total? Other folks are citing (e.g.) 5 instances per 100KLOC, which is 0.005%. I'm curious how these 20 examples compare.

If the reality is that low, I think we need to consider whether the cost in language complexity is worth a very few lines of improved code.

[bzamayo (Benjamin Mayo)]

On balance, I don't think we should implement this.

To add in my stats, I found six references of the for _ in _ ?? [] formulation across two (UIKit) app codebases. 5/6 of these were iterating over UITableView.indexPathsForSelectedRows which returns nil instead of zero elements. If the imported library had shaken off its Objective-C heritage, there would only be one other instance of the optional iteration pattern under discussion.

I know that I will have structured the app to avoid optionality, with guard statements, etc, and occasionally small helper functions that check optionality preconditions before getting to a for loop in the first place. Probably some of these would have been more elegantly expressed with a dedicated optional-silencing for syntax. It's hard to see it as a common-enough problem though, overall.

@Ben_Cohen's reversed() ?? [] array example highlighted a significant downside in the standard terse technique, so it seems like there should be something in the Swift syntax to deal with these cases. Perhaps better documentation to use guard or if let is enough. Selfishly, based on occurrences in my own projects, I do not see justification for a special sugar here.

That being said, if we were to go forward, the postfix ? is what I would prefer to the for? spelling in the proposal.

[mss (Srihari)]

I don't know if counting the number of occurrences of the direct workarounds in the current codebase is a fair representation if the usefulness of optional iteration. In several places I represent the empty state in collections with and empty instance rather than nil just because of the lack of this. And finding instances where I might have used optional collections had there been optional iteration is significantly more time consuming and difficult than a regex search.

[Alexey (Alexey Kravchenko)]

Now I see a trend to avoid nil collections in for-loop in my Swift code unlike my ObjC code. And not because it is impossible to use it inside for-loop without ?? []. I use if let or guard to prevent such cases as soon as possible. Also I try to translate optional to non-optional as soon as possible also.

Why array сan be nil or why we can not initialize it as empty array right away without optional use?

• May be nil is indication of error? We see that Swift go for explicity errors processing. Then we should process nil explicity and any syntax sugar will simply «fix it» without any thinking through (even try? require thinking over). It is dangerously in hands of unexperienced developer. And why in case of error we can not return just empty array instead of nil? In any case nil as error not best solution when it is not really necessary especially without try or another error approaches.
• May be nil is indication of some code sections/entities was not initialized yet and now it is time to complete it? But again we should not want to just «fix it» because this code leads to break class invariant and any «fix it» can lead to code inconsistency.
• nil and empty is synonyms? It is more relevant for ObjC but not Swift.

Yes, there are cases out of that three ones (especially in case of non-collection types) but now I think that we can resolve most of proposal issues via Swift optionals evolution in more fundamental ways other than just add ? to one more keyword. Most of optionals use cases was translated from ObjC and Cocoa, but what if we could decrease optionals usage in every-day code? For example, we can add safe out of init non-optional properties initilization (for example, in iOS viewDidLoad or awakeFromNib are often synonyms of init) and it will allow us to avoid optionals in code. We can promote to not use nil to indicate error for collections and use empty collections instead. We can also promote that nil as synonym of empty is bad solution. We can promote to translate optionals to non-optional values as soon as possible. And may be proposal use cases will become rare.

I’m not opposite to this proposal, this can really help is some cases and may be for let is good alternative to if let and guard. But I think that we can have more suitable solutions that can just prevent optionals to reach for-loop places in code. Still code without optionals is more maintainable.

[Tino]

I'm generally skeptical when it comes to polls or collecting stats in reviews, and this one is no exception - I think we should be very careful not to overrate the results:

• There's no regex that will catch all situations where the proposal would improve the situation: guard let is hard to detect, and even a small // The array isn't nil, we can iterate it could hide a match
• It's extremely easy to introduce bias... I hope no one would actually try to manipulate the statistics, but would everybody publish numbers that harm his standpoint, or spend extra time to improve an analysis that already suits him?
• If we did a thorough investigation, I bet would find lots of features that are implemented, but almost never used... so where is the bar for significance?

That said, here are the results from the biggest (non-public) Swift codebase I could access directly, and that is in big parts not written by myself:

github.com/AlDanial/cloc v 1.80  T=17.24 s (324.5 files/s, 25666.7 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Swift                           379           5635            881          22960
--------------------------------------------------------------------------------

rg --stats -F "?? []" .

16 matches
16 matched lines
13 files contained matches
14463 files searched
2256 bytes printed
24234148 bytes searched

9 out of those 16 matches are true hits, temporary variables haven't been used in this codebase.

rg -U 'if[\space]*let[\space]*[\word]+[\space]*=[\space]*[word]*.*\n[\space]*for' .
found three uses of if let/for in

Of course, those numbers fully support my standpoint ;-)

[wneumann]

It seems that it would also be useful to know how many total for _ in loops there are in the codebase, as @michelf provided. If it's 5 loops over optional sequences in 100,000 LOC that seems mighty low, but if there are just 6 total for in loops in those 100,000 lines, 5 of which are over optional sequences, that tells a different story than 5 out of 100 loops in 100,000 LOC.

[JoeyKL]

Hope I’m not being annoying, but can someone explain what advantages the proposal has over this method? Feel as though I’m missing something.

[Jens]

@JoeyKL

extension Optional<S> where S: Sequence { func orEmpty() -> AnySequence<S.Element> { switch self { case nil: return EmptyCollection() default: return self } } } let nilAsCanBe: [Int]? = nil for i in nilAsCanBe.orEmpty() { // ... }

Can you please provide an implementation that compiles (with Xcode 10)?

Also search this topic for "AnySequence" in order to see some related previous discussion.

[JoeyKL]

Oops, fair enough. Here's what I meant:

extension Optional where Wrapped: Sequence {
    func orEmpty() -> AnySequence<Wrapped.Element> {
        switch self {
        case nil:
            return AnySequence(EmptyCollection())
        case .some(let s):
            return AnySequence(s)
        }
    }
}

Apologies, I was working from my phone.

[Tino]

@JoeyKL

why can't we just add .orEmpty() like Kotlin has?

It has been mentioned and discussed - my iPhone won‘t let me make quotes now, but you can search for orEmpty.
One of the downsides is that it either needs a special type (and really small overhead), or AnySequence (with less small penalty).

Back at home - update:

@Joe_Groff

Introducing an orEmpty method on Sequence that produces a wrapper

(but looking for AnySequence might indeed be the best way to get an overview)

[gwendal.roue (Gwendal Roué)]

@JoeyKL

Hope I’m not being annoying, but can someone explain what advantages the proposal has over this method? Feel as though I’m missing something.

The best answer so far to your question (about orEmpty) is this paragraph:

@Ben_Cohen

At several points in this thread there seems to be an underlying assumption that solutions achieved in the library are unambiguously better than solutions by changing the compiler. While that's often the case, this is not a universal rule. Over the life of Swift, we have had occasions in the past where something has been done in the standard library, often with overloading tricks, in lieu of a neater solution in the compiler. This frequently turns out to be a mistake that we need to unwind. And bear in mind that, after ABI stability, things in the standard library will be harder to unwind.

[Joe_Groff]

Not to speak for @Ben_Cohen, but I think he was specifically speaking to the solution of overloading ??, which could have some rather severe side effects. A separate orEmpty method would have fewer systemic effects.

[Ben_Cohen]

@Joe_Groff

Not to speak for @Ben_Cohen, but I think he was specifically speaking to the solution of overloading ?? , which could have some rather severe side effects. A separate orEmpty method would have fewer systemic effects.

Yeah – I was mainly warning against "tricks" in the library to resolve the issue. An orEmpty method would be ok.

However, I also think in this case (though not all cases) you should think of the library and the compiler as one and the same. I don't think a library solution to this specific use case is somehow fundamentally better or more principled than changing the compiler to allow optional chain iteration with for.

It's also worth considering that orEmpty might end up leaning heavily on the optimizer to hoist the nil check up from every iteration to before you start iterating. This won't happen without specialization. Whereas a compiler solution that rewrites an optional chain into an if would be guaranteed to happen. Hard to say if that extra branch is material in practice though, without prototyping both and benchmarking.

[ebg]

I haven't seen a compelling reason why this needs any syntax change at all, certainly the proposal itself doesn't have a detailed case against it.

It's already understood that with an empty array let arr = [Int](), a for-in loop won't execute. It would seem almost obvious to a programmer that for a nil optional array, let arr: [Int]? = nil, a for-in loop also won't execute.

If anything, introducing for e in arr? is making things more confusing than no syntax change, because it becomes inconsistent when we introduce optional chaining: for e in obj?.arr