SE-0231 — Optional iteration

Yeah, this was one of the possibilites that came up, which I listed in my summary. I added a link to the summary into the original post to hopefully make this more discoverable.

Interesting. It makes me wonder why Sequence.reversed() returns an array in the first place. Why not return a Sequence?

The issue AIUI is that there are overloads that do both, and ?? [] contextually leaves only the array-producing form as a possibility.

2 Likes

The pattern .+ in .+ \?\? \[\] produces only 1 true positive:

for e in n.geometry?.elements ?? [] { … }

and many false positives such as:

for set in a1 { x4.formUnion(set ?? []) }

and

let objs: [TagItem] = selectedTagName.map { selTagName in document.catalog.tagItems.filter { $0.name == selTagName } } ?? []

in our 200k line codebase.

And the pattern if[[:space:]]let[[:space:]]*\w+[[:space:]]*=[[:space:]]*.*[[:space:]]*\{[[:space:]]*for produces zero matches. I had to double check by removing the last "for" which resulted in a lot of matches so for some reason our code base doesn't have a single occurrence of if let a = b { for.

1 Like

In just over 100K lines of Swift, excluding empty lines and comments (cloc); I found 5 instances using @algal's regex for:

  • if let ... for ...
  • for ... in ...?? []

I still don't see the driving need for this; re a few of these instances could fairly easily be avoided.

2 Likes

I only see one returning Array. I mean if Sequence.reversed() returns the same Sequence type(Self)—which feels very natural—the problematic code won't compile at all. There must be good reasons that the current API returns Array instead of Self(same for sorted()), what are they?

To reverse something that doesn't provide iteration from the last to first element (not just sequences, but forward-only collections as well), you must stream all the elements into some kind of storage first, then return them in reverse order. If you're going to do that, it may as well be an array because having an array is a useful thing to have. You could instead return some kind of opaque reversed thing, but that wouldn't help – you still have to allocate memory for all those elements.

3 Likes

Re your discussion with @Ling_Wang
That trap seems very specific to the combination of .? and ?? []. Neither of the following IMO would result in this:

  • Rust's unwrap_or
  • Java's orElse
  • Scala's getOrElse

...and I certainly can't easily recreate that with a similar extension method.

Range conforms to BidirectionalCollection which contains a default implementation as well.

I get the ubiquitousness and usefulness of Array. But any Sequence can be easily converted to Array. Anyway I believe that's not the reason Sequence.reversed() can't return Self. One reason I can think of is some Sequences are not reversable, for example, lowerBound must be less that upperBound for Range so Range is irreversable as a Sequence. Is that the only reason though?

Yes, it's pretty specific to Swift, and how type checking works in conjunction with overloaded functions returning different types. It's not completely specific to this case though. There are similar unfortunate overload choice situations without involving optionals or array literals.

I feel like we should probably take some of these conversations about these specifics to one of the user or dev forums, rather than clutter the review thread. For review purposes, it's enough to establish the there's a significant downside to the ?? [] solution, that several alternative solutions (both language and library level) don't suffer from.

3 Likes

Note, though, I offered a potential ?? overload that removes the downside (leaves lazy sequences lazy). I only repeat this because this thread has been so busy today, it’s easy to miss things.

Overloading ?? is a non-starter. A recent prototype overloading it for the Never case showed adding a third overload to that operator can have catastrophic consequences for type checker performance. These kinds of overload are also really problematic as they lead to unexpected consequences from edge cases that can cause things like user-inexplicable type checker diagnostic when they make a typo or type error in their code.

At several points in this thread there seems to be an underlying assumption that solutions achieved in the library are unambiguously better than solutions by changing the compiler. While that's often the case, this is not a universal rule. Over the life of Swift, we have had occasions in the past where something has been done in the standard library, often with overloading tricks, in lieu of a neater solution in the compiler. This frequently turns out to be a mistake that we need to unwind. And bear in mind that, after ABI stability, things in the standard library will be harder to unwind.

5 Likes

My 10,000-foot view of the topic is that, fundamentally, one major reason that optional types help us to write better code is that they provide a speed bump for users to think about the nil case. This necessarily introduces, like real-life speed bumps, interruptions that decrease fluidity.

My hunch is this, that there will never be a point at which optionals will not prompt users to complain about lack of fluidity; that is, a perfectly fluid design for optionals is one in which there are no optionals at all, only null pointers.

Therefore, I believe there needs to be some sort of consensus that in Swift there will be so much fluidity and no more. Otherwise, what we will have here is a true slippery slope where each successive idea proposed about adding convenience to the use of optionals erodes further their role in helping to prompt readers and writers of code to slow down and think about the nil case.


Edit to expand: Consider the common critique that too many novice users use the fix-it to add '!' without truly understanding the implications of unwrapping or considering the nil case. When I see the principle enunciated by some that "wherever I use '!' I should be able to use '?'," I fear that if adopted the takeaway for some users (certainly not those on this forum, of course) would be, "I should be able to get all the convenience of not having to think about unwrapping provided by '!' with none of the crashiness." And that sounds an awful lot like null pointers.

What seems palatable about optional chaining is that it's optional in, optional out. Yes, it is true that when using forEach, a method that's not pure, one can avoid considering how to deal with the nil case by using optional chaining. But that's rather a specific use of a general tool. By contrast, this proposal is solely about allowing a single character operator to cause a for loop to take an optional in without giving an optional out.

15 Likes

Out of curiosity, how many lines of code is the compatibility suite in total? Other folks are citing (e.g.) 5 instances per 100KLOC, which is 0.005%. I'm curious how these 20 examples compare.

If the reality is that low, I think we need to consider whether the cost in language complexity is worth a very few lines of improved code.

On balance, I don't think we should implement this.

To add in my stats, I found six references of the for _ in _ ?? [] formulation across two (UIKit) app codebases. 5/6 of these were iterating over UITableView.indexPathsForSelectedRows which returns nil instead of zero elements. If the imported library had shaken off its Objective-C heritage, there would only be one other instance of the optional iteration pattern under discussion.

I know that I will have structured the app to avoid optionality, with guard statements, etc, and occasionally small helper functions that check optionality preconditions before getting to a for loop in the first place. Probably some of these would have been more elegantly expressed with a dedicated optional-silencing for syntax. It's hard to see it as a common-enough problem though, overall.

@Ben_Cohen's reversed() ?? [] array example highlighted a significant downside in the standard terse technique, so it seems like there should be something in the Swift syntax to deal with these cases. Perhaps better documentation to use guard or if let is enough. Selfishly, based on occurrences in my own projects, I do not see justification for a special sugar here.

That being said, if we were to go forward, the postfix ? is what I would prefer to the for? spelling in the proposal.

I don't know if counting the number of occurrences of the direct workarounds in the current codebase is a fair representation if the usefulness of optional iteration. In several places I represent the empty state in collections with and empty instance rather than nil just because of the lack of this. And finding instances where I might have used optional collections had there been optional iteration is significantly more time consuming and difficult than a regex search.

3 Likes

Now I see a trend to avoid nil collections in for-loop in my Swift code unlike my ObjC code. And not because it is impossible to use it inside for-loop without ?? []. I use if let or guard to prevent such cases as soon as possible. Also I try to translate optional to non-optional as soon as possible also.

Why array сan be nil or why we can not initialize it as empty array right away without optional use?

  • May be nil is indication of error? We see that Swift go for explicity errors processing. Then we should process nil explicity and any syntax sugar will simply «fix it» without any thinking through (even try? require thinking over). It is dangerously in hands of unexperienced developer. And why in case of error we can not return just empty array instead of nil? In any case nil as error not best solution when it is not really necessary especially without try or another error approaches.
  • May be nil is indication of some code sections/entities was not initialized yet and now it is time to complete it? But again we should not want to just «fix it» because this code leads to break class invariant and any «fix it» can lead to code inconsistency.
  • nil and empty is synonyms? It is more relevant for ObjC but not Swift.

Yes, there are cases out of that three ones (especially in case of non-collection types) but now I think that we can resolve most of proposal issues via Swift optionals evolution in more fundamental ways other than just add ? to one more keyword. Most of optionals use cases was translated from ObjC and Cocoa, but what if we could decrease optionals usage in every-day code? For example, we can add safe out of init non-optional properties initilization (for example, in iOS viewDidLoad or awakeFromNib are often synonyms of init) and it will allow us to avoid optionals in code. We can promote to not use nil to indicate error for collections and use empty collections instead. We can also promote that nil as synonym of empty is bad solution. We can promote to translate optionals to non-optional values as soon as possible. And may be proposal use cases will become rare.

I’m not opposite to this proposal, this can really help is some cases and may be for let is good alternative to if let and guard. But I think that we can have more suitable solutions that can just prevent optionals to reach for-loop places in code. Still code without optionals is more maintainable.

I'm generally skeptical when it comes to polls or collecting stats in reviews, and this one is no exception - I think we should be very careful not to overrate the results:

  • There's no regex that will catch all situations where the proposal would improve the situation: guard let is hard to detect, and even a small // The array isn't nil, we can iterate it could hide a match
  • It's extremely easy to introduce bias... I hope no one would actually try to manipulate the statistics, but would everybody publish numbers that harm his standpoint, or spend extra time to improve an analysis that already suits him?
  • If we did a thorough investigation, I bet would find lots of features that are implemented, but almost never used... so where is the bar for significance?

That said, here are the results from the biggest (non-public) Swift codebase I could access directly, and that is in big parts not written by myself:

github.com/AlDanial/cloc v 1.80  T=17.24 s (324.5 files/s, 25666.7 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Swift                           379           5635            881          22960
--------------------------------------------------------------------------------

rg --stats -F "?? []" .

16 matches
16 matched lines
13 files contained matches
14463 files searched
2256 bytes printed
24234148 bytes searched

9 out of those 16 matches are true hits, temporary variables haven't been used in this codebase.

rg -U 'if[\space]*let[\space]*[\word]+[\space]*=[\space]*[word]*.*\n[\space]*for' .
found three uses of if let/for in

Of course, those numbers fully support my standpoint ;-)

1 Like

It seems that it would also be useful to know how many total for _ in loops there are in the codebase, as @michelf provided. If it's 5 loops over optional sequences in 100,000 LOC that seems mighty low, but if there are just 6 total for in loops in those 100,000 lines, 5 of which are over optional sequences, that tells a different story than 5 out of 100 loops in 100,000 LOC.

2 Likes
Terms of Service

Privacy Policy

Cookie Policy