SE-0231 — Optional iteration

I referred to orEmpty in the post. I think it is almost, but not quite, as good as the optional chaining solution, and would be my second choice.

It is clearly inferior though:

  • A projection runs the risk of not being optimized away properly. A manual if let wrapping the for loop never runs this risk. I am guessing the actual code generated for the optional chaining solution is more like the latter than the former.
  • It composes poorly with optional chaining: maybeRange?.reversed().orEmpty doesn't compile for reasons that are hard for users to understand.
  • It is inconsistent. Optional chaining of methods is just a flatMap, why not solve that in the library too? We have optional chaining for function/method/subscript invocation, and for assigning through optionals. Optional chaining composes well with while. Optional chaining doing nothing for for is the odd one out here.
  • There is also a mismatch between for x in a! but for x in a.orEmpty. Elsewhere, ! means trap on nil and ? means no-op on nil, and it should here too.

I don't buy that orEmpty is more discoverable. I don't think developers, faced with a compilation error because they are trying to for an optional, are going to look at the methods available on Optional for a solution. They are going to do what they know: unwrap it with an if let, or coalesce it. The way to drive discovery of this feature, either way we solve it, like you say, is with a fixit.

I also don't feel that orEmpty is clearer. I don't think anyone familiar with optional chaining and Swift in general won't guess what for x in d[k]? means when they see it. orEmpty doesn't even say what it does – we don't use the "or" idiom anywhere else in Swift.

10 Likes

@anthonylatsis Are you sure your nil coalescing pattern match is correct? I found several matches for “?? []” in the Swift repo when I was searching my various codebases.

When not mobile I hope to post my results. But I am mildly in favor of “for x in y?”. There needs to be an explicit indicator that the control flow may be skipped.

UPD

I've expanded my table with some open source Apple projects.


It's correct, but might not be covering as much cases as yours. Mind sharing the regex and searching tool so I can try it out?


While sorting through matches I realized that I had missed an important fact when I was advocating degenerate optional patterns. I use sequence? to exemplify, but forget that optional expressions come in all sorts, including those with keywords, like try? foo(...) or some as? Another. This really complicates things if we want to avoid parenthesizing, because every type of expression would need some kind of syntax, while the majority already express optionality pretty well on their own. So I guess this turns the scales for me in favor of optional chaining.

1 Like

MHO on your points:

This argument could be made against any algorithm, I don't consider this to be compelling.

I find this to be a very compelling point, and I didn't think about that. Thanks!

There are lots of reasons, I don't find this argument compelling.

? certainly does not mean "noop on nil", it is a monadic operator that performs an operation when non-nil and wraps the result in an optional, propagating a nil if present.

-Chris

5 Likes

I think it's problematic to make arguments based on comparison with other features, so just to get a feeling for the numbers, here's how many "repeat {" I found in the compatibility suite:

43 matches
43 matched lines
34 files contained matches
6114 files searched

Whereas optional iteration seems to be quite import when parsing JSON or XML, repeat was repeatedly ;-) used in tests and when dealing with sockets.

All matches
./JSQCoreDataKit/Source/Migrate.swift
135:    repeat {

./Guitar/Sources/GuitarPadding.swift
64:        repeat { copySelf.insert(token[token.startIndex], at: startIndex) } while copySelf.count < length
91:        repeat { copySelf.insert(token[token.startIndex], at: endIndex) } while copySelf.count < length

./Kingfisher/Sources/ImageDownloader.swift
593:        repeat {

./Perfect/Tests/PerfectLibTests/PerfectLibTests.swift
516:			repeat {

./Perfect/Sources/PerfectLib/Utilities.swift
63:		repeat {

./core/Sources/Bits/Byte+Digit.swift
57:        repeat {

./BlueSocket/Tests/SocketTests/SocketTests.swift
146:			repeat {
244:			repeat {

./BlueSocket/Sources/Socket/Socket.swift
1424:		repeat {
1537:		repeat {
2960:				repeat {
3508:		repeat {
3520:				repeat {
3677:		repeat {

./Deferred/Sources/Deferred/Atomics.swift
87:    repeat {

./ProcedureKit/Tests/ProcedureKitStressTests/ProcedureStressTests.swift
214:                    repeat {

./Kitura/Tests/KituraTests/TestRangeHeaderDataExtensions.swift
32:        repeat {

./JSQDataSourcesKit/Example/UITests/XCTestCase+Extensions.swift
102:        repeat {

./RxSwift/Tests/Microoptimizations/main.swift
24:repeat {

./RxSwift/RxSwift/Observers/TailRecursiveSink.swift
73:        repeat {

./RxSwift/RxSwift/Schedulers/VirtualTimeScheduler.swift
139:        repeat {
179:        repeat {

./swift-nio/Tests/NIOWebSocketTests/EndToEndTests.swift
57:    repeat {

./swift-nio/Tests/NIOHTTP1Tests/HTTPDecoderTest.swift
222:        repeat {

./swift-nio/Tests/NIOTests/TestUtils.swift
229:    repeat {

./swift-nio/Sources/NIO/PendingWritesManager.swift
465:            repeat {

./swift-nio/Sources/NIO/BlockingIOThreadPool.swift
125:        repeat {

./swift-nio/Sources/NIO/RecvByteBufferAllocator.swift
119:        repeat {

./Starscream/Sources/Starscream/Compression.swift
82:        repeat {
144:            repeat {

./Starscream/Sources/Starscream/WebSocket.swift
1155:        repeat {

./plank/Sources/Core/FileGenerator.swift
244:    repeat {

./Sourcery/Sourcery/Parsing/FileParser.swift
847:        repeat {

./GRDB.swift/GRDB/QueryInterface/SQLGenerationContext.swift
323:                repeat {

./IBAnimatable/Sources/Extensions/UIBezierPathExtension.swift
113:    repeat {

./ReSwift/ReSwift/Utils/Assertions.swift
21:    repeat {

./SwifterSwift/Sources/Extensions/UIKit/UIViewExtensions.swift
237:        repeat {

./Kitura/Sources/Kitura/bodyParser/BodyParser.swift
191:        repeat {

./Sourcery/Pods/xcproj/Sources/xcproj/PBXProj.swift
135:            repeat {

./Sourcery/Pods/Nimble/Sources/Nimble/Utils/Stringers.swift
90:        repeat {

./Sourcery/Pods/Nimble/Sources/Nimble/Matchers/EndWith.swift
12:            repeat {

./Sourcery/Pods/Nimble/Carthage/Checkouts/CwlPreconditionTesting/Sources/CwlPreconditionTesting/CwlCatchBadInstruction.swift
96:		repeat { do {

Thanks to everyone who participated in the review discussion. The formal review period ended yesterday. However, if you have any additional review feedback to give, or want to continue the ongoing side discussions in this thread, you are welcome to do so, until the core team makes its final decision regarding this proposal.

Will the (degenerate) optional chain solution make each of these work?

  1. for e in try? s?.throwingAndReturningOptSeq()? { … }
  2. for e in try? s?.throwingAndReturningOptAny()? as? [Int] { … }
  3. for e in try? s?.throwingAndReturningOptAny() as? [Int] { … }
Since each !-counterpart works today (click for demo).
struct S {
    func throwingAndReturningOptSeq() throws -> [Int]? { return [1, 2, 3] }
    func throwingAndReturningOptAny() throws -> Any? { return [1, 2, 3] }
}

let s: S? = S()

// All these work:
for e in try! s!.throwingAndReturningOptSeq()! { print(e) }
for e in try! s!.throwingAndReturningOptAny()! as! [Int] { print(e) }
for e in try! s!.throwingAndReturningOptAny() as! [Int] { print(e) }
2 Likes

I would expect to have to use parentheses:

for e in (try? s?.throwingAndReturningOptSeq())? { }
for e in (try? s?.throwingAndReturningOptSeq() as? [Int])? { }

#2 and #3 are equivalent, since I'm pretty sure no one has suggested the syntax shown in #2.

I agree that this is a problem for the .orEmpty() solution, but more importantly this also highlights a general "problem" of optional chaining; the flip side of its convenience:

In order to use optional chaining and never be surprised, the user must never forget that at each "link" in the chain, there are two types, the outer/resulting type and the type at the "link" / the user's cursor.

In regular chaining, there's only one type, the resulting type is always the same as the type at the user's cursor (which is also true for "force-unwrapped chaining").

The convenience of optional chaining is achieved by hiding its "nestedness" from the syntax and thus the user.

Optional chaining is nice, but it's important to remember that conveniences achieved via "white lies" have a price. While making some situations easier, they can also blur core concepts and introduce common misconceptions, which makes it hard to form an effective/correct intuition about the language.

It should never be hard for users to understand why their code doesn't compile, but if its hard to understand why maybeRange?.reversed().orEmpty() doesn't compile, then it's because it's hard to understand optional chaining, which is a (big) problem.


Thanks, I'm asking because all the three examples do currently work if you replace the ? with a ! (see code in the post), and I was under the impression that one of the goals was to make ? and ! analogous.

I was only considering it for looping at the moment. Making it fully equivalent has much farther ranging implications. Personally I’m not sure it would make sense, but I haven’t really considered it fully.

For the sake of curiosity, I used those exact regular expressions to search my codebase at work using Xcode, and got the following results:

?? literal: 1
if let ... for: 1
gaurd let (pattern 1): 4 (5 total, 1 false positive)
guard let (pattern 2): 4 (7 total, 2 false positive)
for-in: 184

Swift LOC: 111,767

The false positives are cases where the guard let patterns return hits where the guarded variable is not the sequence being looped over.

1 Like

The trap you’ve pointed out is the most compelling reason I can see to explore solutions, but this proposal solves the wrong problem, because the trap will still be there in some other form. Fixing the way shadowing works in the language seems like a much more urgent problem, and one that we may not be able to solve later.

6 Likes

Hard for me, too! Mind explaining why?

2 Likes
Here's an implementation of a simple solution
public struct FlattenedOptionalSequence<Base: Sequence>: IteratorProtocol, Sequence {
    let baseNext: () -> Base.Element?
    init(_ optionalSequence: Optional<Base>) {
        switch optionalSequence {
        case .none: baseNext = { return nil }
        case .some(let s):
            var baseIterator = s.makeIterator()
            baseNext = { return baseIterator.next() }
        }
    }
    public func makeIterator() -> FlattenedOptionalSequence<Base> { return self }
    public mutating func next() -> Base.Element? { return baseNext() }
}

struct Empty { }
let empty = Empty()
func ??<S: Sequence>(lhs: S?, rhs: Empty) -> FlattenedOptionalSequence<S> {
    return FlattenedOptionalSequence(lhs)
}

let maybeRange: Range<Int>? = 0 ..< 3

for e in maybeRange?.reversed() ?? empty {
    print(e) // prints 2 1 0
}

let maybeHugeRange: Range<Int>? = 0..<Int.max

for i in maybeHugeRange?.reversed() ?? empty {
    print(i) // prints 9223372036854775806 and breaks, no doom
    break
}

that works for any sequence and has no problems with:

let maybeRange: Range<Int>? = 0 ..< 3

for e in maybeRange?.reversed() ?? empty {
    print(e) // prints 2 1 0
}

or:

let maybeHugeRange: Range<Int>? = 0..<Int.max

for i in maybeHugeRange?.reversed() ?? empty {
    print(i) // prints 9223372036854775806 and breaks, no doom
    break
}

My tests didn't show any performance overhead compared to having the loop inside an if let.

1 Like

Because orEmpty was proposed as a method on Optional so it is not available as part of an optional chain (unless a double optional is involved). Parentheses would be required: (maybeRange?.reversed()).orEmpty.

I’m still a little confused, is this allowed?

for x in sequence?.reversed()? { ... }

The discussion has shown that always appending ?, as the alternative solution suggests, doesn't prove to be a reasonable rule in the general case, especially in relation to expressions that already acknowledge optional behavior:

for x in try? sequence.something() { ... }

for x in data as? [String] { ... }

So right now the most promising compiler-side solution is to only require ? where it isn't acknowledged, so:

for x in optionalSequence? { ... }  

for x in optionalSequence?.reversed() { ... }
1 Like
What is your evaluation of the proposal?

+0.5, for both bike shedding and importance reasons.

First, I wonder if another syntax would be more consistent with the language, for instance:

for person in let people = people { ... }

might be more consistent with the rest of the language.

Second, I wonder if this is really needed enough to complicate the grammar at all.

Is the problem being addressed significant enough to warrant a change to Swift?

To be honest, I don't think so.

I haven't run into this case, as I typically avoid optional except when needed. I don't have much data where an empty sequence and a missing sequence would have different meanings, so by the time I try to process data it would have some other mechanism to either skip processing the sequence, or have been converted to an empty sequence.

If a missing and empty sequence mean two different things, then people will generally be using optionality for a distinct processing path. If not - why bother have a missing sequence in your data model at all?

Does this proposal fit well with the feel and direction of Swift?

I would imagine in terms of simplifying control flow (focusing on the business logic than complex, nested conditional logic)

If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

None

How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Argued in the original thread :slight_smile:

2 Likes

Thanks again to everyone who participated in this review discussion. The core team has decided to reject the proposal, but would like to consider a new proposal focused on the optional-chaining alternative.

3 Likes