Mutating `[T]?` with a default value, or: returning inout, or: extending Any

Evolution Discussion
,
1

Problem

struct Foo { var items: [Int]? }

var foo = Foo()

Now I want to add something to foo.items. But because it's optional, I can't do foo.items.append(3): I first need to provide a default value of [].

Workarounds today

Awkwardly long, and the new item is spelled twice:

if foo.items == nil { foo.items = [3] }
foo.items?.append(3)

Awkwardly long, and uses force-unwrap when we know items is non-nil (safe for now, but the compiler isn't helping us keep it safe as the code is refactored):

if foo.items == nil { foo.items = [] }
foo.items!.append(3)

We can hide the awkwardness in a generic operator, but still need a force-unwrap:
(If we could return inout T from the function, we could do (foo.items ??= []).append(3), but that is not supported.)

foo.items ??= []
foo.items!.append(3)

// via:
infix operator ??= : AssignmentPrecedence
func ??=<T>(lhs: inout T?, rhs: @autoclosure () -> T?) {
    lhs = lhs ?? rhs()
}

Subscripts allow us to provide both get & set directions, so this is pretty good, but can only be defined by extension Foo, not in general: (see below for a more general version I found while writing this post)

foo[createIfEmpty: \.items].append(3)

// via:
extension Foo {
  subscript<T>(createIfEmpty keyPath: WritableKeyPath<Self, [T]?>) -> [T] {
    get {
      if let value = self[keyPath: keyPath] {
        return value
      }
      return []
    }
    set {
      self[keyPath: keyPath] = newValue
    }

    // ...or something like:
    _read {
      yield self[keyPath: keyPath] ?? []
    }
    _modify {
      if self[keyPath: keyPath] == nil { self[keyPath: keyPath] = [] }
      yield &self[keyPath: keyPath]!
    }
  }
}

Okay, so we can work around this with operators which can be defined as global/free functions... But because we can't return inout T, the modification has to be wrapped in a closure (unlike subscript):

foo ~~ (\.items, { $0.append(3) })

infix operator ~~
func ~~<T, U>(lhs: inout T, rhs: (WritableKeyPath<T, [U]?>, (inout [U]) -> Void)) {
    if lhs[keyPath: rhs.0] != nil {
        rhs.1(&lhs[keyPath: rhs.0]!)
    } else {
        var newValue: [U] = []
        rhs.1(&newValue)
        lhs[keyPath: rhs.0] = newValue
    }
}

Or perhaps a more ergonomic version with a second operator (if I may be allowed to abuse operator names from Control.Lens.Setter), although it now requires an @escaping closure:

foo&\.items %~ { $0.append(3) }

precedencegroup ApplyPrecedence { }
precedencegroup ModifyPrecedence { higherThan: ApplyPrecedence }
infix operator & : ApplyPrecedence
infix operator %~ : ModifyPrecedence

func & <T, U>(lhs: inout T, rhs: (WritableKeyPath<T, [U]?>, (inout [U]) -> Void)) {
    if lhs[keyPath: rhs.0] != nil {
        rhs.1(&lhs[keyPath: rhs.0]!)
    } else {
        var newValue: [U] = []
        rhs.1(&newValue)
        lhs[keyPath: rhs.0] = newValue
    }
}
func %~ <T, U>(lhs: WritableKeyPath<T, U?>, rhs: @escaping (inout U) -> Void) -> (WritableKeyPath<T, U?>, (inout U) -> Void) {
    return (lhs, rhs)
}

And actually, I realized as I was writing this post that there's another nicer solution in just putting the subscript directly on Optional without using KeyPaths:

foo.items[withDefault: []].append(3)

extension Optional {
  subscript(withDefault value: @autoclosure () -> Wrapped) -> Wrapped {
    get { return self ?? value() }
    set { self = newValue }

    // or:
    _read { yield self ?? value() }
    _modify {
      if self == nil { self = value() }
      yield &self!  // can we avoid this force-unwrap? unsafelyUnwrapped can only be used as a getter...
    }
}

Better solutions?

So it's clear that the problem can be worked around, but I'd argue most of the solutions above aren't very Swifty, as they're either overly verbose, overly opaque, or use unnecessary force-unwrapping.

The one I do like is extension Optional { subscript(withDefault value: Wrapped) }. However, I still feel it's limiting: we have to repeat [withDefault: []] on every access, or use ?. or !. each time:

foo.items[withDefault: []].append(2)
foo.items[withDefault: []].append(3) // repetitive
foo.items!.append(4) // prone to breakage if above code is moved

I feel like I'd want to write either:

inout var items = foo.items.withDefault([])  // or just "foo.items ?? []"
items.append(2)
items.append(3)

or

foo.items[withDefault: []].tap {
  $0.append(2)
  $0.append(3)
}

And these would require language features we don't have: returning inout, and/or extensions on Any.

Returning inout

I used subscripts above not because this "feels like" a subscript access, but because subscripts allow modifying a value with extra logic before (unwrapping the Optional) and cleanup afterward (re-wrapping). I imagine this is much easier to support for subscripts, whose lifetime is limited to a single statement, than for general long-lived values.

However @John_McCall's Ownership Manifesto does have a section on returning inout "ephemerals" from regular functions:

… The correctness of this depends not only on the method being able to clean up after the access to the property, but on the continued validity of the variable to which self was bound. What we really want is to maintain the current context in the callee, along with all the active scopes in the caller, and simply enter a new nested scope in the caller with the ephemeral as a sort of argument. But this is a well-understood situation in programming languages: it is just a kind of co-routine. …

So that's why yield is used in _modify, and it feels natural to do it similarly in a non-subscript function:

extension Optional {
  mutating func withDefault(_ value: @autoclosure () -> Wrapped) -> inout Wrapped {
    if self == nil { self = value() }
    yield &self!
  }
}

Now I imagine the hard part is making the compiler correctly track/extend the lifetime of the inout (or coroutine), and the lifetime of the object it's referencing. I'm sure there is some precedent in Rust here, but I don't know it well enough to say. I haven't been following the work on coroutines, but I'm curious if the current plans can help address this.

extending Any

The other natural solution I can see is with something like Ruby's tap. This can be done with a free function today:

tap(foo.items[withDefault: []]) {
  $0.append(2)
  $0.append(3)
}

func tap<T>(_ value: T, _ body: (inout T) -> Void) -> T {
  var value = value
  body(&value)
  return value
}

But that feels awkward, especially as we're getting more used to the method-chaining style of SwiftUI modifiers. (And I'm not sure if the extra local var affects optimization at all.)

(Aside: hmm, I see @beccadax added a TapExpr to the compiler in #20214 :open_mouth:)

I'd prefer this, although I don't know if there are technical reasons you can't extend Any/AnyObject, so I'd be interested to hear if anyone knows more:

foo.items[withDefault: []].tap {
  $0.append(2)
  $0.append(3)
}

extension Any {
  mutating func tap(_ body: (inout Self) -> Void) -> Self {
    body(&self)  // does this violate the law of exclusivity in a mutating function?
    return self
  }
}

Note that the subscript is still required in order to extract a non-Optional "inout" ephemeral from the original Optional. But as I mentioned before, I don't think this API feels like it should be a subscript access; I'd rather have foo.items.withDefault([]).tap { ... } but it isn't possible today as far as I know. So maybe we should allow both returning inout and extending Any?

2 Likes
2

If you're going to replace nil with the default value at the very first mutation anyway, wouldn't it be better to just use that default value (var items: [Int] = [])?

3

You're right that my simple example doesn't really motivate why [Int]? is used instead of just [Int]. For a little more context, I originally ran into this problem with URLComponents.queryItems : [URLQueryItem]?.

I also think real-world software is sufficiently complex that there are plenty of situations where it's useful to distinguish between an empty collection and the absence of a collection (such as when a list is loading asynchronously and hasn't been confirmed empty yet).

I also expect the features I described would have a lot more uses than simply mutating optional collections, but maybe I need to do some more work to gather use cases :slight_smile:

1 Like
4

Something like this would be my suggestion. It is in the same vein as the subscript we added for dictionary keys and would be strongly consistent therefore with existing precedent.

foo.bar[default: [1, 2, 3]].append(4)
2 Likes
5

The difference is that for the Dictionary subscript, there's actually a key being used as well, and subscripts are the primary way to access values from a dictionary. Optional is a single-value collection so doesn't need keys, and people are used to accessing values via map and pattern bindings. That's why I feel like the subscript doesn't really belong here.

Not to mention a "default default" (e.g. using [] as the default whenever Wrapped is an Array): something like foo.items.orEmpty().append(3) can't be written as a subscript because there would be nothing to put inside the brackets: foo.items[orEmpty].append(3) means something different.

1 Like
6

Yes, this wouldn't extend to a "default default" like you describe. But since the underlying philosophy of Swift is not to have such "default defaults" (i.e., explicitly not to use 0 as the default for numeric types and [] for collection types), this limitation would be a plus and not a minus for the standard library itself!

You can, of course, write your own API spelled orEmpty, but [] is both more concise and more explicit than "Empty" in spelling out what that actually means for the default value that's used.

3 Likes
7

Well, many people aren't used to using map on Optional because they tend to associate the operation with collections. Sure, you can explain monads until you're blue in the face, but the much simpler way of explaining it is that Optional is like a collection that can have either zero or one element. And subscripts are how we access elements of a collection, so using such notation is a pro and not a con from that perspective.

1 Like
8

Hmm, maybe I am out of touch, but I haven't seen much Swift code that seemingly avoided or was so unaware of Optional.map as to write unnecessarily verbose pattern-matching code (although I would expect there is less familiarity with flatMap). And since Optional isn't actually a Collection, I find it hard to believe that people would find subscript notation to be at all natural on an Optional.

It's funny to imagine how it could be used if it were actually a collection: does unwrapping require a runtime-checked opt[opt.startIndex] operation? :grimacing: Does .first simply return self? :exploding_head: Although it has been discussed:

It might even be more confusing for actual collection types, where collection[ something ] almost always indicates we're extracting an element of the collection, but here we're just providing a different, mutable, view of the collection.

1 Like
9

Mmm, the ship has sailed there. Slices do exactly exact with subscript notation.

10

Interesting points! My argument against using a subscript is getting weaker. I'm tempted to say that slices fall into the "term of art" category where Swift didn't really try to design the API in a vacuum, but that doesn't change the fact that they do indeed provide a view. Maybe my problem with subscripts for defaults is that I feel like it's unnatural to think of it as a view/slice in the same way I think of actual slices. But maybe that's just me.

Side note: this led me to look into UnboundedRange and found interesting comment from Dave — maybe Optional as a collection would just use the [] spelling :)