Non-`Copyable` `Optional` types

Hi everyone. I've been working on adding support for Optional to wrap non-copyable types, and I'd like to start getting feedback on a proposal to do so. I have a PR up with the current draft proposal text:

For convenience, here is the text of the first draft. Thanks for taking a look!


Non-Copyable Optional types

Introduction

This proposal adds support for using the Optional type and its basic operations to wrap noncopyable types.

Motivation

Optional is a fundamental type used to represent potentially-missing values, a need that extends to noncopyable types. Because Optional is a generic type, and noncopyable types are not currently allowed to be used as generic arguments, this is not currently supported, leaving noncopyable types susceptible to suboptimal design choices such as using sentinel values or type-specific optional-like types to represent unavailable cases. For noncopyable values, Swift furthermore imposes the constraint that values can't be used after being consumed, but sometimes it is necessary to consume a value in situations where it cannot be statically proven safe to do so, such as when the value is owned by an actor, object, or global variable. Optional serves a role in the analogous situation with initialization, where a nil value can be used to stand in for a value that will be initialized later in cases where Swift's static initialization requirements can't express, and Optional could also enable dynamic consumption, allowing a nil value to stand in for a value after it's been consumed.

A complete design for noncopyable generics ought to include the ability to retrofit existing currency types from the standard library to support their use with noncopyable types, such as Optional, Array, and so on, and also allow external libraries to extend their existing APIs to support noncopyable types. However, we think Optional is important enough to support ahead of fully general noncopyable generics. Optional also has a lot of special support built into the language, and we need to design and specify how those features interact with noncopyable types irrespective of the general noncopyable generics design. Even after noncopyable generics are implemented, it is likely that proposals specific to other standard library types will follow describing how those types should support noncopyable type arguments.

Proposed solution

We propose to extend the Optional type to support being parameterized by a noncopyable type, making the Optional type itself noncopyable. Noncopyable types can be used with all of the builtin operations for unwrapping and manipulating Optionals, including x!, x?, if let, if case, and switch. Additionally, we introduce a take() method on Optional, which can be used to mutate an Optional value to nil while giving up ownership of the value previously inside of it.

Detailed design

Optional of noncopyable type

Optional is allowed to wrap a noncopyable type. The resulting Optional<T> type is itself also noncopyable when this occurs. Noncopyable Optional types may be inferred, or spelled explicitly using any of the sugar syntaxes provided for Optional, including implicitly-unwrapped Optionals.

struct File: ~Copyable { ... }

let maybeFile: File? = ...
let maybeFile2: File! = ...
let maybeFile3: Optional<File> = ...
let maybeFile4: Optional = File(...) // type parameter `File` inferred from context

Noncopyable Optional types are subject to the same constraints as other noncopyable types. A parameter of noncopyable Optional type must explicitly specify whether it uses a borrowing, consuming, or inout ownership convention:

func maybeClose(_ file: File?) // error: no ownership specifier
func maybeClose(_ file: consuming File?) // OK

Generics still require copyability, so a noncopyable Optional type does not conform to any protocols, cannot be stored in an existential, and cannot be passed to functions or methods generic over Optional:

let any: Any = maybeFile // error: maybeFile isn't copyable

func foo<T>(_ optional: Optional<T>) {}
foo(maybeFile) // error: can't substitute noncopyable type File for T

protocol P {}
extension Optional: P {}

func bar<T: P>(_ p: P)
bar(maybeFile) // error: noncopyable type 'File?' does not conform to P

Unconstrained extensions to Optional are implicitly generic over the Wrapped type parameter, and therefore are also unavailable on noncopyable types:

extension Optional {
    func bas() {}
}

maybeFile.bas() // error: can't substitute noncopyable type File for Wrapped

Extensions constrained to a specific noncopyable type are however allowed and can be used on values of the matching type:

extension Optional<File> {
    consuming func maybeClose() { self?.close() }
}

maybeFile.maybeClose() // OK

Operations on noncopyable Optional

The language-builtin operations for Optional work with noncopyable values. Both the force-unwrap operator x! and chaining operator x? can be any of borrowing, mutating, or consuming, yielding access to the unwrapped value in the following postfix expression:

struct File: ~Copyable {
    borrowing func write()
    mutating func redirect(to: borrowing File)
    consuming func close()
}

let f1: File? = File(...)
f1?.write() // OK to borrow
f1?.write() // OK to borrow again
f1?.redirect(to: File(...)) // error, mutation of read-only value

var f2: File? = File(...)
f2?.write() // OK to borrow
f2?.redirect(to: f1!) // OK to mutate
f2?.close() // OK to consume
f2?.write() // error, use after consume

The implicit unwrapping of an implicitly-unwrapped Optional is similarly borrowing, mutating, or consuming, depending on how the unwrapped value is used.

let f3: File! = File(...)
f3.write() // OK to borrow
f3.close() // OK to consume after
f3.write() // error, use after consume

if let, if case, and switch can be used with noncopyable Optional values. These are currently consuming operations, and as when other noncopyable types are pattern matched, they must be explicitly consume-d when doing so for now:

let f1: File? = File(...)

if let f = consume f1 { // consumes f1 to bind f
    f.close() // OK to consume f
}

f1?.close() // error, use after consume

When borrowing binding and pattern matching forms are implemented, they will be supported for Optional as well.

Noncopyable Optional values may be constructed containing a value using implicit conversion from the wrapped type, the .some enum case, and/or the Optional(x) initializer. These all consume the value, moving it inside of the Optional wrapper. Empty Optional values may be constructed using the nil literal or .none.

The take() method

One important use case for noncopyable optionals is to dynamically move or consume values from an owner for which consumption cannot be statically proven safe. If an object or actor owns a noncopyable value, for example, it must own a valid value for the object's entire lifetime. It is normally not possible to consume values owned by objects:

class FileOwner {
    var file: File


    // We would like to be able to give up ownership of the file, but
    // that would leave the object in an invalid state.
    func giveUpFile() -> File {
        return file // error: can't move `file` to return it
    }
}

We can mutate the file in place, leaving another valid file behind after moving the old file out, but that requires having a dummy file or sentinel value:

extension File {
    mutating func replaceWithDummy() {
        // We can consume the current `self` in a mutating method...
        let result = consume self
        // ...but we need to leave a new value behind before we return back
        self = File("/dev/null")
        return result
    }
}

class FileOwner {
    var file: File


    func giveUpFile() -> File {
        return file.replaceWithDummy()
    }
}

If we wrap the value in Optional, then we can use nil to safely represent the absence of a value after it's been moved away. If we had support for noncopyable generics, then we could write this as a mutating method on Optional:

extension Optional where Wrapped: ~Copyable {
    mutating func take() -> Wrapped? {
        let result = consume self
        self = nil
        return result
    }
}

class FileOwner {
    var file: File?

    func giveUpFile() -> File {
        // Now we can use take() to dynamically take the file away from
        // the object, leaving nil behind (or raising a fatal error if
        // someone else already did)
        return file.take()!
    }
}

We propose to support the take() method generically on noncopyable types as a special case, which will be subsumed by a normal method once noncopyable generics are fully supported, since it is otherwise difficult to express using the builtin operations on Optional that are supported.

Source compatibility

This proposal ought to be purely additive, not affecting the behavior of existing code. This proposal is also intended to be forward-compatible with a future version of Swift that does support noncopyable generics in the general case. We do not expect the semantics of noncopyable Optional operations to
differ from those that we would implement given fully general noncopyable generics support. Also, although this proposal leaves methods generic over Optional<T> and unconstrained extensions on Optional as requiring copyability, it is almost certain that we would need to do for source compatibility even with noncopyable generics support, since existing generic implementations and extensions can assume that their arguments are copyable. Therefore, we expect that generic functions accepting noncopyable Optional types will require some sort of opt-in syntax, such as extension Optional where Wrapped: ~Copyable, and therefore the lack of generics support in this proposal is not a future source compatibility concern.

ABI compatibility

Optional wraps noncopyable types using the same layout mechanisms as it does for copyable types, and nongeneric noncopyable types do not require runtime support, so there are no ABI compatibility or back deployment concerns with enabling support for noncopyable Optional for concrete noncopyable types.

Implications on adoption

There should be no backward compatibility or deployment limitations adopting noncopyable Optional types.

Alternatives considered

Should take return a non-Optional?

Since the primary intended purpose of take is to allow for dynamic consumption of a value, one could argue that it should return the unwrapped value as a non-Optional, and raise a fatal error if no value is available to take:

extension Optional where Wrapped: ~Copyable {
    mutating func take() -> Wrapped {
        let result = consume self!
        self = nil
        return result
    }
}

Our proposal favors returning the Optional value as is, because we feel that is the more compositional approach. The caller may call take() and choose whether force-unwrapping, throwing an error, falling back to a default value, or choosing some other arbitrary execution path is appropriate.

What should take be named?

take() could be named something tying it more closely to other noncopyable type concepts, such as move() or consume(). However, we want to make sure that there is a clear distinction between what x.take() does, which is to dynamically reset x to nil, from what consume x does, which is to statically end the lifetime of x. We hope that using a distinct term like take makes this difference easier to understand and discuss. The name take is also used in Rust for the same operation.

Future directions

Noncopyable generics

General support for noncopyable generics would subsume the special case behavior for Optional and the take() method, and enable us to make more of Optional's library interface available for noncopyable types.

Borrowing pattern matching

We plan to add support for borrowing forms of if let unwrapping and pattern matching, which would be directly applicable to Optional.

29 Likes

Cool, supporting Optional as a special case while we work out the full generics design makes sense to me. On take(), I wonder if the motivation for its inclusion, especially its inclusion now before the generics story is fleshed out, could be a bit better motivated. The pitch says:

But the core of the take() operation appears to be:

let consumedValue = consume maybeValue
maybeValue = nil

which seems relatively trivial for users of optional noncopyable types to write themselves. Or is the proper 'manual' way of replacing take() more subtle than I'm giving it credit for?

3 Likes

It's true you could write it today, but in order for it to work in dynamic situations like object-owned noncopyable values, the consume and reassignment have to happen within an enclosing inout access. So today you'd need to write:

let consumedValue = {
  let r = $0
  $0 = nil
  return r
}(&object.value)

to do that. Without generics support, you would have to write that whole thing out at every point you need to use it. It's not impossible, but it's nonobvious and awkward enough that it seems to me worth providing a method for.

9 Likes

Ah, got it, thanks for explaining the catch there. Though since the consume-and-replace operation must definitionally take place during an exclusivity window, is there room to add a rule that would lift the inout access into the local scope so that the 'simple' version I wrote above would work?

// we're consuming out of 'maybeValue', begin mutable access and assume local exclusivity
let consumedValue = consume maybeValue
maybeValue = nil
// 'maybeValue' has been replaced, end mutable access
1 Like

There's a tradeoff doing that in the general case, since implicitly extending a dynamic exclusivity window for a global/static/class property raises the likelihood of the program crashing from a dynamic exclusivity violation, so we generally err on the side of keeping implicit exclusivity windows as small as practical. If you pass something as an inout argument, or borrow a noncopyable value, then we obviously have to have a prolonged access, but in other cases where we have options we avoid it.

If we introduce borrow/inout bindings, though, then that could be a way to perform the swap in-line with slightly less boilerplate:

inout owner = object.value
let consumedValue = consume owner
owner = nil

though that still strikes me as relatively bulky.

4 Likes

From the implementation perspective, is there an intermediate stage possible between this pitch and generalized support for noncopyable types which would permit specifically extension Optional where Wrapped: ~Copyable?

I ask because, if that’s both possible and a significantly narrower undertaking, it would seem a more natural “resting place” where not only the proposed take() but also users’ own extensions would be expressible in Swift.

An extension on Optional doesn't really have any formal constraints compared to any other generic context. We can ensure that a fixed set of operations on Optional can always be generated as inline code without ever calling into runtime generics or otherwise involving the Swift runtime, but we can't make that guarantee for an arbitrary method defined on Optional.

4 Likes

This makes sense, but could you get any smaller than "begin when the value is consumed out of the binding and end when a value is replaced"? Any access during that window would always be invalid, no?

I agree that even the solution with inout bindings is a little heavy, but since some of the justification for the special take() method is to tide us over until we have a more complete generics/ownership story, I guess my question boils down to: in a world where we have inout bindings and users can define their own extensions on Optional where Wrapped: ~Copyable, would we still want take()? If so I think it would be good for a proposal to justify that a bit more directly. And if we think not, then would the pain of not having take() in the interim outweigh the undesirability of having to keep it around ~forever once the noncopyability story is more complete?

2 Likes

Ah, so it’s not so much implementation difficulty as regards special-casing Optional specifically, but the issue that we don’t want any runtime dependency here.

Put another way, if we had today the right knobs and toggles to enforce the right “baremetal” subset of Swift, arbitrary extensions which opt into that could be supported, IIUC?

Runtime support for noncopyable types is definitely a concern, but in its full generality, there are a number of open design questions about how we should present noncopyable generics in the language. Optional's core functionality is narrow enough thankfully not to bump into too many of them, and it's self contained enough to hopefully be able to explain what's supported.

3 Likes

I do feel like it’s a bit rough to have only ?. for borrowing chaining—no map and no if let. I understand that both of those have problems:

  • if let foo = borrow bar, or if borrow foo = bar, however you want to spell it, is basically equivalent to the whole feature of local borrows.

  • We aren’t planning to allow overriding on ownership, so map has to remain an owned-to-owned function. Introducing something map-like using the same mechanisms as take means committing to keeping it forever, even if it can eventually become a regular function.

But without some way to do this, anything that wants to do a momentary access for something that’s not a method has to test the value, unwrap it, use it, and put it back. Or more likely, add an extension method to the wrapped type to just do the thing.

5 Likes

My hope is that we will have borrow/inout bindings ready in the near future too, which should fill that gap, hopefully before we get to fully general noncopyable generics support. That would be necessary to generalize map in its current form, where it currently takes a borrowing-to-owned closure, so that the map operation itself would also be borrowing-to-owned.

3 Likes

A naive question perhaps, but is take() truly necessary? Can the following not be made to work equivalently?

class FileOwner {
    var file: File?

    func giveUpFile() -> File {
        guard let actualFile = consume file else { fatalError(…) }
        return actualFile
    }
}

I know that technically that's consuming the whole file member variable, not its contents, but what if doing so just reset the file member variable to .none, instead? e.g. through a new CustomConsumable protocol (or whatever name) that lets a type override the default behaviour (of making the value intrinsically invalid) by specifying a replacement value instead?

Only time & experience will tell for sure, but it feels like it'll be weird if making something optional suddenly changes how it has to be used, beyond the optionality itself. I know optionals are technically binary enums, but I prefer to think of them (generally) as more like a container where consuming from it moves the contents somewhere else, but leaves the container itself still valid (but empty).

Unless there's a purpose in distinguishing between .none and intrinsic invalidity, for a non-copyable optional? I can't think of an example where it'd practically matter - other than the difference in compiler warnings about accessing the old value, but then do you really want those given you've already stated you want the value to be optional? Feels like recursive optionals. Plus if that matters new diagnostics could be added to say "Optional 'foo' read after being consumed - value is always nil; are you sure you meant to do that?".

1 Like

What if there is a need to consume an Optional conditionally (depending on the wrapped valu)?
I guess we would need something that takes the wrapped value by consuming, and conditionally returns it back as an Optional?

extension Optional {
  mutating func _mapMaybeConsuming<Result>(_ body: (consuming Wrapped) -> (Result, Wrapped?)) -> Result? {
    switch consume self {
    case .none:
      self = .none
      return .none
    case let .some(value):
      self = .none
      let resultPair = body(consume value)
      let result = resultPair.0
      self = (consume resultPair).1
      return result
    }
  }
}

Even if we did that, that variant of consuming operation still has to formally be a mutating operation on the location being taken from, and it would have the same overall semantics as the take() method proposed. That's quite a bit different from the primitive consume operator, which doesn't require mutability of its base, only the ability to take ownership of it. Since take() doesn't really have any "magic" capabilities that can't be expressed by a normal function, but consume x does, I think it makes sense to have things be normal methods when they can be. (Conversely, when we had originally proposed consume x to use special syntax, instead of looking like a function move(x) with special powers, those special powers were a common argument against it looking like a normal function.)

We also generally try to introduce protocols only when there's a generic set of rules that can be set for them which generic functions can be written against. Modelling CustomConsumable as a protocol raises similar questions to a DefaultInitializable protocol with an init() requirement as to what the "empty" value generically means across types: can it be nil? An empty array? 0? Some sentinel value? There are some similarities between some of those, but not very many common properties. Furthermore, in C++, std::move(x) leaves a "valid" value behind in x, and it's a point of confusion in C++ what can validly be done with x after it's been moved from; some types will assert, crash, or UB if you try to use them again without reassigning them, some do guarantee they replace valid values (though the nature of that valid value is type dependent), for some it depends on what language version or C++ standard library you happen to be using. I feel like a clear demarcation between lifetime-ending static consume and mutating take() will lead to a clearer programming model overall. As a common idiom, perhaps other types like Array could also provide take() methods, but that doesn't necessarily mean we have to unify them with a protocol.

9 Likes

I'm not convinced out-right, but I respect (and appreciate) your detailed explanation. Only time will tell how this is used and what turns out to be the right approach, but the good thing is I don't see any reasons why this behaviour ("consume vs take", so-to-speak) couldn't be amended at any later time, if need be. I'm happy to see Optional getting non-copyable support soon, in any case.

1 Like

If I understand you correctly this will introduce different behaviour depending on where from you are consuming:

class Foo {
  var x: File?

  func foo() {
    let y = consume x
    // x can be accessed, and == nil
  }
}

func bar(x: File?) {
  let y = consume x
  // x can't be accessed
}

Then imaging you have some kind of observation of x (didSet for example). How would you reason changes in observation results?

When we get inout bindings, then you should be able to do something like:

if inout value = maybeValue {
  mutate(&value)
}

allowing you to do some more elaborate mutation on the value in place.

inout bindings are cool, but I didn't get how are they helpful when I want to take the value if the value matches some condition? I.e.

struct Value: ~Copyable {
  var id: Int
}

var maybeValue: Value?

if inout value = maybeValue {
  if value.id % 2 == 0 {
    // I want to `consume value` here and never return it to `maybeValue`
  }
}

You can’t exit flow control with maybeValue being in an indeterminate state of initialization. Otherwise the compiler couldn’t determine whether subsequent loads from maybeValue are valid.