Borrowing and consuming pattern matching for noncopyable types

Hi everyone. I've been working on improving pattern matching support for noncopyable types so that they can be matched by borrowing in addition to consuming them, introducing borrowing x bindings into matching patterns as a way to indicate a borrow binding. This is available in top-of-tree snapshots now under the feature flag -enable-experimental-feature BorrowingSwitch, using the temporary syntax _borrowing x for borrow bindings. I'm ready now to propose making this an official language feature and would appreciate feedback on the design and implementation. Here is my draft proposal:

Thanks for reading! I'll also post the text of the initial draft below for convenience and context:


Borrowing and consuming pattern matching for noncopyable types

  • Proposal: SE-ABCD
  • Authors: Joe Groff,
  • Review Manager: TBD
  • Status: Awaiting review
  • Implementation: on main, using the BorrowingSwitch feature flag and _borrowing x binding spelling
  • Upcoming Feature Flag: BorrowingSwitch

Introduction

Pattern matching over noncopyable types, particularly noncopyable enums, can be generalized to allow for pattern matches that borrow their subject, in addition to the existing support for consuming pattern matches.

Motivation

SE-0390 introduced noncopyable types, allowing for programs to define structs and enums whose values cannot be copied. However, it restricted switch over noncopyable values to be a consuming operation, meaning that nothing can be done with a value after it's been matched against. This severely limits the expressivity of noncopyable enums in particular, since switching over them is the only way to access their associated values.

Proposed solution

We lift the restriction that noncopyable pattern matches must consume their subject value. To enable this, we introduce borrowing bindings into patterns, and formalize the ownership behavior of patterns during matching and dispatch to case blocks. switch statements infer their ownership
behavior
based on the necessary ownership behavior of the patterns in the switch.

Detailed design

borrowing bindings

Patterns can currently contain var and let bindings, which take part of the matched value and bind it to a new independent variable in the matching case block:

enum MyCopyableEnum {
    case foo(String)

    func doStuff() { ... }
}

var x: MyCopyableEnum = ...

switch x {
case .foo(let y):
    // We can pass `y` off somewhere else, or capture it indefinitely
    // in a closure, such as to use it in a detached task
    Task.detached {
        print(y)
    }

    // We can use `x` and update it independently without disturbing `y`
    x.doStuff()
    x = MyEnum.foo("38")

}

For copyable types, we can ensure the pattern bindings are independent by copying the matched part into the new variable, but for noncopyable bindings, their values can't be copied and need to be moved out of the original value, consuming the original in the process:

struct Handle: ~Copyable {
    var value: Int

    borrowing func access() { ... }

    consuming func close() { ... }
}

enum MyNCEnum: ~Copyable {
    case foo(Handle)

    borrowing func doStuff() { ... }
    consuming func throwAway() { ... }
}

var x: MyNCEnum = ...
switch x {
case .foo(let y):
    // We can pass `y` off somewhere else, or capture it indefinitely
    // in a closure, such as to use it in a detached task
    Task.detached {
        y.access()
    }
    
    // ...but we can't copy `Handle`s, so in order to support that, we have to
    // have moved `y` out of `x`, leaving `x` consumed and unable to be used
    // again
    x.doStuff() // error: 'x' consumed
}

// Since the pattern match had to consume the value, we can't even use it
// after the switch is done.
x.doStuff() // error: 'x' consumed

We introduce a new borrowing binding modifier. A borrowing binding references the matched part of the value as it currently exists in the subject value without copying it, instead putting the subject under a borrowing access in order to access the matched part.

var x: MyNCEnum = ...
switch x {
case .foo(borrowing y):
    // `y` is now borrowed directly out of `x`. This means we can access it
    // borrowing operations:
    y.access()

    // and we can still access `x` with borrowing operations as well:
    x.doStuff()

    // However, we can't consume `y` or extend its lifetime beyond the borrow
    Task.detached {
        y.access() // error, can't capture borrow `y` in an escaping closure
    }
    y.close() // error, can't consume `y`
    
    // And we also can't consume or modify `x` while `y` is borrowed out of it
    x = .foo(Handle(value: 42)) // error, can't modify x while borrowed
    x.throwAway() // error, can't consume x while borrowed
}

// And now `x` was only borrowed by the `switch`, so we can continue using
// it afterward
x.doStuff()
x.throwAway()
x = .foo(Handle(value: 1738))

borrowing bindings can also be formed when the subject of the pattern match and/or the subpattern have Copyable types. Like borrowing parameter bindings, a borrowing pattern binding is not implicitly copyable in the body of the case, but can be explicitly copied using the copy operator.

var x: MyCopyableEnum = ...

switch x {
case .foo(borrowing y):
    // We can use `y` in borrowing ways.

    // But we can't implicitly extend its lifetime or perform consuming
    // operations on it, since those would need to copy
    var myString = "hello"
    myString.append(y) // error, consumes `y`
    Task.detached {
        print(y) // error, can't extend lifetime of borrow without copying
    }

    // Explicit copying makes it ok
    Task.detached {[y = copy y] in
        print(y)
    }
    myString.append(copy y)

    // `x` is still copyable, so we can update it independently without
    // disturbing `y`
    x.doStuff()
    x = MyEnum.foo("38")

}

To maintain source compatibility, borrowing is parsed as a contextual keyword only when it appears immediately before an identifier name. In other positions, it parses as a declaration reference as before, forming an enum case pattern or expression pattern depending on what the name borrowing refers to.

switch y {
case borrowing(x): // parses as an expression pattern
    ...
case borrowing(let x): // parses as an enum case pattern binding `x` as a let
    ...
case borrowing.foo(x): // parses as an expression pattern
    ...
case borrowing.foo(let x): // parses as an enum case pattern binding `x` as a let
    ...
case borrowing x: // parses as a pattern binding `x` as a borrow
    ...
case borrowing(borrowing x) // parses as an enum case pattern binding `x` as a borrow
    ...
}

This does mean that, unlike let and var, borrowing cannot be applied over a compound pattern to mark all of the identifiers in the subpatterns as bindings.

case borrowing .foo(x, y): // parses as `borrowing.foo(x, y)`, a method call expression pattern

case borrowing (x, y): // parses as `borrowing(x, y)`, a function call expression pattern

Refining the ownership behavior of switch

The order in which switch patterns are evaluated is unspecified in Swift, aside from the property that when multiple patterns can match a value, the earliest matching case condition takes priority. Therefore, it is important that matching dispatch cannot mutate or consume the subject until a final match has been chosen. For copyable values, this means that pattern matching operations can't mutate the subject, but they can be copied as necessary to keep an instance of the subject available throughout the pattern match even if a match operation wants to consume an instance of part of the value.

Copying isn't an option for noncopyable types, so noncopyable types strictly cannot undergo consuming operations until the pattern match is complete. For many kinds of pattern matches, this
doesn't need to affect their expressivity, since checking whether a type matches the pattern criteria can be done nondestructively separate from consuming the value to form variable bindings. Matching enum cases and tuples (when noncopyable tuples are supported) for instance is still possible even if they contain consuming let or var bindings as subpatterns:

extension Handle {
    var isReady: Bool { ... }
}

let x: MyNCEnum = ...
switch x {
// OK to have `let y` in multiple patterns because we can delay consuming
// `x` to form bindings until we establish a match
case .foo(let y) where y.isReady:
    y.close()
case .foo(let y):
    y.close()
}

However, when a pattern has a where clause, variable bindings cannot be consumed in the where clause even if the binding is consumable in the case body:

extension Handle {
    consuming func tryClose() -> Bool { ... }
}

let x: MyNCEnum = ...
switch x {
// error: cannot consume `y` in a "where" clause
case .foo(let y) where y.tryClose():
    // OK to consume in the case body
    y.close()
case .foo(let y):
    y.close()
}

Similarly, an expression subpattern whose ~= operator consumes the subject cannot be used to test a noncopyable subpattern.

extension Handle {
    static func ~=(identifier: Int, handle: consuming Handle) -> Bool { ... }
}

switch x {
// error: uses a `~=` operator that would consume the subject before
// a match is chosen
case .foo(42):
    ....
case .foo(let y):
    ...
}

Noncopyable types do not yet support dynamic casting, but it is worth anticipating how is and as patterns will work given this restriction. An is T pattern only needs to determine whether the value being matched can be cast to T or not, which can generally be answered nondestructively. However, in order to form the value of type T, many kinds of casting, including casts that bridge or which wrap the value in an existential container, need to consume or copy parts of the input value in order to form the result. The cast can still be separated into a check whether the type matches, using a borrowing access, followed by constructing the actual cast result by consuming if necessary. However, for this to be allowed, the subpattern p of the p as T pattern would need to be irrefutable, and the pattern could not have an associated where clause, since we would be unable to back out of the pattern match once a consuming cast is performed.

Determining the ownership behavior of a switch operation

Whether a switch borrows or consumes its subject can be determined from the type of the subject and the patterns involved in the switch. Based on the criteria below, a switch may be one of:

  • copying, meaning that the subject is semantically copied, and additional copies of some or all of the subject value can be made during pattern matching
  • borrowing, meaning that the subject is borrowed for the duration of the switch block.
  • consuming, meaning that the subject is consumed by the switch block.

These modes can be thought of as being increasing in strictness. The compiler looks recursively through the patterns in the switch and increases the strictness of the switch behavior when it sees a pattern requiring stricter ownership behavior. For copyable subjects, copying is the baseline mode,
whereas for noncopyable subjects, borrowing is the baseline mode. While looking through the patterns:

  • if there is a borrowing binding subpattern, then the switch behavior is at least borrowing.
  • if there is a let or var binding subpattern, and the subpattern is of a noncopyable type, then the switch behavior is consuming. If the subpattern is copyable, then let bindings do not affect the behavior of the switch, since the binding value can be copied if necessary to form the binding.
  • if there is an as T subpattern, and the type of the value being matched is noncopyable, then the switch behavior is consuming. If the value being matched is copyable, there is no effect on the behavior of the switch.

For example, given the following copyable definition:

enum CopyableEnum {
    case foo(Int)
    case bar(Int, String)
}

then the following patterns have ownership behavior as indicated below:

case let x: // copying
case borrowing x: // borrowing

case .foo(let x): // copying
case .foo(borrowing x): // borrowing

case .bar(let x, let y): // copying
case .bar(borrowing x, let y): // borrowing
case .bar(let x, borrowing y): // borrowing
case .bar(borrowing x, borrowing y): // borrowing

And for a noncopyable enum definition:

struct NC: ~Copyable {}

enum NoncopyableEnum: ~Copyable {
    case copyable(Int)
    case noncopyable(NC)
}

then the following patterns have ownership behavior as indicated below:

case let x: // consuming
case borrowing x: // borrowing

case .copyable(let x): // borrowing (because `x: Int` is copyable)
case .copyable(borrowing x): // borrowing

case .noncopyable(let x): // consuming
case .noncopyable(borrowing x): // borrowing

case conditions in if, while, for, and guard

Patterns can also appear in if, while, for, and guard forms as part of case conditions, such as if case <pattern> = <subject> { }. These behave just like switches with one case containing the pattern, corresponding to a true condition result with bindings, and a default branch corresponding
to a false condition result. Therefore, the ownership behavior of the case condition on the subject follows the behavior of that one pattern.

Source compatibility

SE-0390 explicitly required that a switch over a noncopyable variable use the consume operator. This will continue to work in most cases, forcing the lifetime of the binding to end regardless of whether the switch actually consumes it or not. In some cases, the formal lifetime of the value or parts of it may end up different than the previous implementation, but because enums cannot yet have deinits, noncopyable tuples are not yet supported, and structs with deinits cannot be partially destructured and must be consumed as a whole, it is unlikely that this will be noticeable in real world code.

Previously, it was theoretically legal for noncopyable switches to use consuming ~= operators, or to consume pattern bindings in the where clause of a pattern. This proposal now expressly forbids these formulations. We believe it is impossible to exploit these capabilities in practice under the
old implementation, since doing so would leave the value partially or fully consumed on the failure path where the ~= match or where clause fails, leading to either mysterious ownership error messages, compiler crashes, or both.

ABI compatibility

This proposal has no effect on ABI.

Future directions

inout pattern matches

With this proposal, pattern matches are able to borrow and consume their subjects, but they still aren't able to take exclusive inout access to a value and bind parts of it for in-place mutation. This proposal lays the groundwork for supporting this in the future; we could introduce inout bindings in patterns, and introducing mutating switch behavior as a level of ownership strictness between borrowing and consuming.

Automatic borrow deduction for let bindings, and explicitly consuming bindings

When working with copyable types, although let and var bindings formally bind independent copies of their values, in cases where it's semantically equivalent, the compiler optimizes aways the copy and borrows the original value in place, with the idea that developers do not need to think about ownership if the compiler does an acceptable job of optimizing their code. By similar means, we could say that let pattern bindings for noncopyable types borrow rather than consume their binding automatically if the binding is not used in a way that requires it to consume the binding. This would give developers a "do what I mean" model for noncopyable types closer to the convenience of copyable types. This should be a backward compatible change since it would allow for strictly more code to compile than does currently when let bindings are always consuming.

Conversely, performance-minded developers would also like to have explicit control over ownership behavior and copying, while working with either copyable or noncopyable types. To that end, we could add explicitly consuming bindings to patterns as well, which would not be implicitly copyable, and which would force the switch behavior mode on the subject to become consuming even if the subject is copyable.

enum deinit

SE-0390 left enums without the ability to have a deinit, based on the fact that the initial implementation of noncopyable types only supported consuming switches. Noncopyable types with deinits generally cannot be decomposed, since doing so would bypass the deinit and potentially violate invariants maintained by init and deinit on the type, so an enum with a deinit would be completely unusable when the only primitive operation supported on it is consuming switch. Now that this proposal allows for borrowing switches, we could allow enums to have deinits, with the restriction that such enums cannot be decomposed by a consuming switch.

Alternatives considered

Explicit marking of switch ownership

SE-0390 required all switch statements on noncopyable bindings to use an explicit consume. Rather than infer the ownership behavior from the patterns in a switch, as we propose, we could alternatively keep the requirement that a noncopyable switch explicitly mark its ownership. Using the borrow operator which has previously been proposed could serve as an explicit marker that a switch should perform a borrow on its subject. This proposal chooses not to require these explicit markers, though consume (and borrow when it's introduced) can still be explicitly used if the developer chooses to enforce that a particular switch either consumes or borrows its subject.

25 Likes

Is there room for this to change in Swift 6?

The Language Steering Group has chosen to narrow any remaining source breaks to those necessary for safe concurrency. We could consider it for a later Swift version, maybe. Another possibility might be to allow the binding introducer come before case, as in let case .foo(a, b):, borrowing case .foo(a, b):, eventually consuming case .foo(a, b): and so on. Theres's a part of me feels like the shorthand was a bit of a mistake, though, since it has led to people accidentally introducing bugs where they expected to match against a constant, but the name was parsed as a variable binding matching anything because of an outer let they forgot about.

7 Likes

I agree with this so strongly that swift-format has a rule that bans let/var from being distributed through complex bindings. I also wonder, as we continue adding more binding specifiers than just let and var, how frequently all the bindings of a complex pattern would actually want the same specifier. I suppose in practice "let-everything" would probably remain the most common, but I think it's still better as a reader to know which variables in a pattern are new bindings and which aren't solely by looking left one keyword.

10 Likes

I think it's great that we're tackling this feature overall, but I have some issues with how it ends up being surfaced in the language. It's almost inherent in the keywords that let is an easy thing to reach for, while borrowing is very explicit and intentional. That's a dichotomy we should lean in to. So I think it's unfortunate that getting a switch to borrow the original value under this proposal requires rewriting all of the cases to use borrowing, and it's even more unfortunate that it's possible to mismatch borrowing with the overall effect of the switch.

We need to distinguish here between the operation of the switch and the way it accesses its operand:

  • A consuming switch destructively breaks down the operand value and naturally presents its patterns with consumable values. If the operand has non-copyable type, a consuming switch must work on an owned value, so if the operand expression refers to a value in storage, the switch must immediately consume the value out of that storage. (If the operand expression is e.g. the r-value result of a call, it is of course naturally owned by the switch anyway, and there's no reason not to do a consuming switch.)
  • A borrowing switch non-destructively breaks down the operand value and naturally presents its patterns with borrowed values. If the operand has non-copyable type, a borrowing switch must borrow a value, so if the operand expression refers to a value in storage, a borrowing switch must borrow that storage for the duration of the switch.

Our only compatibility restriction is that all currently-possible switches (i.e. switches on copyable types that don't use any new syntax like borrowing patterns) have to observably work like they do today. That pretty much boils down to not introducing potential exclusivity conflicts by borrowing the switch operand from mutable storage. Note that whether the switch decomposition is done as a borrowing switch or a consuming switch is not directly observable; we just might need to introduce copies.

With that in mind, I would suggest the following changes. First, the rules to decide the overall emission of the switch:

  1. switch should default to a borrowing switch for non-copyable types if the operand expression refers to storage. It's relatively uncommon to need a consuming switch on an existing value, and it's acceptable (and can be very good for readability) to require people to use an extra keyword to get those semantics.

  2. We should allow programmers to explicitly borrow the operand to switch. This forces the use of a borrowing switch, forces the operand to be borrowed, and imposes additional restrictions on cases. Note that this is (almost) exactly the default behavior of switch for non-copyable operands. We don't currently have a borrow operator, though, and I think this could be subsetted out if need be.

  3. Other switches on non-copyable operands should be consuming switches. That includes the existing switch consume.

  4. Whether a default switch with a copyable operand is borrowing or consuming is unspecified. For compatibility, we just need to make sure that the difference isn't directly detectable. I would suggest for the implementation that we should always borrow the operand whenever we can do so without lingering accesses (most importantly, when the operand is a reference to a let or a parameter).

Note that those rules make it very easy to figure out whether a switch of a non-copyable value is borrowing or consuming: it's always determined by the head of the switch, with no need to look at the cases.

Second, the rules for the bindings in individual case patterns:

  1. let patterns should be allowed in either borrowing or consuming switches.

    • In a consuming switch, the let is bound to the owned value, which can then be consumed within the switch like any non-pattern let.
    • In a borrowing switch, the let is bound to a borrowed value. However, if the value is copyable, it behaves more like a normal let constant: for example, the binding can still be used in implicitly-consuming ways, and doing so just copies the value. Using the consume operator on a copyable let bound this way just copies the value and then forbids the binding from being used again.
  2. borrowing bindings in patterns should only be allowed in borrowing switches, and only if (1) the operand is non-copyable (meaning the conditions above to get a borrowing switch must have applied) or (2) the switch operand is explicitly borrowed. This supports the intentionality of the keyword: if you can put borrowing on a pattern, you know that it will ultimately be bound to a value borrowed directly from the original operand. The binding is then restricted from being implicitly copied, just like a borrowing parameter.

9 Likes

Personally, I think it could be a good idea to go even further and make it so let, if let, and switch as a whole are ownership-neutral and just infer it from context. I mentioned this idea before, but basically, I think it would be ergonomically better if users could just use let when they're in the middle of writing code, without committing to either a consuming or borrowing access at the moment.

2 Likes

I completely agree with this, I think the outer let shorthand was a mistake, and we banned it in our codebase. In fact, I think it should be removed in Swift 6. Incidentally, it caused lots of confusion in places where thinking about matching a value against a pattern is the right way to think about a problem, rather than just trying to remember, at syntactic level, what combination of if, case, let is required for a certain use (the very existence of a website like https://fuckingifcaseletsyntax.com is a testament to this). Also, unfortunately, some otherwise great sources of knowledge about Swift spread the usage of the shorthand syntax, resulting in lots of Swift devs defaulting to it without really understanding that pattern matching was the underlying mechanism, that eventually led to syntax frustration, and even hostility towards an otherwise excellent approach.

2 Likes

Thanks for the thorough feedback @John_McCall. One concern I have with using let to contextually mean a borrow or consuming binding is that it would make let in switches inconsistent with standalone let bindings as they're currently implemented, which currently always consume their binding. I agree that a pattern match has a nice naturally-confined scope, as well as context in which to pre-specify how you want the match to interact with the subject value, but standalone let bindings don't have those benefits as clearly. Our current thinking for standalone borrow bindings is that they would be spelled explicitly borrowing x = <expr> (and similarly inout x = &<expr> or mutating x = &<expr> for inout bindings) so I was trying to maintain consistency with that future direction.

I also don't want to discount outright the ability to "mismatch" the behavior of the bindings in one or more cases with the overall ownership behavior of the switch. I admittedly can't think of a concrete reason to borrow in one branch but consume in another (maybe you want to guarantee the binding will die to its usual deinit on one path, I dunno), but it seems like a much more useful capability once we look ahead to mutating pattern matches; I could easily see code wanting to communicate that it allows mutation of the value via one case but not another.

2 Likes

I see what you're saying, but is this really a semantic property of a let? For example, if we saw this:

let x = y.foo

I don't think it would actually be wrong to emit this as a borrowed reference to foo if doing so didn't have an observable impact (such as introducing potential exclusivity conflicts).

I would say that the core semantic property of a let is that it behaves like an immutable and independent value; that may require a copy to be performed, but if it doesn't, the copy isn't required regardless. Allowing let patterns to borrow values is not inconsistent with that.

My suggestion doesn't permanently rule this out; it just reserves it for the future if we find we have a reason to allow it.

4 Likes

I agree with your assessment, though achieving that illusion of independence for noncopyable values when it may borrow seems like it would require the sort of use-based analysis @ellie20 is advocating for. If you write something like:

let x = y.foo
// pass x off to an escaping use
Task {[x = x] use(x) }

that has to either be an error because x is just a borrow, or consume y.foo so that we can transfer ownership of x. For copyable types, we can copy our way to independence and hope the optimizer cleans up later, but for noncopyable types, it seems like we'd have to know ahead of time how the value is used to maintain that illusion.

1 Like

Right, if we wanted to allow let x = y.foo to just borrow when it would otherwise be unimplementable because of non-copyability — which does seem like a nice convenience — we would have to diagnose attempts to consume it. I think that's okay for let, and programmers could use a consuming binding to be explicit about that not being allowed (and also make the binding no-implicit-copy, which seems like a reason to want consuming bindings separate from let even if we didn't make let loose about borrowing).

On the subject of mix-and-match ownership behavior, I have gotten a few bug reports from early adopters of this feature where they attempt to do something like

enum Maybe: ~Copyable {
  case yup(SomeNoncopyable)
  case nope
}

switch maybe {
case .nope: 
  return // Error: return without reinitializing `maybe` after consume
case .yup(let value):
  maybe = .yup(process(value))
  return
}

being somewhat surprised that they're made to reassign maybe on the "nope" path. It makes sense from the perspective that the switch as a whole is consuming, but I can see where the surprise comes from when some branches actively match components out of the value to make a new value, while some just passively match no-payload cases without matching anything. I wonder if it'd be too mystical to treat the borrowing cases here as if they did an implicit maybe = $originalValue to reinitialize the value to its original state, when the value is reinitialized on its consuming paths. (One could of course argue that this whole production is really working around our lack of inout pattern matching, and that that would be the ultimate better expression of this, though one thing an inout pattern match wouldn't directly be able to express is transitions between enum cases inside the switch case blocks.)

3 Likes

Yeah, I feel like we shouldn't make up special-case behavior just to make consuming switches pretend to be mutating ones.

If a mutating switch is over a local variable, we could definitely handle the transition case with local analysis. You'd just lose access to the variables bound by the pattern after you reassigned the original variable.

1 Like

Hi everyone. Based on the feedback in this thread, I've revised the proposal and implementation in top of tree. For noncopyable switches, switch consume x will still force a consuming switch on a variable x, but the default is now to do a borrowing switch on a variable or stored property reference. let bindings in patterns will copy, borrow, or consume based on the prevailing behavior of the switch.

If you've been experimenting with the existing implementation, code should mostly work as before. The one difference would be that consuming switches on variable bindings may need to change back to switch consume x in order to force a consuming switch of the current variable value.

5 Likes