`borrow` and `take` parameter ownership modifiers

Hi everyone. I'd like to revive a pitch from a while back to turn our __shared and __owned parameter ownership modifiers into an official language feature. During the recent review of SE-0366, for the move operator, the community raised the importance of considering these modifiers, and their naming, as part of the naming of the move operator itself. I like the idea of using the same name for the operator and the related parameter modifier, and I think take is a nice name for both, and borrow is also a good name both for the borrowing parameter modifier and for an operator analogous to move that forces a borrow-in-place without copying. So I'd like to start this new pitch thread there. Here is the draft PR, and I'll include the initial draft of the revised proposal below:

borrow and take parameter ownership modifiers

Introduction

We propose new borrow and take parameter modifiers to allow developers to explicitly choose the ownership convention that a function uses to receive immutable parameters. This allows for fine-tuning of performance by reducing the number of ARC calls or copies needed to call a function, and provides a necessary prerequisite feature for move-only types to specify whether a function consumes a move-only value or not.

Pitch threads:

Motivation

Swift uses automatic reference counting to manage the lifetimes of reference-counted objects. There are two broad conventions that the compiler uses to maintain memory safety when passing an object by value from a caller to a callee in a function call:

  • The callee can borrow the parameter. The caller guarantees that its argument object will stay alive for the duration of the call, and the callee does not need to release it (except to balance any additional retains it performs itself).
  • The callee can take the parameter. The callee becomes responsible for either releasing the parameter or passing ownership of it along somewhere else. If a caller doesn't want to give up its own ownership of its argument, it must retain the argument so that the callee can take the extra reference count.

These two conventions generalize to value types, where a "retain" becomes an independent copy of the value, and "release" the destruction and deallocation of the copy. By default Swift chooses which convention to use based on some rules informed by the typical behavior of Swift code: initializers and property setters are more likely to use their parameters to construct or update another value, so it is likely more efficient for them to take their parameters and forward ownership to the new value they construct. Other functions default to borrowing their parameters, since we have found this to be more efficient in most situations.

These choices typically work well, but aren't always optimal. Although the optimizer supports "function signature optimization" that can change the convention used by a function when it sees an opportunity to reduce overall ARC traffic, the circumstances in which we can automate this are limited. The ownership convention becomes part of the ABI for public API, so cannot be changed once established for ABI-stable libraries. The optimizer also does not try to optimize polymorphic interfaces, such as non-final class methods or protocol requirements. If a programmer wants behavior different from the default in these circumstances, there is currently no way to do so.

Looking to the future, as part of our ongoing project to add ownership to Swift, we will eventually have move-only values and types. Since move-only types do not have the ability to be copied, the distinction between the two conventions becomes an important part of the API contract: functions that borrow move-only values make temporary use of the value and leave it valid for further use, like reading from a file handle, whereas functions that take a move-only value consume it and prevent its further use, like closing a file handle. Relying on implicit selection of the parameter convention will not suffice for these types.

Proposed solution

We can give developers direct control over the ownership convention of parameters by introducing two new parameter modifiers borrow and take.

Detailed design

borrow and take become contextual keywords inside parameter type declarations. They can appear in the same places as the inout modifier, and are mutually exclusive with each other and with inout. In a func, subscript, or init declaration, they appear as follows:

func foo(_: borrow Foo)
func foo(_: take Foo)
func foo(_: inout Foo)

In a closure:

bar { (a: borrow Foo) in a.foo() }
bar { (a: take Foo) in a.foo() }
bar { (a: inout Foo) in a.foo() }

In a function type:

let f: (borrow Foo) -> Void = { a in a.foo() }
let f: (take Foo) -> Void = { a in a.foo() }
let f: (inout Foo) -> Void = { a in a.foo() }

TODO: How to apply these modifiers to self, the newValue of accessors, ...

take or borrow on a parameter do not affect the caller-side syntax for passing an argument to the affected declaration. For typical Swift code, adding, removing, or changing these modifiers does not have any source-breaking effects. (See "related directions" below for interactions with other language features being considered currently or in the near future which might interact with these modifiers in ways that cause them to break source.)

Protocol requirements can also use take and borrow, and the modifiers will affect the convention used by the generic interface to call the requirement. The requirement may still be satisfied by an implementation that uses different conventions for parameters of copyable types:

protocol P {
  func foo(x: take Foo, y: borrow Foo)
}

// These are valid conformances:

struct A: P {
  func foo(x: Foo, y: Foo)
}

struct B: P {
  func foo(x: borrow Foo, y: take Foo)
}

struct C: P {
  func foo(x: take Foo, y: borrow Foo)
}

Function values can also be implicitly converted to function types that change the convention of parameters of copyable types among unspecified, borrow, or take:

let f = { (a: Foo) in print(a) }

let g: (borrow Foo) -> Void = f
let h: (take Foo) -> Void = f

let f2: (Foo) -> Void = h

Source compatibility

Adding take or borrow to a parameter in the language today does not otherwise affect source compatibility. Callers can continue to call the function as normal, and the function body can use the parameter as it already does. A method with take or borrow modifiers on its parameters can still be used to satisfy a protocol requirement with different modifiers. The compiler will introduce implicit copies as needed to maintain the expected conventions. This allows for API authors to use take and borrow annotations to fine-tune the copying behavior of their implementations, without forcing clients to be aware of ownership to use the annotated APIs. Source-only packages can add, remove, or adjust these annotations on copyable types over time without breaking their clients.

This will change if we introduce features that limit the compiler's ability to implicitly copy values, such as move-only types, "no implicit copy" values or scopes, and take or borrow operators in expressions. Changing the parameter convention changes where copies may be necessary to perform the call. Passing an uncopyable value as a take argument ends its lifetime, and that value cannot be used again after it's taken.

Effect on ABI stability

take or borrow affects the ABI-level calling convention and cannot be changed without breaking ABI-stable libraries.

Effect on API resilience

take or borrow break ABI for ABI-stable libraries, but are intended to have minimal impact on source-level API. When using copyable types, adding or changing these annotations to an API should not affect its existing clients.

Alternatives considered

Naming

We have considered alternative naming schemes for these modifiers:

  • The current implementation in the compiler uses __shared and __owned, and we could remove the underscores to make these simply shared and owned. These names refer to the way a borrowed parameter receives a "shared" borrow (as opposed to the "exclusive" borrow on an inout parameter), whereas a taken parameter becomes "owned" by the callee. We found that the "shared" versus "exclusive" language for discussing borrows, while technically correct, is unnecessarily confusing for explaining the model.
  • A previous pitch used the names nonconsuming and consuming.

The names take and borrow arose during the first review of SE-0366. These names also work well as names for operators that explicitly transfer ownership of a variable or borrow it in place, discussed below as the take and borrow operators under Related Directions. We think it is helpful to align the naming of those operators with the naming of these parameter modifiers, since it helps reinforce the relationship between the calling conventions and the expression operators: to explicitly transfer ownership of an argument in a call site to a parameter in a function, use foo(take x) at the call site, and use func foo(_: take T) in the function declaration.

Effect on call sites and uses of the parameter

This proposal designs the take and borrow modifiers to have minimal source impact when applied to parameters, on the expectation that, in typical Swift code that isn't using move-only types or other copy-controlling features, adjusting the convention is a useful optimization on its own without otherwise changing the programming model, letting the optimizer automatically minimize copies once the convention is manually optimized.

It could alternatively be argued that explicitly stating the convention for a value argument indicates that the developer is interested in guaranteeing that the optimization occurs, and having the annotation imply changed behavior at call sites or inside the function definition, such as disabling implicit copies of the parameter inside the function, or implicitly taking an argument to a take parameter and ending its lifetime inside the caller after the
call site. We believe that it is better to keep the behavior of the call in expressions independent of the declaration (to the degree possible with implicitly copyable values), and that explicit operators on the call site can be used in the important, but relatively rare, cases where the default optimizer behavior is insufficient to get optimal code.

Applying borrow and take modifiers to the self parameter of methods

This proposal does not yet specify how to control the calling convention of the self parameter for methods. As currently implemented, the __consuming modifier can be applied to the method declaration to make self be taken, similar to how the mutating method modifier makes self be inout. We could continue that scheme with new function-level modifiers:

struct Foo {
  mutating func mutate() // self is inout
  taking func take() // self is take
  borrowing func borrow() // self is borrow
}

Alternatively, we could explore schemes to allow the self parameter to be declared explicitly, which would allow for the take and borrow modifiers as proposed for other parameters to also be applied to an explicit self parameter declaration.

Related directions

take operator

Currently under review as SE-0366, it is useful to have an operator that explicitly ends the lifetime of a variable before the end of its scope. This allows the compiler to reliably destroy the value of the variable, or transfer ownership, at the point of its last use, without depending on optimization and vague ARC optimizer rules. When the lifetime of the variable ends in an argument to a take parameter, then we can transfer ownership to the callee without any copies:

func consume(x: take Foo)

func produce() {
  let x = Foo()
  consume(x: take x)
  doOtherStuffNotInvolvingX()
}

borrow operator

Relatedly, there are circumstances where the compiler defaults to copying when it is theoretically possible to borrow, particularly when working with shared mutable state such as global or static variables, escaped closure captures, and class stored properties. The compiler does this to avoid running afoul of the law of exclusivity with mutations. In the example below, if callUseFoo() passed global to useFoo by borrow instead of passing a copy, then the mutation of global inside of useFoo would trigger a dynamic exclusivity failure (or UB if exclusivity checks are disabled):

var global = Foo()

func useFoo(x: borrow Foo) {
  // We need exclusive access to `global` here
  global = Foo()
}

func callUseFoo() {
  // callUseFoo doesn't know whether `useFoo` accesses global,
  // so we want to avoid imposing shared access to it for longer
  // than necessary, and we'll pass a copy of the value. This:
  useFoo(x: global)

  // will compile more like:

  /*
  let globalCopy = copy(global)
  useFoo(x: globalCopy)
  destroy(globalCopy)
   */
}

It is difficult for the compiler to conclusively prove that there aren't potential interfering writes to shared mutable state, so although it may in theory eliminate the defensive copy if it proves that useFoo, it is unlikely to do so in practice. The developer may know that the program will not attempt to modify the same object or global variable during a call, and want to suppress this copy. An explicit borrow operator could allow for this:

var global = Foo()

func useFooWithoutTouchingGlobal(x: borrow Foo) {
  /* global not used here */
}

func callUseFoo() {
  // The programmer knows that `useFooWithoutTouchingGlobal` won't
  // touch `global`, so we'd like to pass it without copying
  useFooWithoutTouchingGlobal(x: borrow global)
}

If useFooWithoutTouchingGlobal did in fact attempt to mutate global while the caller is borrowing it, an exclusivity failure would be raised.

Move-only types, uncopyable values, and related features

The take versus borrow distinction becomes much more important and prominent for values that cannot be implicitly copied. We have plans to introduce move-only types, whose values are never copyable, as well as attributes that suppress the compiler's implicit copying behavior selectively for particular variables or scopes. Operations that borrow a value allow the same value to continue being used, whereas operations that take a value destroy it and prevent its continued use. This makes the convention used for move-only parameters a much more important part of their API contract:

moveonly struct FileHandle { ... }

// Operations that open a file handle return new FileHandle values
func open(path: FilePath) throws -> FileHandle

// Operations that operate on an open file handle and leave it open
// borrow the FileHandle
func read(from: borrow FileHandle) throws -> Data

// Operations that close the file handle and make it unusable take
// the FileHandle
func close(file: take FileHandle)

func hackPasswords() throws -> HackedPasswords {
  let fd = try open(path: "/etc/passwd")
  // `read` borrows fd, so we can continue using it after
  let contents = try read(from: fd)
  // `close` takes fd from us, so we can't use it again
  close(fd)

  let moreContents = try read(from: fd) // compiler error: use after take

  return hackPasswordData(contents)
}

Acknowledgments

Thanks to Robert Widmann for the original underscored implementation of __owned and __shared: https://forums.swift.org/t/ownership-annotations/11276.

47 Likes

Exciting!

I understand why borrow gives you an immutable object (mutable borrow is inout), and I understand how a taken value can be a read-only value when we look at copyable types, but it doesn't work in my head for move-only types. Shouldn't taken objects be mutable? If you don't need to modify a move-only value, why wouldn't you borrow it instead, aside from the comparatively rare case of explicitly ending its lifetime?

Effectively, we have two axes (borrowed/taken and mutable/immutable, comparable to by value/by reference and const/mutable in C++), but just 3 choices (borrowed+mutable, borrowed+immutable, taken+immutable) and I'd naĂŻvely think that taken+mutable would be more useful.

How do borrow/take apply to closures (and what does that mean for @escaping)?

2 Likes
  1. In a world where SE-0366 is implemented, how does move relate to borrow and take? It seemed on that discussion that __owned and move were not mutually exclusive and that __shared and move were not mutually exclusive. Is that correct?

  2. On the move pitch I suggested move be spelled moved for the aesthetic symmetry with __owned and __shared, and I appreciate that the proposed take and borrow give that symmetry with move. The proposed modifiers for methods really highlights how inout is the oddball and does make me wonder if there’s a case for changing the spelling of inout to mutate to bring it to parity with these new names. That alignment would obviously be a source-breaking change, but I’m not versed on what exactly makes a change ABI-breaking, so I don’t know if a spelling change that doesn’t change what’s generated is ABI-breaking, but my guess would be no.

Exciting! Will read through it all soon, but wanted to give you a headsup that something seems to be wrong with line breaks?

It is quite hard to read this pitch on mobile, looks like this:

5 Likes

This looks great @Joe_Groff. I think the naming here is clear and the proposed semantics are reasonable, so this is a strong +1.

Also +1 from my side. I really like the new naming and overall this flows way better to explain in written and spoken language!

Given the position of these keywords before a concrete type name, I would suggest borrowed and taken to make it more readable.

Also +1 on renaming inout accordingly.

3 Likes

I think this has been brought up before, but isn’t @escaping on function parameters similar to take? From my understanding, a non-escaping function doesn’t escape the lifetime of the callee, and is this guaranteed to remain in the same position, hence why non-escaping closures can be stack allocated. This seems similar to borrow. Escaping closures, on the other hand, escape their callee’s lifetime and thus have to be heap allocated, so that they’re not tied to the call stack. Of course, even if the caller moved the escaping function into the callee, the closure would still escape. I’m just curious if escaping closures have any overlap with other ownership features.

I see non-escaping closures as being borrow+nocopy, and escaping ones as take+copyable. Maybe there are more use cases that could benefit from a borrow+nocopy parameter guaranty.


A thought: if we had a generic function accepting move-only types, non-escaping could be a way to accept them without having to label the generic type as move-only. For instance:

func test(param: noescape some Collection) { ... }

could accept move-only collections because the parameter is borrow+nocopy. A simple borrow would not be sufficient unless the generic type constraints include the nocopy restriction in another way, as types are normally assumed to allow copying.

1 Like

I'll tell you what feels the most awkward to me - that these are verbs, not adjectives.

Having borrow String or take any Collection<Int> in parameter lists just feels weird. Like I'm writing a recipe, not a declaration. What do we call them? "Take parameters"?

I also think names like "take" are optimising for brevity, but end up being cryptic. I'd prefer something which refers more strongly to "lifetime" or "ownership". The concept of borrowing, for example, makes little sense unless contrasted with ownership.

6 Likes

If we are making public parameters modifier usage ABI then why not require conformances to a protocol to only be satisfied by matching parameter modifiers?. In addition why not also require that the use sites add matching sigils ( in the same way that inout requires & at the use site )

Perhaps I am missing something.

1 Like

I think you're right that it'll be common when you're taking a parameter to want to turn around and mutate it in-place. Back in the early days of Swift, we allowed var as a parameter modifier, but had removed it out of concerns it would be confusing next to inout, but maybe we could bring that back?

func foo(var x: take [Int]) {
  ...
}

Nonescaping closures are effectively already always uncopyable and passed inout. It wouldn't be allowed to take or borrow a nonescaping closure. Escaping closures shouldn't need any special consideration, they could be passed with either convention. The call operation on an escaping closure is currently always borrow, so there would be limited benefit to passing by take at this point, except to transfer ownership from one owner to another without retains.

After the SE-0366 discussion, I think there's a lot of benefit to having the name of the "move operator" align with the "take" modifier, so we'd call it take x instead. That makes it so that, if you want to guarantee a "perfect forwarding" chain that passes a value along without copies, you can use take on the declaration-side parameters and call-site-side arguments along the path:

func foo(x: take T)

func bar(y: take T) {
  foo(x: take y)
}

func bas(z: take T) {
  bar(y: take z)
}

let w = T()
doStuff(with: w)
bas(z: take w)

You're right that @nonescaping is, if not exactly the same as borrowing a move-only value, very closely related. For @escaping closure parameters, though, the "escape" has already happened when the closure is formed, and once you have a closure value, there isn't very much different about it from an ownership perspective from other reference types in Swift. Your function might invoke the closure, which is a borrowing operation, and/or it might store the closure in a new owning container, in which case it might be beneficial to take the parameter to avoid a retain. So it's worth calling out that borrow/take/inout don't really make sense for nonescaping closure arguments, but I don't think escaping closures need special consideration.

Sorry, I'll fix that.

A goal of this current design is to minimize the source-level impact of using these modifiers on "typical" Swift that isn't using move-only types or manual performance control features. A protocol can apply ownership modifiers to its requirement parameters to allow for the compiler's function signature optimization to make the implementer's methods match them automatically on the other side of the protocol abstraction. For instance, the implementation of an append method on RangeReplaceableCollection more than likely could benefit from being able to take ownership of its argument to move it into the collection. If a protocol wants to accommodate both move-only and copyable conformers in the future, then it may need to have specific ownership on parameters to make move-only conformers possible at all; looking at append again, a move-only collection implementation would need to be able to take append's argument in order to move it into the collection without copying. Accepting modifier mismatches allows for protocols to accommodate move-only implementations without changing the programming model for copyable implementations.

4 Likes

borrowed and taken then.
(also "mutable" for "inout" for consistency).

1 Like

I do want to voice a small concern over the take spelling because of how developers describe functions today as “taking arguments” (sometimes might be called “accepting arguments”).

func foo(_: borrow Foo)
func foo(_: take Foo)
func foo(_: inout Foo)

These 3 functions could be described as each taking 1 argument. But only 1 takes a take argument. I’d also describe the first as taking a borrow argument.

If one was having the conversation in a spoken format, I could envision a real Who’s on first kind of situation with take arguments.

To stick with the notion of physical ownership, spelling it as transfer would help with the situation. Not sure if other nuances of that spelling have been discussed.

Then it reads like this and clears up the ambiguity with the existing use of take.

  • The callee can borrow the parameter.
  • The callee can transfer the parameter.
18 Likes

It's also worth remembering that there are plans for borrow variables, so however we end up referring to these in parameter lists should work in standalone code. I think it is worth calling that out specifically in "Related directions", because it is a very direct extension of these concepts.

Borrow variables are important; otherwise working with write-through views becomes extremely verbose. I think people will likely be exposed to borrow variables (especially mutable borrows) more often than borrow parameters.

url.pathComponents.insert(prefix, at: url.pathComponents.index(after: url.pathComponents.startIndex))
^^^^^^^^^^^^^^^^^^                    ^^^^^^^^^^^^^^^^^^              ^^^^^^^^^^^^^^^^^^

// vs

ref path = url.pathComponents
path.insert(prefix, at: path.index(after: path.startIndex))

// + "take path"? "endLifetime path"? We'd need to end the borrow somehow.
// A discussion for another time, but worth keeping in mind.

The ref name proposed in the ARC thread is also interesting. I think you could make an argument that the difference between a String and ref String can be made more obvious to a newcomer than the difference between a String and borrowed String; the idea of a "reference" has a somewhat stronger implication that I don't quite have a String, and that this variable is tied to something which exists externally (to be precise - its lifetime is bound by that external thing). You could also argue that it is potentially confusable with a "reference type", which is fair.

I think this would be a regression. Wondering if inout needs new alignment so we can complete the matrix of [immutable, mutable] x [borrow, take]

mutable borrow &borrow ( today inout)
immutable borrow borrow
mutable take &take
immutable take take

I think clearly the answer is to call these yoink and yeet.

46 Likes

I think that would be a mistake. Whether the function mutates the parameter or not is irrelevant to the caller, so it should not be part of the signature. Functions signatures can get complex enough already.

1 Like

That's not quite what the matrix is we're talking about. Whether the function outwardly receives the parameter by immutable borrow/mutable borrow/take is mostly independent of whether it inwardly models the parameter binding as a let or var, with maybe the exception that it isn't very useful to receive an inout parameter and bind it to a let.

Without any additional language features beyond what's proposed, note that you could still do this to put your taken parameter in a locally mutable variable:

func foo(x: take [Int]) {
  var myX = take x
  modifyInPlace(&myX)
}
1 Like

Personally it won't even occur to me to need borrowing here. I'd add some sugar and convert that into:

url.pathComponents[insertAt: 1] = prefix // or
url.pathComponents[1, insert: true] = prefix

(here "1" is a short version of path.index(after: path.startIndex), and insertAt is a simple variant of a subscript that inserts instead of replacing.