Clarifying questions regarding `consuming` parameters and the `consume` operator

Development Compiler
1

i am wondering about a few recently-filed issues & topics relating to variable lifetime rules and diagnostics.

the first is Compiler fails to prevent consuming parameter implicitly copied · Issue #83393 · swiftlang/swift · GitHub, which has this example from SE-377:

func foo(x: consuming String) -> (String, String) {
    return (x, x)
}

the evolution document and issue both suggest that this pattern would implicitly copy the parameter x when constructing the return value, and so should produce a diagnostic. this appears to be the case in older compilers, but no longer seems true on the 6.2 branch (example of 6.1 vs 6.2).

is the behavior in the 6.2 compiler expected?

the other question is in regards to Compiler should produce error when consuming a variable captured by an `@escaping` closure · Issue #83282 · swiftlang/swift · GitHub, which alludes to this portion of SE-366:

The operand to consume is required to be a reference to a binding with static lifetime. The following kinds of declarations can currently be referenced as bindings with static lifetime:

  • a local let constant in the immediately-enclosing function,
  • a local var variable in the immediately-enclosing function,
  • one of the immediately-enclosing function's parameters, or
  • the self parameter in a mutating or __consuming method.

A binding with static lifetime also must satisfy the following requirements:

  • it cannot be captured by an @escaping closure or nested function,
  • it cannot have any property wrappers applied,
  • it cannot have any accessors attached, such as get, set, didSet, willSet, _read, or _modify,
  • it cannot be an async let.

the question is whether, in an example like this:

func escape(_ c: @escaping () -> Void) {}

func test() {
    let x = ""
    escape { _ = x }
    _ = consume x
}

should a diagnostic be produced? since x is captured in an escaping closure, it would seem to not satisfy the requirements for being a binding with static lifetime, and should therefore not be allowed to be consume'd. is that the correct interpretation here?

2

For your second question, I think your test() function is valid, because String is copyable, the compiler can make an implicit copy of x when the closure captures it.

However, the compiler fails to produce diagnostics when x is a non-copyable:


func escape(_ c: @escaping () -> Void) {}

struct NC: ~Copyable {}
func test() {
    let x = NC()
    escape { _ = x }
    _ = consume x   // currently OK, but should not be 
}

One of the evolution explicitly mentioned the above case, that a escaping-ly captured value cannot be consumed neither inside nor outside the closure. So I believe there is definitely a miss.

2 Likes
3

But that can't explain why compiler fails to produce diagnostic when x is mutable.

func escape(_ c: @escaping () -> Void) {}

func test() {
    var x = ""
    escape { _ = x }
    _ = consume x
}

EDIT: I find an odd behavior. Compiler produces diagnostic if x is actually modified in the escaped closure. I think compiler should make decision based on a function's signature, instead of what it actually does?

func escape(_ c: @escaping () -> Void) {}

func test() {
    var x = ""
    escape { x = "a" } // modify x
    _ = consume x
}

Also, the sample below demonstrates how it can lead to data race:

func escape(_ c: @escaping () -> Void) {}

func test() {
    var x = ""
    Task { print(x) } // x's storage is read
    var y = consume x
    Task { y = "a" } // the same storage is written simultaneously
}

I'll update #83282 with these information.

4

that seems like a plausible interpretation, but the documentation in the evolution proposals leaves it somewhat ambiguous IMO[1]. is there a semantically 'new' binding that is implicitly created at the point that x is captured by the escaping closure, and that binding has a different lifetime? or is the use of the explicit consume operator supposed to 'promote' the local binding to be treated with the same rules as a ~Copyable type or @_noImplicitCopy parameter would receive (and if so, is that supposed to apply 'retroactively' back to the point the binding was created)?

FWIW, this specific example appears to be diagnosed on main, but not in 6.2 (and earlier).

won't the mutable variable captures in this case have independent storage? IIUC they'll each essentially be a different 'boxed allocation'. given that x isn't ever mutated, this seems like it may be fine (from a data race perspective – the consumption of an escaped binding seems like a bug)?

  1. or i've just not located where this is made explicit ↩︎

5

consume ends the lifetime of the source binding. As both of x & y are local to your function, the consume is just a rename operation and has no requirement to actually produce another variable in memory. Each task almost certainly reads or writes to the same location in memory.

This would be less obviously true if Swift required each object to have a distinct address in memory as in C or Rust, but Swift doesn't even require that a given object has a stable address in memory (class types and allocated pointers notwithstanding).

6

i'm now fairly certain the second case is a bug. SE-366 says:

consume consumes the current value of a binding with static lifetime, which is either an unescaped local let, unescaped local var, or function parameter, with no property wrappers or get/set/read/modify/etc.

which makes it pretty clear that consume should not be applicable in this case since the local binding escaped.

now, as to what to do about it... most of the diagnostics that i've seen for handling consumption of copyable types are in the ConsumeOperatorCopyable*Checker classes. however there's also the DiagnoseInvalidEscapingCaptures logic, which may be a reasonable place to check for this. i guess my intuition would be that the ConsumeOperator* passes are probably more fitting, but curious if anyone has any insights on this.

to digress from the original topic a bit, i don't really see why it would be likely that the Tasks would read & write to the same memory location in a case like this – would you care to elaborate on why you think that would be? is it dependent on the use of the consume operator here (which as stated earlier, i think is invalid)? or is it due to some optimizations that you'd expect that?

7

Like I said, there's no stable addresses, so (as x is dead after the consume) there's no reason a memcpy ever needs to be emitted here. x and y are perfectly free to share the same place in memory. The tasks being allowed to access both when one is potentially uninitialized seems broken though. I was forgetting the tasks don't have to run in order.

8

(I hope I didn't spam the forum with my owership posts. I have been banging my head against the few owership proposals over the weekend and finally come up with a reasonable explanation. I think it might be worth sharing it.)

TLDR: I think the above code is correct, because an implicit copy is made.

I'll first look at a few simpler examples and then explain the above case. For simplicity I'll use escape function, instead of Task. All the code are tested on nightly build.

func escape(_ c: @escaping () -> Void) {}

1. Understand Ownership by Example

  1. Value is copyable and immutable. It's dynamically borrowed and consumed. So an implicit copy is made (see SE-0377).

    func test() {
    let x = "" // x is immutable
    escape { _ = x }
    _ = consume x
}

  2. Value is non-copyable and immutable. It's dynamically borrowed and consumed. This is an error (see SE-0390).

    func test(_ x: consuming String) {
    escape { print(x) }
    _ = consume x
}

    or:

    struct NC: ~Copyable {}
func test() {
    let x = NC()
    escape { _ = x }
    _ = consume x
}

  3. Value is copyable and mutable. It's dynamically mutated and consumed. This is an error (while SE-390 is about non-copyable, I think it applies to this case, because an implicit copy can't be made in this case).

    func test() {
    var x = ""
    escape { x = "a" }
    _ = consume x
}

    or

    func test() {
    var x = ""
    escape { print(x) } // this is dynamically muated because an implicit copy can't be made?
    x = "abc"
    _ = consume x
}

  4. Value is non-copyable and mutable. It's dynamically mutated and consumed. This is an error.

    func test(_ x: consuming String) {
    escape { x = "a" }
    _ = consume x
}

2. A Special Case: a 'var' variable that isn't actually mutated is considered immutable

This looks like case 3, except that x isn't actually mutated in the code. Unlike case 3, it compiles. I guess what happens is that, as compiler knows x isn't mutated (it actually generates a warning on this), escape dynamically borrows x rather than `muate it. It's valid scenario to create an implicit copy when a values is both borrowed and consumed, so the code compiles.

func test() {
    var x = ""
    escape { print(x) }
    _ = consume x
}

That said, I still think the behavior is confusing. I'd hope the code fail and compiler output a note suggesting user to change var to let. The problem with the current behavior is that it could mislead user. For example, suppose a user is developing a framework, using the approach in case 3. The approach doesn't work, but as the user is writing the code in progressive way, the code appears to compile at first, until the user starts to mutate the variable. IMO if a variable is declaured as var, it means it could be modified in future. Compiler shouldn't silently changes it to let just because it's not mutated in the current code yet.

Maybe I should file a bug about this?

3. Case Study: Why the Original Example is Correct

In the original example, value x is copyable and mutable. As x isn't actually mutated, it's dynamically borrowed (see last section) and consumed. So an implicit copy is made and dynamically borrowed by escape. The original copy is consumed and its storage is transferred to y, which is dynamically muatated in another escape. As the two escaping closures work on separate values, there is no error.

func test() {
    var x = ""
    escape { print(x) } 
    var y = consume x
    escape { y = "a" } 
}

Hope it helps. I'm going to close #83282.

EDIT: as @Joe_Groff has confirmed that #83282 is indeed a bug, my hypothesis in section 2 (that a mutable x is dynamically borrowed rather than mutated by an escaping closure, if x isn't actually mutated) no longer holds. I believe a var variable is always dynamically mutated when it's captured by an escaping closure. As a result, my analysis in section 3 is invalid too.

1 Like
9

i largely agree with your analysis, but one thing that still bothers me is that the various proposals specifically say that consume applies only to unescaped bindings (a.k.a. bindings with 'static lifetime'). closer inspection of SE-366 shows there is a section in 'Future Directions' regarding loosening this to accommodate consumption of bindings with dynamic lifetime, but it's unclear if this was implemented or remains a potential future feature.

now perhaps we're supposed to conclude that an implicit immutable capture of a mutable variable binding is a distinct binding & lifetime, e.g.

func test() {
  var x = ""
  escape { print(x) } // implicit immutable capture since x never written
  _ = consume x
}

is equivalent to

func test() {
  var x = ""
  escape { [x] in print(x) } // explicit immutable capture
  _ = consume x
}

and so the binding in the capture should be thought of as having a distinct lifetime. but even if so, the proposal wording still implies that consume shouldn't apply to escaped local lets either, so it's still unclear to me what the intended behavior is.

personally, i feel like i'm unlikely to arrive at the correct interpretation based on the evolution proposals and forum discussions without some further insights from the proposal authors or implementors familiar with the intended behavior of the feature.

@Michael_Gottesman @Joe_Groff @Andrew_Trick – if you have a moment, would one of you care to enlighten us as to how one should think about the consumption of escaped bindings of copyable types as it relates to the definition of 'bindings of static lifetime'?

Relevant quotes from SE-366

consume consumes the current value of a binding with static lifetime, which is either an unescaped local let, unescaped local var, or function parameter, with no property wrappers or get/set/read/modify/etc.

The operand to consume is required to be a reference to a binding with static lifetime. The following kinds of declarations can currently be referenced as bindings with static lifetime:

  • a local let constant in the immediately-enclosing function,
  • a local var variable in the immediately-enclosing function,
  • one of the immediately-enclosing function's parameters, or
  • the self parameter in a mutating or __consuming method.

A binding with static lifetime also must satisfy the following requirements:

  • it cannot be captured by an @escaping closure or nested function,
  • it cannot have any property wrappers applied,
  • it cannot have any accessors attached, such as get, set, didSet, willSet, _read, or _modify,
  • it cannot be an async let.
1 Like
10

Escaped variables don’t have static lifetimes, so it isn’t meaningful to try to consume one. It would make the most sense to me for it to be an error.

3 Likes
11

I think I understand why that is generally true. Just want to point out that since the closure in the defer statement is escaping, the following code isn't allowed:

// It doesn't have deinit, so its fd isn't closed automatically when the value is out of scope.
struct File: ~Copyable {
  private var fd: Int32
  
  init(fd: Int32) { self.fd = fd }

  consuming func close() {
    print("closing file")
  }
}

func test() {
    let file = File(fd: 1)
    defer {
        file.close() // error: noncopyable 'file' cannot be consumed when captured by an escaping closure or borrowed by a non-Escapable type
    }

    // ... perform operations with the file ...
}
12

A defer closure is not escaping. Nonescaping closures can in full generality run multiple times, so can only borrow their captures. The compiler does not yet understand that certain closures run at most once.

2 Likes
13

Just curious, suppose compiler was able to tell that a defer closure only runs once, how would the above code work? According to SE-0390, nonescaping closure captures an immutablve value as a borrowing operation and a mutable value as a mutating operation. There isn't a way to capture a value as a consuming operation.