Calling borrowing func on existential results in consumption of value

Using Swift
1

Hello, I'm encountering an issue with calling a borrowing method on a stored property of an existential type (any Protocol & ~Copyable) erroneously consumes the property, preventing subsequent use. I have found a thread describing the same behavior, but it received no responses. Is this intended behavior or a compiler limitation/bug?

Example:

protocol Buffer: ~Copyable {
    borrowing func bind()
}

struct Renderer: ~Copyable {
    let buffer: any Buffer & ~Copyable
    
    mutating func render() {
        buffer.bind()  
        buffer.bind() 
    }
}

Error:

test.swift:8:19: error: 'self' consumed more than once
 6 |     let buffer: any Buffer & ~Copyable
 7 |     
 8 |     mutating func render() {
   |                   `- error: 'self' consumed more than once
 9 |         buffer.bind()  
   |         `- note: consumed here
10 |         buffer.bind() 
   |         `- note: consumed again here
11 |     }
12 | }

test.swift:8:19: error: missing reinitialization of inout parameter 'self' after consume
 6 |     let buffer: any Buffer & ~Copyable
 7 |     
 8 |     mutating func render() {
   |                   `- error: missing reinitialization of inout parameter 'self' after consume
 9 |         buffer.bind()  
10 |         buffer.bind() 
   |         `- note: consumed here
11 |     }
12 | }

Is there any way to get around this? Thanks in advance.

SE-0517: UniqueBox
2

Though I never got a reply for my topic, I guess this is a bug.

I never published that as an issue on GitHub since had an immediate workaround and did not have a plan to use it this way in perspective.
But I think filing an issue to swift lang on github would help move the issue forward and resolved at some point or/and get a proper reply from compiler devs...

2 Likes
3

I think you’re probably right, as I don’t see any reason for the current behaviour. I guess I’ll file a bug report.

1 Like
4

here are a few possible workarounds. the first is to consume the field (and self), perform the calls on the local binding, and then reinitialize self before returning:

struct Renderer: ~Copyable {
    var buffer: any Buffer & ~Copyable
    
    mutating func render() {
        let buf = consume self.buffer

        buf.bind()
        buf.bind()

        self = Renderer(buffer: buf)
    }
}

this does require changing from a let to a var for the stored property (you hit a different compiler bug otherwise).

the next is similar, though maybe has slightly worse tradeoffs depending on your view. if you make the field optional then you can do a similar 'temporary detachment' of the value and re-assign it before returning.

struct Renderer2: ~Copyable {
    var buffer: (any Buffer & ~Copyable)!
    
    mutating func render() {
        let buf = self.buffer.take()!

        buf.bind()
        buf.bind()

        self.buffer = consume buf
    }
}

the compiler won't stop you if you forget to re-assign the property though in this case.

lastly, you could make the container generic over the stored property's type, which will then 'just work':

struct Renderer3<Buf: Buffer & ~Copyable>: ~Copyable {
    let buffer: Buf
    
    mutating func render() {
        buffer.bind()
        buffer.bind()
    }
}

not sure if these are exhaustive options, but that's what came to me. you can play around with them here: Compiler Explorer.

it's also not clear to me if this restriction is necessary or a current limitation, and at the least the diagnostics could probably be more illuminating, so i also support your filling an issue for this.

2 Likes
5

As far as I'm aware, it should not be possible to construct an existential with a suppressed conformance to ~Copyable. That sounds like the bug to me.

The act of creating an existential is a consuming operation (it must move the value's data into an existential box.) The act of mutating the value is therefore consuming because it must place the mutated value into a new existential box (or a copy of the old one.)

6

That sounds disappointing. I am not very into details of existential creation, but to me that looks a bit of technical details of existential implementation rather than expected behavior.

For example, I am using non-copyable existential for modeling an API where end-user doesn't know about concrete type but it is supposed to be handled as non-copyable (see Calling borrowing method of stored property existential leads to consume of self)

7

this seems at odds with the evolution docs. it contains this example:

protocol Pizza: ~Copyable {}
struct UniquePizza: Pizza, ~Copyable {}

let t: any Pizza /* & Copyable */ = UniquePizza()  // error
let _: any Pizza & ~Copyable = UniquePizza()  // ok
1 Like
8

Yes, sorry, I wasn't clear: while it's de jure allowed by the docs, I'm saying I think it shouldn't be possible to do it because of these sorts of constraints.

9

If possible I'd use generics instead of existentials. Generics can be significantly faster because the compiler can see what types you use with them and potentially emit specialized copies where useful.

If generics aren't an option, I would consider instead using a class hierarchy for this sort of thing (like what AnySequence does under the covers):

private class AnyBuffer: Buffer {
  init() {}

  func bind() {
    fatalError("Subclass responsibility")
  }
}

private final class ConcreteBuffer<T: Buffer & ~Copyable>: AnyBuffer {
  var buffer: T

  init(_ buffer: consuming T) {
    self.buffer = buffer
    super.init()
  }

  override func bind() {
    buffer.bind()
  }
}

struct Renderer: ~Copyable {
  private var buffer: AnyBuffer

  init<T: Buffer & ~Copyable>(_ buffer: consuming T) {
    buffer = ConcreteBuffer(buffer)
  }

  mutating func render() {
    buffer.bind()
    buffer.bind()
  }
}

But again, generics would be preferable!

1 Like
10

Unfortunately, this is not the case when protocol and implementation are located in different projects. One is defining API and second is implementing the runtime.

However, I should say that I did not notice any significient overhead in a pretty loaded environment.
I still have traces and see some insignificant overhead of virtual call (aka dispatching trunk) and metadata accessor but that doesn't look awful. I.e. there are no any calls to swift_getGenericMetadata or any other time consuming runtime/system things.

Could you suggest if I understand right with the example below?

protocol SomeNonCopyable: ~Copyable {
    func foo1(...) {}
    func foo2(...) {}
    ...
    func fooN(...) {}
}

struct AnySomeNonCopyable: ~Copyable {
    private let value: any (SomeNonCopyable & ~Copyable)

    func foo1(...) { value.foo1(...) }
    func foo2(...) { value.foo2(...) }
    ...
    func fooN(...) { value.fooN(...) }
}

Or let me know if I missed the point..

11

See my example. The point was to avoid using an existential with a move-only value, and to instead use a private inner class for type erasure.

12

The act of mutating the value is therefore consuming because it must place the mutated value into a new existential box (or a copy of the old one.)

Why is this the case? Can’t the boxed value simply be mutated (or borrowed) in place? This is possible in Rust with the equivalent Box<dyn Trait>:

If possible I'd use generics instead of existentials.

Unfortunately this isn’t possible for me, as the rendering API is chosen at runtime, so the buffer could be of OpenGL, etc, etc. I guess I’ll have to use a private inner class for this?

13

I suspect that it's mostly a matter of compiler work to support mutation-in-place of an existential that hasn't been done to date.

My gut also says (and please don't take this as authoritative in any way) that mutation of an existential in-place would admit violating Swift's Law of Exclusivity if that existential refers to a move-only value and that value has been heap-promoted for being too large to fit inline. A compiler engineer might be able to correct me here though.

14

Maybe I missed the feature with class overrides (should it be open class then?) and they still can be defined as non-copyable for the end user of API.

But if not, I am not sure about the class in sense of modeling an API because then if I want to make the object non-copyable I still should wrap it inside an additional struct to model it as non-copyable object.

I have a bit different thing and I would like to model an object that is sendable and non-copyable so that end user of API can manipulate with it within a single task (thread) but free to move and use in other if needed. Similarly to the principle of fearless concurrency.

For your example it would mean an additional layer after AnyBuffer, i.e.:

struct UserSurfacingBuffer: ~Copyable, Sendable { // would be UserSurfacingShard in my case
     let anyBuffer: AnyBuffer // would be AnyShard with all wrapping things similar to buffer

     mutating func bind() { self.anyBuffer.bind() }
}

Though that may be a bit of work, I wonder if this approach with classes (and maybe implementation in other binary as class directly) plus using a user wrapping structure would be beneficial performance-wise. This is an interesting question that worth measuring in both combinations in my case.

However, my point was that it is rather surprising that existentials produce such unexpected side effects with non-copyable struct rather than performance considerations.

15

Please re-read the code snippet I shared. :slightly_smiling_face:

16

If the existential can hold a non-copyable value, then surely the existential must itself be non-copyable. I can't think of any reason why you couldn't mutate it in place provided you have exclusive access to the existential.

1 Like
17

As I said, I suspect this is just a matter of the compiler work not having been done to support mutation-in-place.

18

To solve my issue, I wrote a (not perfect) macro to generate the type erased inner classes from a protocol definition. Maybe this will help @blindspotbounty (certainly will help me).

Macro declarations 
@attached(peer, names: prefixed(Any))
/// Generate a struct with the name `Any$Protocol`, with two private inner classes:
/// - `Erased`
/// - `Concrete<T>: Erased`
///
/// The struct stores an instance of `Concrete`, cast to `Erased`, so an instance of any type conforming
/// to the protocol can be passed around as an `Any$Protocol`. An instance of the struct can be created using `Any$Protocol(from:)`,
/// or any one of the initialisers of the protocol, with an extra parameter of `T: Protocol>: `of type: T.Type`.
///
/// Static members are not supported, use `AnyErasedProtocolClass` for that (with the caveat that `type(of:)` must be used to
/// get the metatype of the `Concrete` class type.
internal macro AnyErasedProtocolStruct() =
	#externalMacro(module: "AnyErasedProtocolMacro", type: "AnyErasedProtocolMacro")
@attached(peer, names: prefixed(Any))
/// Generate a class with the name `Any$ProtocolName`, with a private inner class `Concrete<T>: Erased`.
///
/// The stores an instance of `Concrete`, cast to `Erased`, so an instance of any type conforming
/// to the protocol can be passed around as an `Any$Protocol`. An instance of erased can be created using `Any$Protocol.create(from:)`.
internal macro AnyErasedProtocolClass() =
	#externalMacro(module: "AnyErasedProtocolMacro", type: "AnyErasedProtocolMacro")

The implementation is here:

Example input 
@AnyErasedProtocolStruct
protocol IndexBuffer: ~Copyable {
   init(indices: [UInt32], loader: Loader)
   
   borrowing func bind()
   borrowing func unbind()

   var count: Int { borrowing get }
}

Generates:

struct AnyIndexBuffer: ~Copyable {
    private let inner: Erased
    init(from inner: consuming some ~Copyable & IndexBuffer) {
        self.inner = Concrete(from: inner)
    }
    private class Erased {
        borrowing func bind() {
            fatalError("Subclass responsibility")
        }
        borrowing func unbind() {
            fatalError("Subclass responsibility")
        }
        var count: Int {
            borrowing get {
                fatalError("Subclass responsibility")
            }
        }
    }
    private final class Concrete<$s8Dartmoor11IndexBuffer23AnyErasedProtocolStructfMp_1TfMu_: IndexBuffer & ~Copyable>: Erased {
        private var inner: $s8Dartmoor11IndexBuffer23AnyErasedProtocolStructfMp_1TfMu_
        init(from inner: consuming $s8Dartmoor11IndexBuffer23AnyErasedProtocolStructfMp_1TfMu_) {
            self.inner = inner
        }
        init(indices: [UInt32], loader: Loader) {
            inner = $s8Dartmoor11IndexBuffer23AnyErasedProtocolStructfMp_1TfMu_(
            indices: indices,
            loader: loader,)
        }
        override borrowing func bind() {
            inner
            .bind()
        }
        override borrowing func unbind() {
            inner
            .unbind()
        }
        override var count: Int {
            borrowing get {
                inner.count
            }
        }
    }
    init<$s8Dartmoor11IndexBuffer23AnyErasedProtocolStructfMp_1TfMu0_: ~Copyable & IndexBuffer>(of _: $s8Dartmoor11IndexBuffer23AnyErasedProtocolStructfMp_1TfMu0_.Type, indices: [UInt32], loader: Loader) {
        inner = Concrete<$s8Dartmoor11IndexBuffer23AnyErasedProtocolStructfMp_1TfMu0_>(
        indices: indices,
        loader: loader,)
    }

    	borrowing func bind() {
        inner.bind()
    }
    	borrowing func unbind() {
        inner.unbind()
    }
    var count: Int {
        borrowing get {
            inner.count
        }
    }
}
1 Like