Why does the withUnsafeMutableBufferPointer closure take an inout parameter?

I wonder why the closure in the Array method

    mutating func withUnsafeMutableBufferPointer<R>(_ body: (inout UnsafeMutableBufferPointer<Array.Element>) throws -> R) rethrows -> R

takes an _inout_ parameter. The closure is called with a pointer to the contiguous array storage, and I fail to imagine an example where it makes sense to assign a new value to the parameter.

Any insights are welcome!

Regards, Martin

Thank you Howard for you response! However, I have a follow-up question:

Why are UnsafeMutableBufferPointer methods which modify not the buffer pointer itself, but only the pointed-to memory, mutating at all?

UnsafeMutable(Buffer)Pointer already have non-mutating subscript setters, which means that I can modify the pointed-to memory even if the buffer pointer itself is a constant. For example, this compiles and runs without problems:

    func foo(bufptr: UnsafeMutableBufferPointer<Int>) {
        let tmp = bufptr[0]
        bufptr[0] = bufptr[1]
        bufptr[1] = tmp
    }

    var a = [1, 2, 3, 4, 5]
    a.withUnsafeMutableBufferPointer { foo(bufptr: $0) }

Doing the same with swapAt() does not compile:

    func bar(bufptr: UnsafeMutableBufferPointer<Int>) {
        bufptr.swapAt(0, 1)
        // error: cannot use mutating member on immutable value: 'bufptr' is a 'let' constant
    }

which means that I have to use a variable copy:

    func bar(bufptr: UnsafeMutableBufferPointer<Int>) {
        var bufptr = bufptr
        bufptr.swapAt(0, 1)
    }

So my "feeling" is that methods (like swapAt) which modify the pointed-to memory of an UnsafeMutableBufferPointer should be non-mutating.

That would then also allow (coming back to my original question) that withUnsafeMutableBufferPointer() passes a _constant_ buffer pointer to the closure, and _might_ make the check at

   https://github.com/apple/swift/blob/master/stdlib/public/core/Arrays.swift.gyb#L1750
   
obsolete (which verifies that the closure did not modify the pointer or length).

I am probably overlooking something, so please let me know where I am wrong!

Regards, Martin

···

On 9. Oct 2017, at 01:15, Howard Lovatt <howard.lovatt@gmail.com> wrote:

If it isn't an `inout` then it is a `let` not a `var` and hence you can't call mutating methods on it. There is no 'invar' in Swift, best you can do is `inout`.

  -- Howard.

On 9 October 2017 at 06:14, Martin R via swift-users <swift-users@swift.org> wrote:
I wonder why the closure in the Array method

    mutating func withUnsafeMutableBufferPointer<R>(_ body: (inout UnsafeMutableBufferPointer<Array.Element>) throws -> R) rethrows -> R

takes an _inout_ parameter. The closure is called with a pointer to the contiguous array storage, and I fail to imagine an example where it makes sense to assign a new value to the parameter.

Any insights are welcome!

Regards, Martin
_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

Don’t forget that UnsafeMutableBufferPointer conforms to MutableCollection, and that many MutableCollection methods (such as swapAt) must be mutating. The BufferPointer types are unusual among Collection conformers in not having value semantics.

Cheers,
Guillaume Lessard

···

On Oct 10, 2017, at 01:50, Martin R via swift-users <swift-users@swift.org> wrote:

Thank you Howard for you response! However, I have a follow-up question:

Why are UnsafeMutableBufferPointer methods which modify not the buffer pointer itself, but only the pointed-to memory, mutating at all?

But an implementation of a mutating protocol method does not have to be mutating:

    protocol Foo {
        mutating func test()
    }

    struct Bar: Foo {
        func test() { }
    }

    let bar = Bar()
    bar.test()

Isn't that a similar situation as with the subscript setter, which is mutating in general for MutableCollection, but non-mutating for UnsafeMutableBufferPointer?

Martin

···

On 11. Oct 2017, at 22:11, Guillaume Lessard <glessard@tffenterprises.com> wrote:

On Oct 10, 2017, at 01:50, Martin R via swift-users <swift-users@swift.org> wrote:

Thank you Howard for you response! However, I have a follow-up question:

Why are UnsafeMutableBufferPointer methods which modify not the buffer pointer itself, but only the pointed-to memory, mutating at all?

Don’t forget that UnsafeMutableBufferPointer conforms to MutableCollection, and that many MutableCollection methods (such as swapAt) must be mutating. The BufferPointer types are unusual among Collection conformers in not having value semantics.

Cheers,
Guillaume Lessard

A lot of the MutableCollection implementation is in protocol extensions (swapAt, for example.)

Should an additional version be written just for the Unsafe*BufferPointer types?

Guillaume Lessard

A lot of the MutableCollection implementation is in protocol extensions
(swapAt, for example.)

Should an additional version be written just for the Unsafe*BufferPointer
types?

Makes sense to me, given the examples above. It doesn’t seem to be a high
priority task though, and one one suited to a contributor pull request
rather than taking resources away from the core team.

Would this kind of change need to go through swift evolution or is it a “no
brainer”?

Geordie

···

Guillaume Lessard via swift-users <swift-users@swift.org> schrieb am Mi. 11. Okt. 2017 um 23:49:

Guillaume Lessard

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

It’s not a bug-fix and it is an API change, so it seems it would have to go through evolution.

Guillaume Lessard

I’ll just point out that it’s already the case that methods defined in Unsafe*BufferPointer that write to memory are “nonmutating”. So I think it’s both a “no brainer” and needs to go through evolution.

-Andy

···

On Oct 12, 2017, at 3:52 AM, Geordie Jay via swift-users <swift-users@swift.org> wrote:

Guillaume Lessard via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>> schrieb am Mi. 11. Okt. 2017 um 23:49:
A lot of the MutableCollection implementation is in protocol extensions (swapAt, for example.)

Should an additional version be written just for the Unsafe*BufferPointer types?

Makes sense to me, given the examples above. It doesn’t seem to be a high priority task though, and one one suited to a contributor pull request rather than taking resources away from the core team.

Would this kind of change need to go through swift evolution or is it a “no brainer”?

Geordie

I’m not familiar with the specifics of this “proposal” but if it really is just moving something obvious from being a mutating member to a nonmutating member, then I’m sure the core team can find a fast-path way to accept it without a formal review period.

-Chris

···

On Oct 13, 2017, at 7:40 PM, Andrew Trick via swift-users <swift-users@swift.org> wrote:

On Oct 12, 2017, at 3:52 AM, Geordie Jay via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>> wrote:

Guillaume Lessard via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>> schrieb am Mi. 11. Okt. 2017 um 23:49:
A lot of the MutableCollection implementation is in protocol extensions (swapAt, for example.)

Should an additional version be written just for the Unsafe*BufferPointer types?

Makes sense to me, given the examples above. It doesn’t seem to be a high priority task though, and one one suited to a contributor pull request rather than taking resources away from the core team.

Would this kind of change need to go through swift evolution or is it a “no brainer”?

Geordie

I’ll just point out that it’s already the case that methods defined in Unsafe*BufferPointer that write to memory are “nonmutating”. So I think it’s both a “no brainer” and needs to go through evolution.

A lot of the MutableCollection implementation is in protocol extensions
(swapAt, for example.)

Should an additional version be written just for the Unsafe*BufferPointer
types?

Makes sense to me, given the examples above. It doesn’t seem to be a high
priority task though, and one one suited to a contributor pull request
rather than taking resources away from the core team.

Would this kind of change need to go through swift evolution or is it a
“no brainer”?

Geordie

I’ll just point out that it’s already the case that methods defined in
Unsafe*BufferPointer that write to memory are “nonmutating”. So I think
it’s both a “no brainer” and needs to go through evolution.

I’m not familiar with the specifics of this “proposal” but if it really is
just moving something obvious from being a mutating member to a nonmutating
member, then I’m sure the core team can find a fast-path way to accept it
without a formal review period.

I’m not 100% sure either tbh, although I can imagine putting in a PR for it
once i understand it.

The issue with changing .swapTo (et al.?) to nonmutating is that
semantically it really is a mutation. But pointers as collections have
different mutation semantics to other collections: Mutating an Array’s
storage is the same as mutating its value, whereas a pointer’s value is its
address, not its storage.

Making the Unsafe*Pointer MutableCollection methods themselves nonmutating
probably wouldn’t be a source-breaking change (it’d just lower the “bar of
entry” to include let constant instances too). I imagine this is
noncontroversial.

The original question though was about why .withUnsafeMutableBufferPointer
takes a closure whose first argument is an **inout**
UnsafeMutableBufferPointer, implying that its base address could legally be
mutated. This was probably done to ease use of MutableCollection methods
given the var parameter. That would no longer be necessary given the first
change.

But removing the inout attribute would be source-breaking. And maybe people
really are moving around the array storage’s base address? This seems like
a very bad idea to me but it’s currently part of that API contract. In any
case this change would need to go through evolution, right?

- Geordie

···

Chris Lattner <clattner@nondot.org> schrieb am Sa. 14. Okt. 2017 um 05:18: > > On Oct 13, 2017, at 7:40 PM, Andrew Trick via swift-users < > swift-users@swift.org> wrote:

On Oct 12, 2017, at 3:52 AM, Geordie Jay via swift-users < > swift-users@swift.org> wrote:
Guillaume Lessard via swift-users <swift-users@swift.org> schrieb am Mi. > 11. Okt. 2017 um 23:49:

-Chris

A lot of the MutableCollection implementation is in protocol extensions (swapAt, for example.)

Should an additional version be written just for the Unsafe*BufferPointer types?

Makes sense to me, given the examples above. It doesn’t seem to be a high priority task though, and one one suited to a contributor pull request rather than taking resources away from the core team.

Would this kind of change need to go through swift evolution or is it a “no brainer”?

Geordie

I’ll just point out that it’s already the case that methods defined in Unsafe*BufferPointer that write to memory are “nonmutating”. So I think it’s both a “no brainer” and needs to go through evolution.

I’m not familiar with the specifics of this “proposal” but if it really is just moving something obvious from being a mutating member to a nonmutating member, then I’m sure the core team can find a fast-path way to accept it without a formal review period.

I’m not 100% sure either tbh, although I can imagine putting in a PR for it once i understand it.

The issue with changing .swapTo (et al.?) to nonmutating is that semantically it really is a mutation. But pointers as collections have different mutation semantics to other collections: Mutating an Array’s storage is the same as mutating its value, whereas a pointer’s value is its address, not its storage.

Making the Unsafe*Pointer MutableCollection methods themselves nonmutating probably wouldn’t be a source-breaking change (it’d just lower the “bar of entry” to include let constant instances too). I imagine this is noncontroversial.

The original question though was about why .withUnsafeMutableBufferPointer takes a closure whose first argument is an *inout* UnsafeMutableBufferPointer, implying that its base address could legally be mutated. This was probably done to ease use of MutableCollection methods given the var parameter. That would no longer be necessary given the first change.

But removing the inout attribute would be source-breaking. And maybe people really are moving around the array storage’s base address? This seems like a very bad idea to me but it’s currently part of that API contract. In any case this change would need to go through evolution, right?

At present, the withUnsafeMutableBufferPointer() method verifies that the base address has not been modified upon return from the closure:

    https://github.com/apple/swift/blob/master/stdlib/public/core/Arrays.swift.gyb#L1750

    // Put the working array back before returning.
    defer {
      _precondition(
        inoutBufferPointer.baseAddress == pointer &&
        inoutBufferPointer.count == count,
        "${Self} withUnsafeMutableBufferPointer: replacing the buffer is not allowed")

      (work, self) = (self, work)
    }

That check would become obsolete if the buffer pointer is a constant.

As far as I can see, MutableCollection defines four mutating methods: swapAt(), sort(), reverse(), and partition(). Implementing those as non-mutating methods of UnsafeMutableBufferPointer seems natural to me.

In addition, it would allow to make the closure parameter in withUnsafeMutableBufferPointer() a constant. Mutating that pointer in the closure is not allowed anyway.

Btw, I came up with these questions because I noticed that this compiles:

    func foo(bufptr: UnsafeMutableBufferPointer<Int>) { }
    var a = [1, 2, 3, 4, 5]
    a.withUnsafeMutableBufferPointer { foo(bufptr: $0) }
    
but this does not compile:

    a.withUnsafeMutableBufferPointer(foo)
    // error: cannot convert value of type '(UnsafeMutableBufferPointer<Int>) -> ()'
    // to expected argument type '(inout UnsafeMutableBufferPointer<Int>) -> _'
    
which looks strange at first sight. With a constant closure parameter, the second version would compile as well.

Any advice how to proceed with this "issue" would be appreciated: File a bug report? Post on swift-evolution? Make a proposal? ... However, I am not familiar with those evolution procedures. I won't mind to sidestep if someone else want to jump in.

Martin

···

On 14. Oct 2017, at 13:15, Geordie Jay via swift-users <swift-users@swift.org> wrote:
Chris Lattner <clattner@nondot.org> schrieb am Sa. 14. Okt. 2017 um 05:18: > >> On Oct 13, 2017, at 7:40 PM, Andrew Trick via swift-users <swift-users@swift.org> wrote:

On Oct 12, 2017, at 3:52 AM, Geordie Jay via swift-users <swift-users@swift.org> wrote:
Guillaume Lessard via swift-users <swift-users@swift.org> schrieb am Mi. 11. Okt. 2017 um 23:49:

- Geordie

-Chris

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

There is a lot of value in these being mutating functions when working with C APIs that need a buffer for it to write to.

If you have a collection and ensure it had allocated it's memory, you can then have the C API write directly into the collection's storage avoiding extra copy operations.

I personally take advantage of this a lot. Because I'm wrapping a C library in Swift as part of my work.

If this was changed to only non mutating functions, it still wouldn't be able to guarantee that the buffer isn't mutating the collection without always doing a deep copy. It is very easy to go from an UnsafeBufferPointer -> UnsafeMutableBufferPointer and have people change things.

···

On Oct 14, 2017, at 7:11 AM, Martin R via swift-users <swift-users@swift.org> wrote:

On 14. Oct 2017, at 13:15, Geordie Jay via swift-users <swift-users@swift.org> wrote:

Chris Lattner <clattner@nondot.org> schrieb am Sa. 14. Okt. 2017 um 05:18: >> >>> On Oct 13, 2017, at 7:40 PM, Andrew Trick via swift-users <swift-users@swift.org> wrote:

On Oct 12, 2017, at 3:52 AM, Geordie Jay via swift-users <swift-users@swift.org> wrote:

Guillaume Lessard via swift-users <swift-users@swift.org> schrieb am Mi. 11. Okt. 2017 um 23:49:
A lot of the MutableCollection implementation is in protocol extensions (swapAt, for example.)

Should an additional version be written just for the Unsafe*BufferPointer types?

Makes sense to me, given the examples above. It doesn’t seem to be a high priority task though, and one one suited to a contributor pull request rather than taking resources away from the core team.

Would this kind of change need to go through swift evolution or is it a “no brainer”?

Geordie

I’ll just point out that it’s already the case that methods defined in Unsafe*BufferPointer that write to memory are “nonmutating”. So I think it’s both a “no brainer” and needs to go through evolution.

I’m not familiar with the specifics of this “proposal” but if it really is just moving something obvious from being a mutating member to a nonmutating member, then I’m sure the core team can find a fast-path way to accept it without a formal review period.

I’m not 100% sure either tbh, although I can imagine putting in a PR for it once i understand it.

The issue with changing .swapTo (et al.?) to nonmutating is that semantically it really is a mutation. But pointers as collections have different mutation semantics to other collections: Mutating an Array’s storage is the same as mutating its value, whereas a pointer’s value is its address, not its storage.

Making the Unsafe*Pointer MutableCollection methods themselves nonmutating probably wouldn’t be a source-breaking change (it’d just lower the “bar of entry” to include let constant instances too). I imagine this is noncontroversial.

The original question though was about why .withUnsafeMutableBufferPointer takes a closure whose first argument is an *inout* UnsafeMutableBufferPointer, implying that its base address could legally be mutated. This was probably done to ease use of MutableCollection methods given the var parameter. That would no longer be necessary given the first change.

But removing the inout attribute would be source-breaking. And maybe people really are moving around the array storage’s base address? This seems like a very bad idea to me but it’s currently part of that API contract. In any case this change would need to go through evolution, right?

At present, the withUnsafeMutableBufferPointer() method verifies that the base address has not been modified upon return from the closure:

   https://github.com/apple/swift/blob/master/stdlib/public/core/Arrays.swift.gyb#L1750

   // Put the working array back before returning.
   defer {
     _precondition(
       inoutBufferPointer.baseAddress == pointer &&
       inoutBufferPointer.count == count,
       "${Self} withUnsafeMutableBufferPointer: replacing the buffer is not allowed")

     (work, self) = (self, work)
   }

That check would become obsolete if the buffer pointer is a constant.

As far as I can see, MutableCollection defines four mutating methods: swapAt(), sort(), reverse(), and partition(). Implementing those as non-mutating methods of UnsafeMutableBufferPointer seems natural to me.

In addition, it would allow to make the closure parameter in withUnsafeMutableBufferPointer() a constant. Mutating that pointer in the closure is not allowed anyway.

Btw, I came up with these questions because I noticed that this compiles:

   func foo(bufptr: UnsafeMutableBufferPointer<Int>) { }
   var a = [1, 2, 3, 4, 5]
   a.withUnsafeMutableBufferPointer { foo(bufptr: $0) }

but this does not compile:

   a.withUnsafeMutableBufferPointer(foo)
   // error: cannot convert value of type '(UnsafeMutableBufferPointer<Int>) -> ()'
   // to expected argument type '(inout UnsafeMutableBufferPointer<Int>) -> _'

which looks strange at first sight. With a constant closure parameter, the second version would compile as well.

Any advice how to proceed with this "issue" would be appreciated: File a bug report? Post on swift-evolution? Make a proposal? ... However, I am not familiar with those evolution procedures. I won't mind to sidestep if someone else want to jump in.

Martin

- Geordie

-Chris

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users

The types wouldn’t change, just whether the method is marked `mutating`.

The reason I have doubts whether it’s worth it is that even if the implementation gets duplicated as non-mutating, it won’t prevent cases where a {*}BufferPointer needs to be a var: whenever it is passed into a generic MutableCollection context, it will need to be a var in order to be usable.
Is it worth it?

Cheers,
Guillaume Lessard

···

On Oct 14, 2017, at 10:29, Braden Scothern via swift-users <swift-users@swift.org> wrote:

There is a lot of value in these being mutating functions when working with C APIs that need a buffer for it to write to.

If you have a collection and ensure it had allocated it's memory, you can then have the C API write directly into the collection's storage avoiding extra copy operations.

I personally take advantage of this a lot. Because I'm wrapping a C library in Swift as part of my work.

If this was changed to only non mutating functions, it still wouldn't be able to guarantee that the buffer isn't mutating the collection without always doing a deep copy. It is very easy to go from an UnsafeBufferPointer -> UnsafeMutableBufferPointer and have people change things.