Pitch: Improved Swift pointers

Hi all, I’ve written up a proposal to modify the unsafe pointer API for
greater consistency, safety, and ease of use.

···

~~~

Swift currently offers two sets of pointer types — singular pointers such
as UnsafeMutablePointer, and vector (buffer) pointers such as UnsafeMutable
*Buffer*Pointer. This implies a natural separation of tasks the two kinds
of pointers are meant to do. For example, buffer pointers implement
Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied roles.
It is possible to allocate an arbitrary number of instances from a type
method on a singular pointer, but not from a buffer pointer. The result of
such an operation returns a singular pointer, even though a buffer pointer
would be more appropriate to capture the information about the *number* of
instances allocated. It’s possible to subscript into a singular pointer,
even though they are not real Collections. Some parts of the current design
turn UnsafePointers into downright *Dangerous*Pointers, leading users to
believe that they have allocated or freed memory when in fact, they have
not.

This proposal seeks to iron out these inconsistencies, and offer a more
convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~

I agree with most of this proposal. Removing the subscript bugs me, but pointer arithmetic currently takes the stride of the type into account, so I don’t necessarily have a problem with that.

I would really appreciate Unsafe*BufferPointer.allocate(count:), and I definitely think the sized deallocators are actively dangerous unless the arguments are used. I fully support their removal.

···

On Jul 12, 2017, at 12:16 PM, Taylor Swift via swift-evolution <swift-evolution@swift.org> wrote:

Hi all, I’ve written up a proposal to modify the unsafe pointer API for greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such as UnsafeMutablePointer, and vector (buffer) pointers such as UnsafeMutableBufferPointer. This implies a natural separation of tasks the two kinds of pointers are meant to do. For example, buffer pointers implement Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied roles. It is possible to allocate an arbitrary number of instances from a type method on a singular pointer, but not from a buffer pointer. The result of such an operation returns a singular pointer, even though a buffer pointer would be more appropriate to capture the information about the number of instances allocated. It’s possible to subscript into a singular pointer, even though they are not real Collections. Some parts of the current design turn UnsafePointers into downright DangerousPointers, leading users to believe that they have allocated or freed memory when in fact, they have not.

This proposal seeks to iron out these inconsistencies, and offer a more convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

If you’re considering moving allocate/deallocate to Unsafe*BufferPointer, then you probably also want to consider doing the same for initialize, deinitialize, and various move functions as well.

···

On Jul 12, 2017, at 12:16 PM, Taylor Swift via swift-evolution <swift-evolution@swift.org> wrote:

Hi all, I’ve written up a proposal to modify the unsafe pointer API for greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such as UnsafeMutablePointer, and vector (buffer) pointers such as UnsafeMutableBufferPointer. This implies a natural separation of tasks the two kinds of pointers are meant to do. For example, buffer pointers implement Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied roles. It is possible to allocate an arbitrary number of instances from a type method on a singular pointer, but not from a buffer pointer. The result of such an operation returns a singular pointer, even though a buffer pointer would be more appropriate to capture the information about the number of instances allocated. It’s possible to subscript into a singular pointer, even though they are not real Collections. Some parts of the current design turn UnsafePointers into downright DangerousPointers, leading users to believe that they have allocated or freed memory when in fact, they have not.

This proposal seeks to iron out these inconsistencies, and offer a more convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I originally thought that would be too many changes for one proposal but if
y’all think it’s a good idea to deal with those functions too I’m happy to
add it to the proposal!

···

On Wed, Jul 12, 2017 at 5:07 PM, Michael Ilseman <milseman@apple.com> wrote:

If you’re considering moving allocate/deallocate to Unsafe*BufferPointer,
then you probably also want to consider doing the same for initialize,
deinitialize, and various move functions as well.

On Jul 12, 2017, at 12:16 PM, Taylor Swift via swift-evolution < > swift-evolution@swift.org> wrote:

Hi all, I’ve written up a proposal to modify the unsafe pointer API for
greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such
as UnsafeMutablePointer, and vector (buffer) pointers such as
UnsafeMutable*Buffer*Pointer. This implies a natural separation of tasks
the two kinds of pointers are meant to do. For example, buffer pointers
implement Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied
roles. It is possible to allocate an arbitrary number of instances from a
type method on a singular pointer, but not from a buffer pointer. The
result of such an operation returns a singular pointer, even though a
buffer pointer would be more appropriate to capture the information about
the *number* of instances allocated. It’s possible to subscript into a
singular pointer, even though they are not real Collections. Some parts
of the current design turn UnsafePointers into downright *Dangerous*Pointers,
leading users to believe that they have allocated or freed memory when in
fact, they have not.

This proposal seeks to iron out these inconsistencies, and offer a more
convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

In my understanding, the address manipulation abstractions are as follows:
An address is an integer denoting an index into the current address space (which, in case of a user-space process, is a virtual address space, uniquely allocated by the operating system for that process). The minimal addressable unit (also known as byte, which is not to be confused with an octet) of the address space is the number of contiguous binary digits (also known as bits) that are uniquely identified by a single address (in most modern architectures that number is 8).
A buffer is a range of addresses, denoting a fragment of the address space, where the delimited contiguous sequence of addresses share common semantics.
A pointer is a buffer with some metadata. In statically typed languages the length of the buffer is a compile-time constant and metadata contains type information of the buffer’s contents.
An array is a buffer, representing a contiguous sequence of pointers.

Notice, that in above terms the address space is essentially a buffer denoting all available addresses.

In Swift, the aforementioned concepts correspond to the following types:
address: UnsafeRawPointer, UnsafeRawMutablePointer, OpaquePointer.
buffer: UnsafeRawBufferPointer, UnsafeMutableRawBufferPointer.
pointer: UnsafePointer, UnsafeMutablePointer, AutoreleasingUnsafeMutablePointer.
array: UnsafeBufferPointer, UnsafeMutableBufferPointer.

If the unsafe tools of Swift would be redesigned, I’d suggest defining them in the following manner:
A comparable and hashable address that can be added an address offset, and subtracted another address to get an address offset. An address would also expose a mutable/immutable property that is the byte referred to by that address.
A buffer, that is a random access mutable/immutable collection of addresses, represented as a range.
A pointer, that is a generic type that is represented by an address and the inherent type information of its generic parameter. The pointer would only expose the address in the form of a buffer and would expose the referred object in the form of a mutable/immutable property of the respective type.
An array, that is a random access mutable/immutable collection of pointers, represented as a pointer coupled by a count.

So, in the end, we’d have the following set of types: UnsafeAddress, UnsafeBuffer, UnsafePointer, UnsafeArray (as well as their mutable counter-parts).
The OpaquePointer would then be a thin wrapper around UnsafeAddress and would probably better off renamed to CStructPointer, due to the fact that all non-C usages of this type should be replaced by usage of UnsafeAddress.
And the AutoreleasingUnsafeMutablePointer would also be a thing wrapper around UnsafeAddress with a possible renaming to NSObjectPointer.

···

On Jul 12, 2017, at 10:26 PM, Taylor Swift via swift-evolution <swift-evolution@swift.org> wrote:

Hi all, I’ve written up a proposal to modify the unsafe pointer API for greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such as UnsafeMutablePointer, and vector (buffer) pointers such as UnsafeMutableBufferPointer. This implies a natural separation of tasks the two kinds of pointers are meant to do. For example, buffer pointers implement Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied roles. It is possible to allocate an arbitrary number of instances from a type method on a singular pointer, but not from a buffer pointer. The result of such an operation returns a singular pointer, even though a buffer pointer would be more appropriate to capture the information about the number of instances allocated. It’s possible to subscript into a singular pointer, even though they are not real Collections. Some parts of the current design turn UnsafePointers into downright DangerousPointers, leading users to believe that they have allocated or freed memory when in fact, they have not.

This proposal seeks to iron out these inconsistencies, and offer a more convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I have no problem with this direction in principle; it sounds like a
good idea. However, before affirming any particular design I would like
to see the results of any such proposal applied to a fairly large body
of code that uses Unsafe[XXX]Pointer today in diverse ways (such as the
Swift standard library itself), to demonstrate that it does in fact
improve the code in practice.

Cheers,

···

on Wed Jul 12 2017, Taylor Swift <swift-evolution@swift.org> wrote:

Hi all, I’ve written up a proposal to modify the unsafe pointer API for
greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such
as UnsafeMutablePointer, and vector (buffer) pointers such as UnsafeMutable
*Buffer*Pointer. This implies a natural separation of tasks the two kinds
of pointers are meant to do. For example, buffer pointers implement
Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied roles.
It is possible to allocate an arbitrary number of instances from a type
method on a singular pointer, but not from a buffer pointer. The result of
such an operation returns a singular pointer, even though a buffer pointer
would be more appropriate to capture the information about the *number* of
instances allocated. It’s possible to subscript into a singular pointer,
even though they are not real Collections. Some parts of the current design
turn UnsafePointers into downright *Dangerous*Pointers, leading users to
believe that they have allocated or freed memory when in fact, they have
not.

This proposal seeks to iron out these inconsistencies, and offer a more
convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

--
-Dave

Hi all, I’ve written up a proposal to modify the unsafe pointer API for greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such as UnsafeMutablePointer, and vector (buffer) pointers such as UnsafeMutableBufferPointer. This implies a natural separation of tasks the two kinds of pointers are meant to do. For example, buffer pointers implement Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied roles. It is possible to allocate an arbitrary number of instances from a type method on a singular pointer, but not from a buffer pointer. The result of such an operation returns a singular pointer, even though a buffer pointer would be more appropriate to capture the information about the number of instances allocated. It’s possible to subscript into a singular pointer, even though they are not real Collections. Some parts of the current design turn UnsafePointers into downright DangerousPointers, leading users to believe that they have allocated or freed memory when in fact, they have not.

This proposal seeks to iron out these inconsistencies, and offer a more convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~

Thanks for taking time to write this up.

General comments:

UnsafeBufferPointer is an API layer on top of UnsafePointer. The role
of UnsafeBufferPointer is direct memory access sans lifetime
management with Collection semantics. The role of UnsafePointer is
primarily C interop. Those C APIs should be wrapped in Swift APIs that
take UnsafeBufferPointer whenever the pointer represents a C array. I
suppose making UnsafePointer less convenient would push developers
toward UnsafeBufferPointer. I don't think that's worth outright
breaking source, but gradual deprecation of convenience methods, like
`susbscript` might be acceptable.

I have mixed feelings about stripping UnsafePointer of basic
functionality. Besides breaking source, doing that would be
inconsistent with its role as a lower API layer. The advantage would
just be descreasing API surface area and forcing developers to use a
higher-level API.

The additive changes you propose are fairly obvious. See [SR-3088]
UnsafeMutableBufferPointer doesn't have an allocating init.

I haven't wanted to waste review cycles on small additive
changes. It may make sense to batch them up into one coherent
proposal. Here are a few more to consider.

- [SR-3929] UnsafeBufferPointer should have init from mutable
- [SR-4340] UnsafeBufferPointer needs a withMemoryRebound method
- [SR-3087] No way to arbitrarily initialise an Array's storage

Point by point:

drop the capacity parameter from UnsafeMutablePointer.allocate() and deallocate().

I do not agree with removing the capacity parameter and adding a
single-instance allocation API. UnsafePointer was not designed for
single instances, it was primarily designed for C-style arrays. I
don't see the value in providing a different unsafe API for single
vs. multiple values.

I agree the primary allocation API should be
UnsafeMutableBufferPointer.allocate(capacity:). There is an argument
to be made for removing UnsafeMutablePointer.allocate(capacity:)
entirely. But, as Michael Ilseman pointed out, that would involve
reevaluating several other members of the UnsafePointer API. I think
it's reasonable for UnsafePointer to retain all its functionality as a
lower level API.

I don't understand what is misleading about
UnsafePointer.deallocate(capacity:). It *is* inconvenienent for the
user to keep track of memory capacity. Presumably that was done so
either the implementation can move away from malloc/free or some sort
of memory tracking can be implemented on the standard library
side. Obviously, UnsafeBufferPointer.deallocate() would be cleaner in
most cases.

add an allocate(count:) type method to UnsafeMutableBufferPointer

`capacity` should be used for allocating uninitialized memory not
`count`. `count` should only refer to a number of initialized objects!

add a deallocate() instance method to UnsafeMutableBufferPointer

Yes, of course! I added a mention of that in SR-3088.

remove subscripts from UnsafePointer and UnsafeMutablePointer

It's often more clear to perform arithmetic on C array indices rather
than pointers. That said, I'm happy to push developers to use
UnsafeBufferPointer whenever that have a known capacity. To me, this
is a question of whether the benefit of making a dangerous thing less
convenient is worth breaking source compatibility.

-Andy

···

On Jul 12, 2017, at 12:16 PM, Taylor Swift via swift-evolution <swift-evolution@swift.org> wrote:

I’ve drafted a new version of the unsafe pointer proposal based on feedback
I’ve gotten from this thread. You can read it here
<https://gist.github.com/kelvin13/1b8ae906be23dff22f7a7c4767f0c907>.

···

~~~

Swift’s pointer types are an important interface for low-level memory
manipulation, but the current API design is not very safe, consistent, or
convenient. Many memory methods demand a capacity: or count: argument,
forcing the user to manually track the size of the memory block, even
though most of the time this is either unnecessary, or redundant as buffer
pointers track this information natively. In some places, this design turns
UnsafePointers into outright *Dangerous*Pointers, leading users to believe
that they have allocated or freed memory when in fact, they have not.

The current API suffers from inconsistent naming, poor usage of default
argument values, missing methods, and excessive verbosity, and encourages
excessively unsafe programming practices. This proposal seeks to iron out
these inconsistencies, and offer a more convenient, more sensible, and less
bug-prone API for Swift pointers.

The previous draft
<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06> of this
proposal was relatively source-breaking, calling for a separation of
functionality between singular pointer types and vector (buffer) pointer
types. This proposal instead separates functionality between
internally-tracked length pointer types and externally-tracked length
pointer types. This results in an equally elegant API with about one-third
less surface area.

<https://gist.github.com/kelvin13/1b8ae906be23dff22f7a7c4767f0c907>

~~~

On Wed, Jul 12, 2017 at 3:16 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

Hi all, I’ve written up a proposal to modify the unsafe pointer API for
greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such
as UnsafeMutablePointer, and vector (buffer) pointers such as
UnsafeMutable*Buffer*Pointer. This implies a natural separation of tasks
the two kinds of pointers are meant to do. For example, buffer pointers
implement Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied
roles. It is possible to allocate an arbitrary number of instances from a
type method on a singular pointer, but not from a buffer pointer. The
result of such an operation returns a singular pointer, even though a
buffer pointer would be more appropriate to capture the information about
the *number* of instances allocated. It’s possible to subscript into a
singular pointer, even though they are not real Collections. Some parts
of the current design turn UnsafePointers into downright *Dangerous*Pointers,
leading users to believe that they have allocated or freed memory when in
fact, they have not.

This proposal seeks to iron out these inconsistencies, and offer a more
convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~

i’m sorry this sounds like something different than what this proposal is
trying to achieve. Right now I’m just trying to smooth out the singular vs.
vector functionality of the existing Swift pointer types. Maybe your idea
would be better in a separate proposal?

···

On Wed, Jul 12, 2017 at 4:16 PM, Gor Gyolchanyan <gor@gyolchanyan.com> wrote:

In my understanding, the address manipulation abstractions are as follows:

   - An *address* is an integer denoting an index into the current *address
   space* (which, in case of a user-space process, is a virtual address
   space, uniquely allocated by the operating system for that process). The *minimal
   addressable unit* (also known as *byte*, which is not to be confused
   with an *octet*) of the address space is the number of contiguous *binary
   digits* (also known as *bits*) that are uniquely identified by a
   single address (in most modern architectures that number is *8*).
   - A *buffer* is a range of addresses, denoting a fragment of the
   address space, where the delimited contiguous sequence of addresses share
   common semantics.
   - A *pointer* is a buffer with some metadata. In statically typed
   languages the length of the buffer is a compile-time constant and metadata
   contains type information of the buffer’s contents.
   - An *array* is a buffer, representing a contiguous sequence of
   pointers.

Notice, that in above terms the address space is essentially a buffer
denoting all available addresses.

In Swift, the aforementioned concepts correspond to the following types:

   - *address*: UnsafeRawPointer, UnsafeRawMutablePointer, OpaquePointer.
   - *buffer*: UnsafeRawBufferPointer, UnsafeMutableRawBufferPointer.
   - *pointer*: UnsafePointer, UnsafeMutablePointer,
   AutoreleasingUnsafeMutablePointer.
   - *array*: UnsafeBufferPointer, UnsafeMutableBufferPointer.

If the unsafe tools of Swift would be redesigned, I’d suggest defining
them in the following manner:

   - A comparable and hashable address that can be added an address
   offset, and subtracted another address to get an address offset. An address
   would also expose a mutable/immutable property that is the byte referred to
   by that address.
   - A buffer, that is a random access mutable/immutable collection of
   addresses, represented as a range.
   - A pointer, that is a generic type that is represented by an address
   and the inherent type information of its generic parameter. The pointer
   would only expose the address in the form of a buffer and would expose the
   referred object in the form of a mutable/immutable property of the
   respective type.
   - An array, that is a random access mutable/immutable collection of
   pointers, represented as a pointer coupled by a count.

So, in the end, we’d have the following set of types: UnsafeAddress,
UnsafeBuffer, UnsafePointer, UnsafeArray (as well as their mutable
counter-parts).
The OpaquePointer would then be a thin wrapper around UnsafeAddress and
would probably better off renamed to CStructPointer, due to the fact that
all non-C usages of this type should be replaced by usage of UnsafeAddress
.
And the AutoreleasingUnsafeMutablePointer would also be a thing wrapper
around UnsafeAddress with a possible renaming to NSObjectPointer.

On Jul 12, 2017, at 10:26 PM, Taylor Swift via swift-evolution < > swift-evolution@swift.org> wrote:

Hi all, I’ve written up a proposal to modify the unsafe pointer API for
greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such
as UnsafeMutablePointer, and vector (buffer) pointers such as
UnsafeMutable*Buffer*Pointer. This implies a natural separation of tasks
the two kinds of pointers are meant to do. For example, buffer pointers
implement Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied
roles. It is possible to allocate an arbitrary number of instances from a
type method on a singular pointer, but not from a buffer pointer. The
result of such an operation returns a singular pointer, even though a
buffer pointer would be more appropriate to capture the information about
the *number* of instances allocated. It’s possible to subscript into a
singular pointer, even though they are not real Collections. Some parts
of the current design turn UnsafePointers into downright *Dangerous*Pointers,
leading users to believe that they have allocated or freed memory when in
fact, they have not.

This proposal seeks to iron out these inconsistencies, and offer a more
convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Hi all, i’ve put out a new revision of the proposal which addresses
Michael’s point about dealing with the memorystate APIs along with the
allocation and deallocation API. I didn’t really think about fixing the
memorystate functions originally, but on closer inspection, they are almost
as problematic as the memory allocation API and deserve a closer look.

Here’s a new version of the proposal with revisions
<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

···

On Wed, Jul 12, 2017 at 6:46 PM, Taylor Swift via swift-evolution < swift-evolution@swift.org> wrote:

I originally thought that would be too many changes for one proposal but
if y’all think it’s a good idea to deal with those functions too I’m happy
to add it to the proposal!

On Wed, Jul 12, 2017 at 5:07 PM, Michael Ilseman <milseman@apple.com> > wrote:

If you’re considering moving allocate/deallocate to Unsafe*BufferPointer,
then you probably also want to consider doing the same for initialize,
deinitialize, and various move functions as well.

On Jul 12, 2017, at 12:16 PM, Taylor Swift via swift-evolution < >> swift-evolution@swift.org> wrote:

Hi all, I’ve written up a proposal to modify the unsafe pointer API for
greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such
as UnsafeMutablePointer, and vector (buffer) pointers such as
UnsafeMutable*Buffer*Pointer. This implies a natural separation of tasks
the two kinds of pointers are meant to do. For example, buffer pointers
implement Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied
roles. It is possible to allocate an arbitrary number of instances from a
type method on a singular pointer, but not from a buffer pointer. The
result of such an operation returns a singular pointer, even though a
buffer pointer would be more appropriate to capture the information about
the *number* of instances allocated. It’s possible to subscript into a
singular pointer, even though they are not real Collections. Some parts
of the current design turn UnsafePointers into downright *Dangerous*Pointers,
leading users to believe that they have allocated or freed memory when in
fact, they have not.

This proposal seeks to iron out these inconsistencies, and offer a more
convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I am not very familiar with the inner workings of the standard library,
however I maintain a few libraries which make extensive use of Swift
pointers, such as https://github.com/kelvin13/maxpng which makes extensive
use of Unsafe_____Pointers. I also write a lot of code that interfaces with
C APIs like Cairo and OpenGL. Most of the ideas in the original proposal
came from me dealing with the current Swift pointer APIs in my own code.
For example I find myself writing this bit of code

let buffer = UnsafeMutableBufferPointer<UInt8>(start:
UnsafeMutablePointer<UInt8>.allocate(capacity: byteCount), count:
byteCount)defer
{
    buffer.baseAddress?.deallocate(capacity: buffer.count)
}

far more than I would like to. While this proposal doesn’t solve every
problem with Swift pointers — for example, we need a UMBP initializer
that takes an immutable buffer pointer before we are really able to
write a lot of examples more concisely, it takes us a great deal
closer to being able to write things like

UnsafeMutablePointer(mutating:
self.zero_line.baseAddress!).deallocate(capacity:
self.zero_line.count)

as

UnsafeMutableBufferPointer(mutating: self.zero_line).deallocate()

···

On Thu, Jul 13, 2017 at 2:47 PM, Dave Abrahams via swift-evolution < swift-evolution@swift.org> wrote:

on Wed Jul 12 2017, Taylor Swift <swift-evolution@swift.org> wrote:

> Hi all, I’ve written up a proposal to modify the unsafe pointer API for
> greater consistency, safety, and ease of use.
>
> ~~~
>
> Swift currently offers two sets of pointer types — singular pointers such
> as UnsafeMutablePointer, and vector (buffer) pointers such as
UnsafeMutable
> *Buffer*Pointer. This implies a natural separation of tasks the two kinds
> of pointers are meant to do. For example, buffer pointers implement
> Collection conformance, while singular pointers do not.
>
> However, some aspects of the pointer design contradict these implied
roles.
> It is possible to allocate an arbitrary number of instances from a type
> method on a singular pointer, but not from a buffer pointer. The result
of
> such an operation returns a singular pointer, even though a buffer
pointer
> would be more appropriate to capture the information about the *number*
of
> instances allocated. It’s possible to subscript into a singular pointer,
> even though they are not real Collections. Some parts of the current
design
> turn UnsafePointers into downright *Dangerous*Pointers, leading users to
> believe that they have allocated or freed memory when in fact, they have
> not.
>
> This proposal seeks to iron out these inconsistencies, and offer a more
> convenient, more sensible, and less bug-prone API for Swift pointers.
>
> <https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

I have no problem with this direction in principle; it sounds like a
good idea. However, before affirming any particular design I would like
to see the results of any such proposal applied to a fairly large body
of code that uses Unsafe[XXX]Pointer today in diverse ways (such as the
Swift standard library itself), to demonstrate that it does in fact
improve the code in practice.

Cheers,

--
-Dave

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Hi all, I’ve written up a proposal to modify the unsafe pointer API for
greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such
as UnsafeMutablePointer, and vector (buffer) pointers such as
UnsafeMutable*Buffer*Pointer. This implies a natural separation of tasks
the two kinds of pointers are meant to do. For example, buffer pointers
implement Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied
roles. It is possible to allocate an arbitrary number of instances from a
type method on a singular pointer, but not from a buffer pointer. The
result of such an operation returns a singular pointer, even though a
buffer pointer would be more appropriate to capture the information about
the *number* of instances allocated. It’s possible to subscript into a
singular pointer, even though they are not real Collections. Some parts
of the current design turn UnsafePointers into downright *Dangerous*Pointers,
leading users to believe that they have allocated or freed memory when in
fact, they have not.

This proposal seeks to iron out these inconsistencies, and offer a more
convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~

Thanks for taking time to write this up.

General comments:

UnsafeBufferPointer is an API layer on top of UnsafePointer. The role
of UnsafeBufferPointer is direct memory access sans lifetime
management with Collection semantics. The role of UnsafePointer is
primarily C interop. Those C APIs should be wrapped in Swift APIs that
take UnsafeBufferPointer whenever the pointer represents a C array. I
suppose making UnsafePointer less convenient would push developers
toward UnsafeBufferPointer. I don't think that's worth outright
breaking source, but gradual deprecation of convenience methods, like
`susbscript` might be acceptable.

Gradual deprecation is exactly what I am proposing. As the document states
<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06#proposed-solution>,
the only methods which should be marked immediately as unavailable are the `
deallocate(capacity:)` methods, for safety and source compatibility
reasons. Removing `deallocate(capacity:)` now and forcing a loud compiler
error prevents catastrophic *silent* source breakage in the future, or
worse, from having to *support our own bug*.

I have mixed feelings about stripping UnsafePointer of basic
functionality. Besides breaking source, doing that would be
inconsistent with its role as a lower API layer. The advantage would
just be descreasing API surface area and forcing developers to use a
higher-level API.

UnsafePointer is as much a high level API as UnsafeBufferPointer is. You
wouldn’t create a buffer pointer of length 1 just so you can “stick with
the high level API”. UnsafePointer and UnsafeBufferPointer are two tools
that do related but different things and they can exist at whatever
abstract level you need them at. After all, UnsafeBufferPointer is nothing
but an UnsafePointer? with a length value attached to it. If you’re
allocating more than one instance of memory, you almost certainly need to
track the length of the buffer anyway.

The additive changes you propose are fairly obvious. See [SR-3088]
UnsafeMutableBufferPointer doesn't have an allocating init.

I haven't wanted to waste review cycles on small additive
changes. It may make sense to batch them up into one coherent
proposal. Here are a few more to consider.

- [SR-3929] UnsafeBufferPointer should have init from mutable
- [SR-4340] UnsafeBufferPointer needs a withMemoryRebound method
- [SR-3087] No way to arbitrarily initialise an Array's storage

The feature requests you mention are all very valuable, however with
Michael’s point about fixing the memorystate API’s, the size of this
proposal has already grown to encompass dozens of methods in five types. I
think this says a lot about just how broken the current system is, but I
think it’s better to try to fix one class of problems at a time, and save
the less closely-related issues for separate proposals.

Point by point:

> drop the capacity parameter from UnsafeMutablePointer.allocate() and
deallocate().

I do not agree with removing the capacity parameter and adding a
single-instance allocation API. UnsafePointer was not designed for
single instances, it was primarily designed for C-style arrays. I
don't see the value in providing a different unsafe API for single
vs. multiple values.

Although it’s common to *receive* Unsafe__Pointers from C API’s, it’s rare
to *create* them from the Swift side. 95% of the time your Swift data lives
in a Swift Array, and you use withUnsafePointer(_:slight_smile: to send them to the C
API, or just pass them directly with Array bridging.

The only example I can think of where I had to allocate memory from the
Swift side to pass to a C API is when I was using the Cairo C library and I
wanted the Swift code to own the image buffer backing the Cairo C structs
and I wanted to manage the memory manually to prevent the buffer backing
from getting deallocated prematurely. I think I ended up using
UnsafeMutableBufferPointer and extracting baseAddresses to manage the
memory. This proposal tries to mitigate that pain of extracting
baseAddresses by giving buffer pointers their own memory management methods.

As for the UnsafePointers you get from C APIs, they almost always come with
a size (or you specify it beforehand with a parameter) so you’re probably
going to be turning them into UnsafeBufferPointers anyway.

I also have to say it’s not common to deallocate something in Swift that
you didn’t previously allocate in Swift.

I agree the primary allocation API should be
UnsafeMutableBufferPointer.allocate(capacity:). There is an argument
to be made for removing UnsafeMutablePointer.allocate(capacity:)
entirely. But, as Michael Ilseman pointed out, that would involve
reevaluating several other members of the UnsafePointer API. I think
it's reasonable for UnsafePointer to retain all its functionality as a
lower level API.

I think duplication of functionality is something to be avoided if possible.

I don't understand what is misleading about
UnsafePointer.deallocate(capacity:). It *is* inconvenienent for the
user to keep track of memory capacity. Presumably that was done so
either the implementation can move away from malloc/free or some sort
of memory tracking can be implemented on the standard library
side. Obviously, UnsafeBufferPointer.deallocate() would be cleaner in
most cases.

It’s misleading because it plain doesn’t deallocate `capacity` instances.
It deletes the whole memory block regardless of what you pass in the
capacity argument. If the implementation is ever “fixed” so that it
actually deallocates `capacity` instances, suddenly every source that uses
`deallocate(capacity:)` will break, and *no one will know* until their app
starts mysteriously crashing. If the method is not removed, we will have to
support this behavior to avoid breaking sources, and basically say “yes the
argument label says it deallocates a capacity, but what it *really* does is
free the whole block and we can’t fix it because existing code assumes this
behavior”.

> add an allocate(count:) type method to UnsafeMutableBufferPointer

`capacity` should be used for allocating uninitialized memory not
`count`. `count` should only refer to a number of initialized objects!

We can decide on what the correct term should be, but the current state of
Swift pointers is that *neither* convention is being followed. Just look at
the API for UnsafeMutableRawPointer. It’s a mess. This proposal at the
minimum establishes a consistent convention. It can be revised if you feel
`capacity` is more appropriate than `count`. If what you mean is that it’s
important to maintain the distinction between “initialized counts” and
“uninitialized counts”, well that can be revised in too.

> add a deallocate() instance method to UnsafeMutableBufferPointer

Yes, of course! I added a mention of that in SR-3088.

> remove subscripts from UnsafePointer and UnsafeMutablePointer

It's often more clear to perform arithmetic on C array indices rather
than pointers. That said, I'm happy to push developers to use
UnsafeBufferPointer whenever that have a known capacity. To me, this
is a question of whether the benefit of making a dangerous thing less
convenient is worth breaking source compatibility.

Again, I think this is more about what the real use patterns are. If you
are subscripting into a C array with integers, then UnsafeBufferPointer is
the tool for the job, since it give you Collection conformance. If you
can’t make an UnsafeBufferPointer, it’s probably because you don’t know the
length of the array, and so you’re probably iterating through it one
element at a time. UnsafeMutablePointer.successor() is perfect for this
job. If you want to extract or set fields at fixed but irregular offsets,
UnsafeRawPointer is the tool for the job. But I’m hard-pressed to think of
a use case for random access into a singular typed pointer.

···

On Thu, Jul 13, 2017 at 6:56 PM, Andrew Trick <atrick@apple.com> wrote:

On Jul 12, 2017, at 12:16 PM, Taylor Swift via swift-evolution < > swift-evolution@swift.org> wrote:

I’ve drafted a new version of the unsafe pointer proposal based on feedback I’ve gotten from this thread. You can read it here <https://gist.github.com/kelvin13/1b8ae906be23dff22f7a7c4767f0c907>.

~~~
Swift’s pointer types are an important interface for low-level memory manipulation, but the current API design is not very safe, consistent, or convenient. Many memory methods demand a capacity: or count: argument, forcing the user to manually track the size of the memory block, even though most of the time this is either unnecessary, or redundant as buffer pointers track this information natively. In some places, this design turns UnsafePointers into outright DangerousPointers, leading users to believe that they have allocated or freed memory when in fact, they have not.

The current API suffers from inconsistent naming, poor usage of default argument values, missing methods, and excessive verbosity, and encourages excessively unsafe programming practices. This proposal seeks to iron out these inconsistencies, and offer a more convenient, more sensible, and less bug-prone API for Swift pointers.

The previous draft <https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06> of this proposal was relatively source-breaking, calling for a separation of functionality between singular pointer types and vector (buffer) pointer types. This proposal instead separates functionality between internally-tracked length pointer types and externally-tracked length pointer types. This results in an equally elegant API with about one-third less surface area.

<https://gist.github.com/kelvin13/1b8ae906be23dff22f7a7c4767f0c907>

~~~

remove the capacity parameter from deallocate(capacity:) and deallocate(bytes:alignedTo:)

That's probably for the best.

add unsized memory methods to UnsafeMutableBufferPointer

Yay!

add an assign(to:count:) method to UnsafeMutablePointer and an assign(to:) method to UnsafeMutableBufferPointer

Sure.

add a default value of 1 to all size parameters on UnsafeMutablePointer and applicable
size parameters on UnsafeMutableRawPointer

I'm not opposed to it.

rename copyBytes(from:count:) to copy(from:bytes:)

LGTM in the interest of consistency. I should not have caved on this the first time around.

bytes refers to, well, a byte quantity that is not assumed to be initialized.
capacity refers to a strided quantity that is not assumed to be initialized.
count refers to a strided quantity that is assumed to be initialized.

That's how I see it.

rename count in UnsafeMutableRawBufferPointer.allocate(count:) to bytes and add an
alignedTo parameter to make it UnsafeMutableRawBufferPointer.allocate(bytes:alignedTo:)

Memory allocation is an issue unto itself. I generally prefer your
proposed API. However...

1. Larger-than-pointer alignments aren't currently respected.

2. Users virtually never want to specify the alignment explicitly. They
   just want platform alignment. Unfortunately, there's no reasonable
   "maximal" alignment to use as a default. I think pointer-alignment
   is an excellent default guarantee.

3. The current allocation builtins seem to presume that
   allocation/deallocation can be made more efficient if the user code
   specifies alignment at deallocation. I don't think
   UnsafeRawBufferPointer should expose that to the user, so I agree
   with your proposal. In fact, I think aligned `free` should be
   handled within the Swift runtime.

Resolving these issues requires changes to the Swift runtime API and
implementation. This might be a good time to revisit that design, but
it might slow down the rest of the proposal.

fix the ordering of the arguments in initializeMemory<Element>(as:at:count:to:)

I think this ordering was an attempt to avoid confusion with binding
memory where `to` refers to a type. However, it should be consistent
with `UnsafePointer.initialize`, so we need to pick one of those to
change.

add the sized memorystate functions withMemoryRebound<Element, Result>(to:count:_:) to
UnsafeMutableBufferPointer, and initializeMemory<Element>(as:at:to:count:),
initializeMemory<Element>(as:from:count:) moveInitializeMemory<Element>(as:from:count:),
and bindMemory<Element>(to:count:) to UnsafeMutableRawBufferPointer

Yay!

add mutable overloads to non-vacating memorystate method arguments

I'm not sure removing the need for implicit casts is a goal. I did
that with the pointer `init` methods, but now I think that should be
cleaned up to reduce API surface. I think smaller API surface wins in
these cases. Is there a usability issue you're solving?

add a init(mutating:) initializer to UnsafeMutableBufferPointer

Yes, finally.

remove initialize<C>(from:) from UnsafeMutablePointer

Yep.

adding an initializer UnsafeMutableBufferPointer<Element>.init(allocatingCount:) instead > of a type method to UnsafeMutableBufferPointer

For the record, I strongly prefer a type method for allocation for the reason you mention, it has important side effects beyond simply initializingn the pointer.

-Andy

···

On Jul 17, 2017, at 10:06 PM, Taylor Swift via swift-evolution <swift-evolution@swift.org> wrote:

Hi all, I’ve written up a proposal to modify the unsafe pointer API for greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such as UnsafeMutablePointer, and vector (buffer) pointers such as UnsafeMutableBufferPointer. This implies a natural separation of tasks the two kinds of pointers are meant to do. For example, buffer pointers implement Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied roles. It is possible to allocate an arbitrary number of instances from a type method on a singular pointer, but not from a buffer pointer. The result of such an operation returns a singular pointer, even though a buffer pointer would be more appropriate to capture the information about the number of instances allocated. It’s possible to subscript into a singular pointer, even though they are not real Collections. Some parts of the current design turn UnsafePointers into downright DangerousPointers, leading users to believe that they have allocated or freed memory when in fact, they have not.

This proposal seeks to iron out these inconsistencies, and offer a more convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~

Thanks for taking time to write this up.

General comments:

UnsafeBufferPointer is an API layer on top of UnsafePointer. The role
of UnsafeBufferPointer is direct memory access sans lifetime
management with Collection semantics. The role of UnsafePointer is
primarily C interop. Those C APIs should be wrapped in Swift APIs that
take UnsafeBufferPointer whenever the pointer represents a C array. I
suppose making UnsafePointer less convenient would push developers
toward UnsafeBufferPointer. I don't think that's worth outright
breaking source, but gradual deprecation of convenience methods, like
`susbscript` might be acceptable.

Gradual deprecation is exactly what I am proposing. As the document states <https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06#proposed-solution>, the only methods which should be marked immediately as unavailable are the `deallocate(capacity:)` methods, for safety and source compatibility reasons. Removing `deallocate(capacity:)` now and forcing a loud compiler error prevents catastrophic *silent* source breakage in the future, or worse, from having to *support our own bug*.

I have mixed feelings about stripping UnsafePointer of basic
functionality. Besides breaking source, doing that would be
inconsistent with its role as a lower API layer. The advantage would
just be descreasing API surface area and forcing developers to use a
higher-level API.

UnsafePointer is as much a high level API as UnsafeBufferPointer is.

No it isn’t. We don’t have support for importing certain function signatures as taking UnsafeBufferPointer and UnsafePointer doesn't conform to Collection even though it nearly always represents an array.

You wouldn’t create a buffer pointer of length 1 just so you can “stick with the high level API”. UnsafePointer and UnsafeBufferPointer are two tools that do related but different things and they can exist at whatever abstract level you need them at. After all, UnsafeBufferPointer is nothing but an UnsafePointer? with a length value attached to it. If you’re allocating more than one instance of memory, you almost certainly need to track the length of the buffer anyway.

You could call this a proposal to "make unsafe pointer APIs easier to use safely". I just want to put an end to the fallacy that the buffer type is for multiple values and the plain old pointer represents single instances.

The additive changes you propose are fairly obvious. See [SR-3088]
UnsafeMutableBufferPointer doesn't have an allocating init.

I haven't wanted to waste review cycles on small additive
changes. It may make sense to batch them up into one coherent
proposal. Here are a few more to consider.

- [SR-3929] UnsafeBufferPointer should have init from mutable
- [SR-4340] UnsafeBufferPointer needs a withMemoryRebound method
- [SR-3087] No way to arbitrarily initialise an Array's storage

The feature requests you mention are all very valuable, however with Michael’s point about fixing the memorystate API’s, the size of this proposal has already grown to encompass dozens of methods in five types. I think this says a lot about just how broken the current system is, but I think it’s better to try to fix one class of problems at a time, and save the less closely-related issues for separate proposals.

Point by point:

> drop the capacity parameter from UnsafeMutablePointer.allocate() and deallocate().

I do not agree with removing the capacity parameter and adding a
single-instance allocation API. UnsafePointer was not designed for
single instances, it was primarily designed for C-style arrays. I
don't see the value in providing a different unsafe API for single
vs. multiple values.

Although it’s common to *receive* Unsafe__Pointers from C API’s, it’s rare to *create* them from the Swift side. 95% of the time your Swift data lives in a Swift Array, and you use withUnsafePointer(_:) to send them to the C API, or just pass them directly with Array bridging.

The only example I can think of where I had to allocate memory from the Swift side to pass to a C API is when I was using the Cairo C library and I wanted the Swift code to own the image buffer backing the Cairo C structs and I wanted to manage the memory manually to prevent the buffer backing from getting deallocated prematurely. I think I ended up using UnsafeMutableBufferPointer and extracting baseAddresses to manage the memory. This proposal tries to mitigate that pain of extracting baseAddresses by giving buffer pointers their own memory management methods.

The usability issue with Optional baseAddress is a very real one. I'm unsure why that hasn't been fixed yet (I think that’s between Jordan and Dave). I don't see that as a justification for the broader changes in this proposal.

As for the UnsafePointers you get from C APIs, they almost always come with a size (or you specify it beforehand with a parameter) so you’re probably going to be turning them into UnsafeBufferPointers anyway.

I also have to say it’s not common to deallocate something in Swift that you didn’t previously allocate in Swift.

Yes. You have a good argument for removing allocate/deallocate completely. My point was that I don't want to add a single instance allocate method. UnsafePointer should not be viewed as a single instance pointer, because that's not how it's used.

I agree the primary allocation API should be
UnsafeMutableBufferPointer.allocate(capacity:). There is an argument
to be made for removing UnsafeMutablePointer.allocate(capacity:)
entirely. But, as Michael Ilseman pointed out, that would involve
reevaluating several other members of the UnsafePointer API. I think
it's reasonable for UnsafePointer to retain all its functionality as a
lower level API.

I think duplication of functionality is something to be avoided if possible.

The issue is whether we need to revisit all the initialize/deinitialize/move API surface if we decide that all the uses that can me moved to UnsafeBufferPointer really should be.

I don't understand what is misleading about
UnsafePointer.deallocate(capacity:). It *is* inconvenienent for the
user to keep track of memory capacity. Presumably that was done so
either the implementation can move away from malloc/free or some sort
of memory tracking can be implemented on the standard library
side. Obviously, UnsafeBufferPointer.deallocate() would be cleaner in
most cases.

It’s misleading because it plain doesn’t deallocate `capacity` instances. It deletes the whole memory block regardless of what you pass in the capacity argument. If the implementation is ever “fixed” so that it actually deallocates `capacity` instances, suddenly every source that uses `deallocate(capacity:)` will break, and *no one will know* until their app starts mysteriously crashing. If the method is not removed, we will have to support this behavior to avoid breaking sources, and basically say “yes the argument label says it deallocates a capacity, but what it *really* does is free the whole block and we can’t fix it because existing code assumes this behavior”.

You could have the same problem with slicing up an UnsafeBufferPointer. I agree that this reinforces the argument for eliminating UnsafeMutablePointer.allocate/deallocate. It also reinforces my argument for not adding a single-instance allocate/deallocate.

> add an allocate(count:) type method to UnsafeMutableBufferPointer

`capacity` should be used for allocating uninitialized memory not
`count`. `count` should only refer to a number of initialized objects!

We can decide on what the correct term should be, but the current state of Swift pointers is that *neither* convention is being followed. Just look at the API for UnsafeMutableRawPointer. It’s a mess. This proposal at the minimum establishes a consistent convention. It can be revised if you feel `capacity` is more appropriate than `count`. If what you mean is that it’s important to maintain the distinction between “initialized counts” and “uninitialized counts”, well that can be revised in too.

You lost me. It’s always been clear to me that

a. There are a lot of redundant initializers to avoid relying on automatic conversion. Those should probably be removed now (to the extent that it doesn’t break source).

b. There are a number of convenience methods we should add to the API. But it’s better keep the API minimal until more developers, such as yourself, have had a chance to offer feedback.

I’m not aware of messiness or inconsistent conventions at the API level.

-Andy

···

On Jul 13, 2017, at 6:55 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:
On Thu, Jul 13, 2017 at 6:56 PM, Andrew Trick <atrick@apple.com <mailto:atrick@apple.com>> wrote:

On Jul 12, 2017, at 12:16 PM, Taylor Swift via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

> add a deallocate() instance method to UnsafeMutableBufferPointer

Yes, of course! I added a mention of that in SR-3088.

> remove subscripts from UnsafePointer and UnsafeMutablePointer

It's often more clear to perform arithmetic on C array indices rather
than pointers. That said, I'm happy to push developers to use
UnsafeBufferPointer whenever that have a known capacity. To me, this
is a question of whether the benefit of making a dangerous thing less
convenient is worth breaking source compatibility.

Again, I think this is more about what the real use patterns are. If you are subscripting into a C array with integers, then UnsafeBufferPointer is the tool for the job, since it give you Collection conformance. If you can’t make an UnsafeBufferPointer, it’s probably because you don’t know the length of the array, and so you’re probably iterating through it one element at a time. UnsafeMutablePointer.successor() is perfect for this job. If you want to extract or set fields at fixed but irregular offsets, UnsafeRawPointer is the tool for the job. But I’m hard-pressed to think of a use case for random access into a singular typed pointer.

I am not very familiar with the inner workings of the standard library, however I maintain a few libraries which make extensive use of Swift pointers, such as https://github.com/kelvin13/maxpng which makes extensive use of Unsafe_____Pointers. I also write a lot of code that interfaces with C APIs like Cairo and OpenGL. Most of the ideas in the original proposal came from me dealing with the current Swift pointer APIs in my own code. For example I find myself writing this bit of code

let buffer = UnsafeMutableBufferPointer<UInt8>(start: UnsafeMutablePointer<UInt8>.allocate(capacity: byteCount), count: byteCount)
defer
{
    buffer.baseAddress?.deallocate(capacity: buffer.count)
}

far more than I would like to. While this proposal doesn’t solve every problem with Swift pointers — for example, we need a UMBP initializer that takes an immutable buffer pointer before we are really able to write a lot of examples more concisely, it takes us a great deal closer to being able to write things like

UnsafeMutablePointer(mutating: self.zero_line.baseAddress!).deallocate(capacity: self.zero_line.count)

as

UnsafeMutableBufferPointer(mutating: self.zero_line).deallocate()

You should not need to do this at all. A pointer does not need to be mutable to deallocate the memory. That’s a bug in the UnsafePointer API.

-Andy

···

On Jul 13, 2017, at 6:16 PM, Taylor Swift via swift-evolution <swift-evolution@swift.org> wrote:

On Thu, Jul 13, 2017 at 2:47 PM, Dave Abrahams via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

on Wed Jul 12 2017, Taylor Swift <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

> Hi all, I’ve written up a proposal to modify the unsafe pointer API for
> greater consistency, safety, and ease of use.
>
> ~~~
>
> Swift currently offers two sets of pointer types — singular pointers such
> as UnsafeMutablePointer, and vector (buffer) pointers such as UnsafeMutable
> *Buffer*Pointer. This implies a natural separation of tasks the two kinds
> of pointers are meant to do. For example, buffer pointers implement
> Collection conformance, while singular pointers do not.
>
> However, some aspects of the pointer design contradict these implied roles.
> It is possible to allocate an arbitrary number of instances from a type
> method on a singular pointer, but not from a buffer pointer. The result of
> such an operation returns a singular pointer, even though a buffer pointer
> would be more appropriate to capture the information about the *number* of
> instances allocated. It’s possible to subscript into a singular pointer,
> even though they are not real Collections. Some parts of the current design
> turn UnsafePointers into downright *Dangerous*Pointers, leading users to
> believe that they have allocated or freed memory when in fact, they have
> not.
>
> This proposal seeks to iron out these inconsistencies, and offer a more
> convenient, more sensible, and less bug-prone API for Swift pointers.
>
> <https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

I have no problem with this direction in principle; it sounds like a
good idea. However, before affirming any particular design I would like
to see the results of any such proposal applied to a fairly large body
of code that uses Unsafe[XXX]Pointer today in diverse ways (such as the
Swift standard library itself), to demonstrate that it does in fact
improve the code in practice.

Cheers,

--
-Dave

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Hi all, I’ve written up a proposal to modify the unsafe pointer API for
greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such
as UnsafeMutablePointer, and vector (buffer) pointers such as
UnsafeMutable*Buffer*Pointer. This implies a natural separation of tasks
the two kinds of pointers are meant to do. For example, buffer pointers
implement Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied
roles. It is possible to allocate an arbitrary number of instances from a
type method on a singular pointer, but not from a buffer pointer. The
result of such an operation returns a singular pointer, even though a
buffer pointer would be more appropriate to capture the information about
the *number* of instances allocated. It’s possible to subscript into a
singular pointer, even though they are not real Collections. Some parts
of the current design turn UnsafePointers into downright *Dangerous*Pointers,
leading users to believe that they have allocated or freed memory when in
fact, they have not.

This proposal seeks to iron out these inconsistencies, and offer a more
convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~

Thanks for taking time to write this up.

General comments:

UnsafeBufferPointer is an API layer on top of UnsafePointer. The role
of UnsafeBufferPointer is direct memory access sans lifetime
management with Collection semantics. The role of UnsafePointer is
primarily C interop. Those C APIs should be wrapped in Swift APIs that
take UnsafeBufferPointer whenever the pointer represents a C array. I
suppose making UnsafePointer less convenient would push developers
toward UnsafeBufferPointer. I don't think that's worth outright
breaking source, but gradual deprecation of convenience methods, like
`susbscript` might be acceptable.

Gradual deprecation is exactly what I am proposing. As the document states
<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06#proposed-solution>,
the only methods which should be marked immediately as unavailable are the `
deallocate(capacity:)` methods, for safety and source compatibility
reasons. Removing `deallocate(capacity:)` now and forcing a loud compiler
error prevents catastrophic *silent* source breakage in the future, or
worse, from having to *support our own bug*.

I have mixed feelings about stripping UnsafePointer of basic
functionality. Besides breaking source, doing that would be
inconsistent with its role as a lower API layer. The advantage would
just be descreasing API surface area and forcing developers to use a
higher-level API.

UnsafePointer is as much a high level API as UnsafeBufferPointer is.

No it isn’t. We don’t have support for importing certain function
signatures as taking UnsafeBufferPointer and UnsafePointer doesn't conform
to Collection even though it nearly always represents an array.

C functions get imported as taking UnsafePointers because buffer pointers
in C are represented as pointer–length pairs. UnsafePointer can’t conform
to Collection because there’s no way to automatically associate that length
information with the pointer. So why do you think UnsafePointers should
support doing Collection-y things if a precondition for performing
collection-y operations is knowing the length? No sense exposing a `capacity`
argument if you don’t have a `count` to go in it. That’s why I don’t really
agree with viewing UnsafePointers as arrays, since trying to do random
access into something you don’t know the length of seems off to me.

You wouldn’t create a buffer pointer of length 1 just so you can “stick
with the high level API”. UnsafePointer and UnsafeBufferPointer are two
tools that do related but different things and they can exist at whatever
abstract level you need them at. After all, UnsafeBufferPointer is nothing
but an UnsafePointer? with a length value attached to it. If you’re
allocating more than one instance of memory, you almost certainly need to
track the length of the buffer anyway.

You could call this a proposal to "make unsafe pointer APIs easier to use
safely". I just want to put an end to the fallacy that the buffer type is
for multiple values and the plain old pointer represents single instances.

I mean, parts of the current API do heavily suggest that the plain pointer
is for single instances — there’s the `pointee` property after all. `move()`
vacates and returns a single element. In fact, I’m not sure when you would
initialize multiple values, and then only vacate the one at offset 0. `
successor()` and `predecessor()` only make sense if a pointer represents
one single thing — it’d be weird if they represented some kind of weird
sliding window where one end is `pointee` and the other end is… ???.

My point being, you can only (safely) use the multiple-instance API if you
have access to the count of elements. But if you have access to the count, *you
effectively have a BufferPointer*. So why not put the buffer tools in the
buffer shed?

The additive changes you propose are fairly obvious. See [SR-3088]

UnsafeMutableBufferPointer doesn't have an allocating init.

I haven't wanted to waste review cycles on small additive
changes. It may make sense to batch them up into one coherent
proposal. Here are a few more to consider.

- [SR-3929] UnsafeBufferPointer should have init from mutable
- [SR-4340] UnsafeBufferPointer needs a withMemoryRebound method
- [SR-3087] No way to arbitrarily initialise an Array's storage

The feature requests you mention are all very valuable, however with
Michael’s point about fixing the memorystate API’s, the size of this
proposal has already grown to encompass dozens of methods in five types. I
think this says a lot about just how broken the current system is, but I
think it’s better to try to fix one class of problems at a time, and save
the less closely-related issues for separate proposals.

Point by point:

> drop the capacity parameter from UnsafeMutablePointer.allocate() and
deallocate().

I do not agree with removing the capacity parameter and adding a
single-instance allocation API. UnsafePointer was not designed for
single instances, it was primarily designed for C-style arrays. I
don't see the value in providing a different unsafe API for single
vs. multiple values.

Although it’s common to *receive* Unsafe__Pointers from C API’s, it’s rare
to *create* them from the Swift side. 95% of the time your Swift data lives
in a Swift Array, and you use withUnsafePointer(_:slight_smile: to send them to the C
API, or just pass them directly with Array bridging.

The only example I can think of where I had to allocate memory from the
Swift side to pass to a C API is when I was using the Cairo C library and I
wanted the Swift code to own the image buffer backing the Cairo C structs
and I wanted to manage the memory manually to prevent the buffer backing
from getting deallocated prematurely. I think I ended up using
UnsafeMutableBufferPointer and extracting baseAddresses to manage the
memory. This proposal tries to mitigate that pain of extracting
baseAddresses by giving buffer pointers their own memory management methods.

The usability issue with Optional baseAddress is a very real one. I'm
unsure why that hasn't been fixed yet (I think that’s between Jordan and
Dave). I don't see that as a justification for the broader changes in this
proposal.

This was actually part of the first drafts
<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06/cc3d7c349e5f5600ae592ee394438f68068df15b>
of the proposal. I was told this had already been discussed at length, and
the community supposedly decided against it a long time ago, so it was
removed from the proposal.

As for the UnsafePointers you get from C APIs, they almost always come
with a size (or you specify it beforehand with a parameter) so you’re
probably going to be turning them into UnsafeBufferPointers anyway.

I also have to say it’s not common to deallocate something in Swift that
you didn’t previously allocate in Swift.

Yes. You have a good argument for removing allocate/deallocate completely.
My point was that I don't want to add a single instance allocate method.
UnsafePointer should not be viewed as a single instance pointer, because
that's not how it's used.

So, should `UnsafeMutableBufferPointer<Element>.allocate(count:
1).baseAddress!` be the preferred way to allocate a single instance?
Or `UnsafeMutablePointer<T>.allocate(count:
1)` if we keep 2 sets of APIs? Writing “count: 1” seems kind of strange to
me.

I agree the primary allocation API should be

UnsafeMutableBufferPointer.allocate(capacity:). There is an argument
to be made for removing UnsafeMutablePointer.allocate(capacity:)
entirely. But, as Michael Ilseman pointed out, that would involve
reevaluating several other members of the UnsafePointer API. I think
it's reasonable for UnsafePointer to retain all its functionality as a
lower level API.

I think duplication of functionality is something to be avoided if
possible.

The issue is whether we need to revisit all the
initialize/deinitialize/move API surface if we decide that all the uses
that can me moved to UnsafeBufferPointer really should be.

I was working that out earlier today. The latest version
<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>
outlines exactly what I think should happen to all the memorystate
functions.

I don't understand what is misleading about

UnsafePointer.deallocate(capacity:). It *is* inconvenienent for the
user to keep track of memory capacity. Presumably that was done so
either the implementation can move away from malloc/free or some sort
of memory tracking can be implemented on the standard library
side. Obviously, UnsafeBufferPointer.deallocate() would be cleaner in
most cases.

It’s misleading because it plain doesn’t deallocate `capacity` instances.
It deletes the whole memory block regardless of what you pass in the
capacity argument. If the implementation is ever “fixed” so that it
actually deallocates `capacity` instances, suddenly every source that uses
`deallocate(capacity:)` will break, and *no one will know* until their app
starts mysteriously crashing. If the method is not removed, we will have to
support this behavior to avoid breaking sources, and basically say “yes the
argument label says it deallocates a capacity, but what it *really* does is
free the whole block and we can’t fix it because existing code assumes this
behavior”.

You could have the same problem with slicing up an UnsafeBufferPointer. I
agree that this reinforces the argument for eliminating
UnsafeMutablePointer.allocate/deallocate. It also reinforces my argument
for not adding a single-instance allocate/deallocate.

You have a good point here, it’s still not very safe, but I hold that it is
still far, far safer than what we have currently. Getting rid of the
`capacity` label helps drive home that swift_slowDealloc doesn’t care what
number of instances you want it to free. Unsafe__Pointers will always be
unsafe, but if we can make them less unsafe, we should.

I’ll admit this is a good argument against “single instance” deallocate. We
don’t want people trying to free each address in a buffer pointer :fearful:. But
it’s not a direct argument against single instance allocate, which I think
would be very useful, and it would be weird to have single instance allocate
but no corresponding deallocate. We also have to consider what happens to
the proposed single-instance memorystate functions, as those *are* safe and
*are* useful. Should single-instance deallocate be the one missing
function? I agree this is definitely something to think carefully about.

add an allocate(count:) type method to UnsafeMutableBufferPointer

`capacity` should be used for allocating uninitialized memory not
`count`. `count` should only refer to a number of initialized objects!

We can decide on what the correct term should be, but the current state of
Swift pointers is that *neither* convention is being followed. Just look at
the API for UnsafeMutableRawPointer. It’s a mess. This proposal at the
minimum establishes a consistent convention. It can be revised if you feel
`capacity` is more appropriate than `count`. If what you mean is that it’s
important to maintain the distinction between “initialized counts” and
“uninitialized counts”, well that can be revised in too.

You lost me. It’s always been clear to me that

a. There are a lot of redundant initializers to avoid relying on automatic
conversion. Those should probably be removed now (to the extent that it
doesn’t break source).

b. There are a number of convenience methods we should add to the API. But
it’s better keep the API minimal until more developers, such as yourself,
have had a chance to offer feedback.

I’m not aware of messiness or inconsistent conventions at the API level.

-Andy

I’m confused I thought we were talking about the naming choices for the
argument labels in those functions. I think defining and abiding by
consistent meanings for `count`, `capacity`, and `bytes` is a good idea,
and it’s part of what this proposal tries to accomplish. Right now half the
time we use `count` to refer to “bytes” and half the time we use it to
refer to “instances”. The same goes for the word “capacity”. This is all
laid out in the document:

“““
*Finally, the naming and design of some UnsafeMutableRawPointer members
deserves to be looked at. The usage of capacity, bytes, and count as
argument labels is wildly inconsistent and confusing. In
copyBytes(from:count:), count refers to the number of bytes, while in
initializeMemory<T>(as:at:count:to:) and
initializeMemory<T>(as:from:count:), count refers to the number of strides.
Meanwhile bindMemory<T>(to:capacity:) uses capacity to refer to this
quantity. The always-problematic deallocate(bytes:alignedTo) method and
allocate(bytes:alignedTo:) type methods use bytes to refer to
byte-quantities. Adding to the confusion, UnsafeMutableRawBufferPointer
offers an allocate(count:) type method (the same signature method we’re
trying to add to UnsafeMutableBufferPointer), except the count in this
method refers to bytes. This kind of API naming begets stride bugs and
makes Swift needlessly difficult to learn.*
”””

The only convenience methods this proposal is trying to add is the
functionality on the buffer pointer types. There seems to be broad support
for adding this functionality as no one has really opposed that part of the
proposal yet. Any other new methods like `UnsafeMutablePointer.assign(to:)`
are there for API consistency.

This proposal also calls for getting rid of one of those “redundant
initializers” :slight_smile:

> add a deallocate() instance method to UnsafeMutableBufferPointer

Yes, of course! I added a mention of that in SR-3088.

> remove subscripts from UnsafePointer and UnsafeMutablePointer

It's often more clear to perform arithmetic on C array indices rather
than pointers. That said, I'm happy to push developers to use
UnsafeBufferPointer whenever that have a known capacity. To me, this
is a question of whether the benefit of making a dangerous thing less
convenient is worth breaking source compatibility.

Again, I think this is more about what the real use patterns are. If you
are subscripting into a C array with integers, then UnsafeBufferPointer is
the tool for the job, since it give you Collection conformance. If you
can’t make an UnsafeBufferPointer, it’s probably because you don’t know the
length of the array, and so you’re probably iterating through it one
element at a time. UnsafeMutablePointer.successor() is perfect for this
job. If you want to extract or set fields at fixed but irregular offsets,
UnsafeRawPointer is the tool for the job. But I’m hard-pressed to think of
a use case for random access into a singular typed pointer.

Thanks for your feedback on my proposal. You’ve given some very helpful
considerations about some of these changes.

···

On Fri, Jul 14, 2017 at 12:22 AM, Andrew Trick <atrick@apple.com> wrote:

On Jul 13, 2017, at 6:55 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:
On Thu, Jul 13, 2017 at 6:56 PM, Andrew Trick <atrick@apple.com> wrote:

On Jul 12, 2017, at 12:16 PM, Taylor Swift via swift-evolution < >> swift-evolution@swift.org> wrote:

Hi all, I’ve written up a proposal to modify the unsafe pointer API for greater consistency, safety, and ease of use.

~~~

Swift currently offers two sets of pointer types — singular pointers such as UnsafeMutablePointer, and vector (buffer) pointers such as UnsafeMutableBufferPointer. This implies a natural separation of tasks the two kinds of pointers are meant to do. For example, buffer pointers implement Collection conformance, while singular pointers do not.

However, some aspects of the pointer design contradict these implied roles. It is possible to allocate an arbitrary number of instances from a type method on a singular pointer, but not from a buffer pointer. The result of such an operation returns a singular pointer, even though a buffer pointer would be more appropriate to capture the information about the number of instances allocated. It’s possible to subscript into a singular pointer, even though they are not real Collections. Some parts of the current design turn UnsafePointers into downright DangerousPointers, leading users to believe that they have allocated or freed memory when in fact, they have not.

This proposal seeks to iron out these inconsistencies, and offer a more convenient, more sensible, and less bug-prone API for Swift pointers.

<https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06>

~~~

Thanks for taking time to write this up.

General comments:

UnsafeBufferPointer is an API layer on top of UnsafePointer. The role
of UnsafeBufferPointer is direct memory access sans lifetime
management with Collection semantics. The role of UnsafePointer is
primarily C interop. Those C APIs should be wrapped in Swift APIs that
take UnsafeBufferPointer whenever the pointer represents a C array. I
suppose making UnsafePointer less convenient would push developers
toward UnsafeBufferPointer. I don't think that's worth outright
breaking source, but gradual deprecation of convenience methods, like
`susbscript` might be acceptable.

Gradual deprecation is exactly what I am proposing. As the document states <https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06#proposed-solution>, the only methods which should be marked immediately as unavailable are the `deallocate(capacity:)` methods, for safety and source compatibility reasons. Removing `deallocate(capacity:)` now and forcing a loud compiler error prevents catastrophic *silent* source breakage in the future, or worse, from having to *support our own bug*.

I have mixed feelings about stripping UnsafePointer of basic
functionality. Besides breaking source, doing that would be
inconsistent with its role as a lower API layer. The advantage would
just be descreasing API surface area and forcing developers to use a
higher-level API.

UnsafePointer is as much a high level API as UnsafeBufferPointer is. You wouldn’t create a buffer pointer of length 1 just so you can “stick with the high level API”. UnsafePointer and UnsafeBufferPointer are two tools that do related but different things and they can exist at whatever abstract level you need them at. After all, UnsafeBufferPointer is nothing but an UnsafePointer? with a length value attached to it. If you’re allocating more than one instance of memory, you almost certainly need to track the length of the buffer anyway.

I disagree. UnsafePointer can be viewed as the low level representation of Swift’s memory model. UnsafeBufferPointer is a convenient facility for programming at this level, as illustrated by e.g. RandomAccessCollection conformance. One could drop UnsafeBufferPointer from the standard library without compromising Swift’s core capabilities, it would just be a large convenience regression.

UnsafePointer is a low level necessity; UnsafeBufferPointer is a low level convenience.

The additive changes you propose are fairly obvious. See [SR-3088]
UnsafeMutableBufferPointer doesn't have an allocating init.

I haven't wanted to waste review cycles on small additive
changes. It may make sense to batch them up into one coherent
proposal. Here are a few more to consider.

- [SR-3929] UnsafeBufferPointer should have init from mutable
- [SR-4340] UnsafeBufferPointer needs a withMemoryRebound method
- [SR-3087] No way to arbitrarily initialise an Array's storage

The feature requests you mention are all very valuable, however with Michael’s point about fixing the memorystate API’s, the size of this proposal has already grown to encompass dozens of methods in five types. I think this says a lot about just how broken the current system is

I fail to see how the current system is broken. It accurately reflects Swift’s memory model while providing the facilities to stay in the bounds of defined behavior. I do agree that it would be far more convenient for UnsafeBufferPointer to provide allocation/deallocation and initialization/deinitialization convenience functionality as well, as I’ve often wanted it myself.

, but I think it’s better to try to fix one class of problems at a time, and save the less closely-related issues for separate proposals.

These seem very closely related. Isn’t the overall goal of this proposal to have API parity for UnsafeBufferPointer?

Point by point:

> drop the capacity parameter from UnsafeMutablePointer.allocate() and deallocate().

I do not agree with removing the capacity parameter and adding a
single-instance allocation API. UnsafePointer was not designed for
single instances, it was primarily designed for C-style arrays. I
don't see the value in providing a different unsafe API for single
vs. multiple values.

Although it’s common to *receive* Unsafe__Pointers from C API’s, it’s rare to *create* them from the Swift side. 95% of the time your Swift data lives in a Swift Array, and you use withUnsafePointer(_:slight_smile: to send them to the C API, or just pass them directly with Array bridging.

The only example I can think of where I had to allocate memory from the Swift side to pass to a C API is when I was using the Cairo C library and I wanted the Swift code to own the image buffer backing the Cairo C structs and I wanted to manage the memory manually to prevent the buffer backing from getting deallocated prematurely. I think I ended up using UnsafeMutableBufferPointer and extracting baseAddresses to manage the memory. This proposal tries to mitigate that pain of extracting baseAddresses by giving buffer pointers their own memory management methods.

As for the UnsafePointers you get from C APIs, they almost always come with a size (or you specify it beforehand with a parameter) so you’re probably going to be turning them into UnsafeBufferPointers anyway.

I also have to say it’s not common to deallocate something in Swift that you didn’t previously allocate in Swift.

I’m not sure it is even defined to use `deallocate` on memory that wasn’t `allocated` in Swift. Andy, thoughts here? Perhaps more clarity in deallocate’s documentation is needed?

I agree the primary allocation API should be
UnsafeMutableBufferPointer.allocate(capacity:). There is an argument
to be made for removing UnsafeMutablePointer.allocate(capacity:)
entirely. But, as Michael Ilseman pointed out, that would involve
reevaluating several other members of the UnsafePointer API. I think
it's reasonable for UnsafePointer to retain all its functionality as a
lower level API.

I think duplication of functionality is something to be avoided if possible.

You have to weigh duplication against convenience. If avoiding duplication is more important, then we should keep the APIs only on UnsafePointer. But, I would prefer convenience in this scenario.

I don't understand what is misleading about
UnsafePointer.deallocate(capacity:). It *is* inconvenienent for the
user to keep track of memory capacity. Presumably that was done so
either the implementation can move away from malloc/free or some sort
of memory tracking can be implemented on the standard library
side. Obviously, UnsafeBufferPointer.deallocate() would be cleaner in
most cases.

It’s misleading because it plain doesn’t deallocate `capacity` instances. It deletes the whole memory block regardless of what you pass in the capacity argument. If the implementation is ever “fixed” so that it actually deallocates `capacity` instances, suddenly every source that uses `deallocate(capacity:)` will break, and *no one will know* until their app starts mysteriously crashing. If the method is not removed, we will have to support this behavior to avoid breaking sources, and basically say “yes the argument label says it deallocates a capacity, but what it *really* does is free the whole block and we can’t fix it because existing code assumes this behavior”.

Note that many of these things are basically programming at the memory semantics level, not necessarily at a execution level.

> add an allocate(count:) type method to UnsafeMutableBufferPointer

`capacity` should be used for allocating uninitialized memory not
`count`. `count` should only refer to a number of initialized objects!

We can decide on what the correct term should be, but the current state of Swift pointers is that *neither* convention is being followed. Just look at the API for UnsafeMutableRawPointer. It’s a mess. This proposal at the minimum establishes a consistent convention. It can be revised if you feel `capacity` is more appropriate than `count`. If what you mean is that it’s important to maintain the distinction between “initialized counts” and “uninitialized counts”, well that can be revised in too.

So I just looked over all of the APIs, and they are very consistent in their use of `count` and `capacity`. Where are you seeing a violation?

> add a deallocate() instance method to UnsafeMutableBufferPointer

Yes, of course! I added a mention of that in SR-3088.

> remove subscripts from UnsafePointer and UnsafeMutablePointer

It's often more clear to perform arithmetic on C array indices rather
than pointers. That said, I'm happy to push developers to use
UnsafeBufferPointer whenever that have a known capacity. To me, this
is a question of whether the benefit of making a dangerous thing less
convenient is worth breaking source compatibility.

Again, I think this is more about what the real use patterns are. If you are subscripting into a C array with integers, then UnsafeBufferPointer is the tool for the job, since it give you Collection conformance. If you can’t make an UnsafeBufferPointer, it’s probably because you don’t know the length of the array, and so you’re probably iterating through it one element at a time. UnsafeMutablePointer.successor() is perfect for this job. If you want to extract or set fields at fixed but irregular offsets, UnsafeRawPointer is the tool for the job. But I’m hard-pressed to think of a use case for random access into a singular typed pointer.

I’m also very interested in more convenience facilities for low level programming and I view UnsafeBufferPointer as the first low level convenience construct. The point of the very careful dance that the Unsafe*Pointers go through is to help the programmer avoid unwittingly falling into undefined behavior.

···

On Jul 13, 2017, at 6:55 PM, Taylor Swift via swift-evolution <swift-evolution@swift.org> wrote:
On Thu, Jul 13, 2017 at 6:56 PM, Andrew Trick <atrick@apple.com <mailto:atrick@apple.com>> wrote:

On Jul 12, 2017, at 12:16 PM, Taylor Swift via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

I agree with all of Andy’s points. I really like this and think it’s a good time to start discussing its details and moving from a pitch to a proposal. Thank you for writing it!

Minor tweak: say “deprecate” instead of “remove” for APIs, which has a better connotation with respect to source compatibility. While I want to have the best APIs, it’s important to make migration smooth. For example, see https://github.com/apple/swift-evolution/blob/master/proposals/0180-string-index-overhaul.md#source-compatibility

The main thing that I think needs to be massaged before a formal proposal is the introduction and motivation section. It contains hyperbole that is distracting and misleading. Some examples:

> Introduction
> …
> but the current API design is not very safe, consistent, or convenient.

This proposal does not address “safe” or unsafety. I think the proposal is very good and important for addressing consistency and convenience, which help encourage programmers to use APIs correctly, but “not very safe” is orthogonal to the proposal.

> In some places, this design turns UnsafePointers into outright DangerousPointers, leading users to believe that they have allocated or freed memory when in fact, they have not.

I see nothing in this proposal that identifies, nor address UnsafePointers as being “DangerousPointers”. This proposal seeks to change idiomatic use to be more consistent and convenient, which is very important, but does not change what “Unsafe” means in Swift. Near as I can tell, the semantics and “dangerousness" of Unsafe*Pointers are unchanged by this proposal.

> The current API suffers from inconsistent naming, poor usage of default argument values, missing methods, and excessive verbosity, and encourages excessively unsafe programming practices.

I agree with everything up until “excessively unsafe programming practices”. What do you mean by “unsafe”? What practice is that? It seems like you might have a very different definition of Unsafe than Swift does. If so, then this will be a very different sort of proposal and you should identify what you mean by “unsafe”.

> This proposal seeks to iron out these inconsistencies, and offer a more convenient, more sensible, and less bug-prone API for Swift pointers.

100% agree and I’m super enthusiastic for this proposal for this reason! My main feedback is to align the motivation and pitch with the message, unless you really do have a different definition of “unsafe” that you’re wanting to pitch.

> This results in an equally elegant API with about one-third less surface area.

:tada:

> Motivation:
> Right now, UnsafeMutableBufferPointer is kind of a black box. To do anything with the memory block it represents, you have to extract baseAddresses and counts. This is unfortunate because UnsafeMutableBufferPointer provides a handy container for tracking the size of a memory buffer, but to actually make use of this information, the buffer pointer must be disassembled.

Note that Unsafe*BufferPointer conforms to RandomAccessCollection and thus gets all the same conveniences of anything else that is Array-like. This means that after it has been properly allocated, initialized, and pointer-casted, it is very convenient for consumers of Unsafe*BufferPointers. The main pain points, and this proposal is excellent at addressing, are on the producers of Unsafe*BufferPointers. For producers, there are a lot of rules and hoops to jump through and the APIs are not conveniently aligned with them. I do think you’ve done a great job of correctly identifying the pain points for people who need to produce Unsafe*BufferPointers.

The rest of the motivation section is excellent! I have done every single “idiom” you highlight and hated having to do it.

···

On Jul 18, 2017, at 11:19 AM, Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:

On Jul 17, 2017, at 10:06 PM, Taylor Swift via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I’ve drafted a new version of the unsafe pointer proposal based on feedback I’ve gotten from this thread. You can read it here <https://gist.github.com/kelvin13/1b8ae906be23dff22f7a7c4767f0c907>.

~~~
Swift’s pointer types are an important interface for low-level memory manipulation, but the current API design is not very safe, consistent, or convenient. Many memory methods demand a capacity: or count: argument, forcing the user to manually track the size of the memory block, even though most of the time this is either unnecessary, or redundant as buffer pointers track this information natively. In some places, this design turns UnsafePointers into outright DangerousPointers, leading users to believe that they have allocated or freed memory when in fact, they have not.

The current API suffers from inconsistent naming, poor usage of default argument values, missing methods, and excessive verbosity, and encourages excessively unsafe programming practices. This proposal seeks to iron out these inconsistencies, and offer a more convenient, more sensible, and less bug-prone API for Swift pointers.

The previous draft <https://gist.github.com/kelvin13/a9c033193a28b1d4960a89b25fbffb06> of this proposal was relatively source-breaking, calling for a separation of functionality between singular pointer types and vector (buffer) pointer types. This proposal instead separates functionality between internally-tracked length pointer types and externally-tracked length pointer types. This results in an equally elegant API with about one-third less surface area.

<https://gist.github.com/kelvin13/1b8ae906be23dff22f7a7c4767f0c907>

~~~

> remove the capacity parameter from deallocate(capacity:) and deallocate(bytes:alignedTo:)

That's probably for the best.

> add unsized memory methods to UnsafeMutableBufferPointer

Yay!

> add an assign(to:count:) method to UnsafeMutablePointer and an assign(to:) method to UnsafeMutableBufferPointer

Sure.

> add a default value of 1 to all size parameters on UnsafeMutablePointer and applicable
> size parameters on UnsafeMutableRawPointer

I'm not opposed to it.

> rename copyBytes(from:count:) to copy(from:bytes:)

LGTM in the interest of consistency. I should not have caved on this the first time around.

> bytes refers to, well, a byte quantity that is not assumed to be initialized.
> capacity refers to a strided quantity that is not assumed to be initialized.
> count refers to a strided quantity that is assumed to be initialized.

That's how I see it.

> rename count in UnsafeMutableRawBufferPointer.allocate(count:) to bytes and add an
> alignedTo parameter to make it UnsafeMutableRawBufferPointer.allocate(bytes:alignedTo:)

Memory allocation is an issue unto itself. I generally prefer your
proposed API. However...

1. Larger-than-pointer alignments aren't currently respected.

2. Users virtually never want to specify the alignment explicitly. They
   just want platform alignment. Unfortunately, there's no reasonable
   "maximal" alignment to use as a default. I think pointer-alignment
   is an excellent default guarantee.

3. The current allocation builtins seem to presume that
   allocation/deallocation can be made more efficient if the user code
   specifies alignment at deallocation. I don't think
   UnsafeRawBufferPointer should expose that to the user, so I agree
   with your proposal. In fact, I think aligned `free` should be
   handled within the Swift runtime.

Resolving these issues requires changes to the Swift runtime API and
implementation. This might be a good time to revisit that design, but
it might slow down the rest of the proposal.

> fix the ordering of the arguments in initializeMemory<Element>(as:at:count:to:)

I think this ordering was an attempt to avoid confusion with binding
memory where `to` refers to a type. However, it should be consistent
with `UnsafePointer.initialize`, so we need to pick one of those to
change.

> add the sized memorystate functions withMemoryRebound<Element, Result>(to:count:_:slight_smile: to
> UnsafeMutableBufferPointer, and initializeMemory<Element>(as:at:to:count:),
> initializeMemory<Element>(as:from:count:) moveInitializeMemory<Element>(as:from:count:),
> and bindMemory<Element>(to:count:) to UnsafeMutableRawBufferPointer

Yay!

> add mutable overloads to non-vacating memorystate method arguments

I'm not sure removing the need for implicit casts is a goal. I did
that with the pointer `init` methods, but now I think that should be
cleaned up to reduce API surface. I think smaller API surface wins in
these cases. Is there a usability issue you're solving?

> add a init(mutating:) initializer to UnsafeMutableBufferPointer

Yes, finally.

> remove initialize<C>(from:) from UnsafeMutablePointer

Yep.

> adding an initializer UnsafeMutableBufferPointer<Element>.init(allocatingCount:) instead > of a type method to UnsafeMutableBufferPointer

For the record, I strongly prefer a type method for allocation for the reason you mention, it has important side effects beyond simply initializingn the pointer.

-Andy
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

> rename count in UnsafeMutableRawBufferPointer.allocate(count:) to bytes
and add an
> alignedTo parameter to make it UnsafeMutableRawBufferPointer.
allocate(bytes:alignedTo:)

Memory allocation is an issue unto itself. I generally prefer your
proposed API. However...

1. Larger-than-pointer alignments aren't currently respected.

2. Users virtually never want to specify the alignment explicitly. They
   just want platform alignment. Unfortunately, there's no reasonable
   "maximal" alignment to use as a default. I think pointer-alignment
   is an excellent default guarantee.

3. The current allocation builtins seem to presume that
   allocation/deallocation can be made more efficient if the user code
   specifies alignment at deallocation. I don't think
   UnsafeRawBufferPointer should expose that to the user, so I agree
   with your proposal. In fact, I think aligned `free` should be
   handled within the Swift runtime.

Resolving these issues requires changes to the Swift runtime API and
implementation. This might be a good time to revisit that design, but
it might slow down the rest of the proposal.

I don’t know much about the Swift runtime, but for now at least, it might
be a good idea to emit a runtime error if an alignment larger than a
pointer is specified, if only when built in debug mode.

> fix the ordering of the arguments in initializeMemory<Element>(as:
at:count:to:)

I think this ordering was an attempt to avoid confusion with binding
memory where `to` refers to a type. However, it should be consistent
with `UnsafePointer.initialize`, so we need to pick one of those to
change.

This would be a non-issue had we just been consistent with the rest of the
stdlib and named this argument `repeating:` instead of `to:`. But
`ptr.initialize(repeating:
255, count: 100)` doesn’t read quite as naturally in English as
`ptr.initialize(to:
255, count: 100)` which is why I left this idea out of the proposal. Now
that you mention the problem with `initializeMemory<Element>(as:
at:count:to:)`, it might be a good idea to add this rename back into it.

> add the sized memorystate functions withMemoryRebound<Element,
>(to:count:_:slight_smile: to
> UnsafeMutableBufferPointer, and initializeMemory<Element>(as:
at:to:count:),
> initializeMemory<Element>(as:from:count:) moveInitializeMemory<Element>(
as:from:count:),
> and bindMemory<Element>(to:count:) to UnsafeMutableRawBufferPointer

Yay!

> add mutable overloads to non-vacating memorystate method arguments

I'm not sure removing the need for implicit casts is a goal. I did
that with the pointer `init` methods, but now I think that should be
cleaned up to reduce API surface. I think smaller API surface wins in
these cases. Is there a usability issue you're solving?

Yes, I can imagine initializing a mutable pointer to some values, and then
wanting to use that pointer as a source to initialize more buffers. Having
to convert a mutable pointer to an immutable pointer is annoying because a
function that takes an immutable pointer obviously shouldn’t care if the
pointer could be mutated anyway. It’s like having to rebind a `var`
variable to a `let` constant before passing it as any non-inout argument to
a function, since function parameters are immutable. At any rate, this only
applies to two out of the seven memorystate operations, so comparably, it’s
not a big API expansion at all.

···

On Tue, Jul 18, 2017 at 2:18 PM, Andrew Trick <atrick@apple.com> wrote:

> add a init(mutating:) initializer to UnsafeMutableBufferPointer

Yes, finally.

> remove initialize<C>(from:) from UnsafeMutablePointer

Yep.

> adding an initializer UnsafeMutableBufferPointer<
>.init(allocatingCount:) instead > of a type method to
UnsafeMutableBufferPointer

For the record, I strongly prefer a type method for allocation for the
reason you mention, it has important side effects beyond simply
initializingn the pointer.

-Andy

Since we’re not bike-shedding the specifics yet, I’ll just give you some background.

We would ultimately like APIs that allocate and initialize in one go. It’s important that the current lower-level (dangerous) APIs make a clear distinction between initialized and uninitialized memory to avoid confusing them with future (safer) APIs. `capacity` always refers to memory that may be uninitialized. I think that’s very clear and helpful.

In the context of pointers `count` should always be in strides. For raw pointers, that happens to be the same as as `bytes`.

I initially proposed copy(bytes:from:), but someone thought that `bytes` in this particular context did not properly convey the "count of bytes" as opposed to the source of the bytes. You’re right, that’s inconsistent with allocate/deallocate(bytes:), because allocateBytes(count:) would be silly. Just be aware that the inconsistency is a result of over-thinking and excessive bike shedding to the detriment of something that looks nice and is easy to remember.

I should also point out that the inconsistencies in functionality across pointer types, in terms of collection support and other convenience, is also known but was deliberately stripped from proposals as “additive”.

-Andy

···

On Jul 13, 2017, at 10:30 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

I’m confused I thought we were talking about the naming choices for the argument labels in those functions. I think defining and abiding by consistent meanings for `count`, `capacity`, and `bytes` is a good idea, and it’s part of what this proposal tries to accomplish. Right now half the time we use `count` to refer to “bytes” and half the time we use it to refer to “instances”. The same goes for the word “capacity”. This is all laid out in the document:

“““
Finally, the naming and design of some UnsafeMutableRawPointer members deserves to be looked at. The usage of capacity, bytes, and count as argument labels is wildly inconsistent and confusing. In copyBytes(from:count:), count refers to the number of bytes, while in initializeMemory<T>(as:at:count:to:) and initializeMemory<T>(as:from:count:), count refers to the number of strides. Meanwhile bindMemory<T>(to:capacity:) uses capacity to refer to this quantity. The always-problematic deallocate(bytes:alignedTo) method and allocate(bytes:alignedTo:) type methods use bytes to refer to byte-quantities. Adding to the confusion, UnsafeMutableRawBufferPointer offers an allocate(count:) type method (the same signature method we’re trying to add to UnsafeMutableBufferPointer), except the count in this method refers to bytes. This kind of API naming begets stride bugs and makes Swift needlessly difficult to learn.
”””

The only convenience methods this proposal is trying to add is the functionality on the buffer pointer types. There seems to be broad support for adding this functionality as no one has really opposed that part of the proposal yet. Any other new methods like `UnsafeMutablePointer.assign(to:)` are there for API consistency.

This proposal also calls for getting rid of one of those “redundant initializers” :)