[Review] SE-0184: Unsafe[Mutable][Raw][Buffer]Pointer: add missing methods, adjust existing labels for clarity, and remove deallocation size

[Andrew_Trick]

Yeah, but you’re deprecating the old API entry point anyway. The `at` argument was never used AFAICT. I doubt code is using the default `count` and it seems like a fixit could handle it.

-Andy

···

On Sep 5, 2017, at 6:28 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

UMRP (raw pointer):
--- func initializeMemory<T>(as:at:(=0)count:(1)to:)
+++ func initializeMemory<T>(as:repeating:count:) // remove default count

still extremely suspicious of this but i’m willing to compromise. also there should be an `initializeMemory<T>(at:to:)` to match the typed methods

Do you mean initializeMemory<T>(as:to:)?

I don’t think it’s necessary since raw buffers are not normally used to hold a single typed element. We don’t need symmetry between raw and typed pointers and I wouldn’t want to add an API that will never be used. So, I would be ok with it only if there’s some use case that I’m overlooking.

The only case i can think of is initializing a buffer header but then again i rarely use raw pointers so i’m not sure how much this feature would be used. However, this function is already <https://developer.apple.com/documentation/swift/unsafemutablerawpointer/2427589-initializememory&gt; in the API (with a default count of 1 and offset of 0) so there’s that.

[taylorswift]

In the new raw initializeMemory methods, Xiaodi and I agreed to make
it more clear that the offset is in terms of `self` rather than
`from`, and to further reduce ambiguity by forcing manual stride
computation and using an explicit "offset" label. The call site will
be just as explicit as dropping down to `baseAddress` but without any
pointer conversion boilerplate.

UMRBP (raw buffer):
+++ func initializeMemory<T>(atByteOffset:as:from:)
+++ func moveInitializeMemory<T>(atByteOffset:as:from:)

Agree, but the label should just be `atByte:`, not `atByteOffset:`.
after all, we don’t use `atOffset:` in the strided case; its obvious
that it’s an offset

The existing APIs use the terminology "byte offset"--for example,
URP.load(fromByteOffset:as:). The rationale is that "at" without a noun
that follows implies, per Swift API naming guidelines, "at index." If you
want to specify, as we do here, what the index _is_, then it's written out
in full.

Yes, it seems overly cumbersome, but I was following existing conventions
which were intentionally very explicit.

it’s visually cumbersome because the “atByteOffset” overwhelms the other
two arguments but I guess i could live with it.

We don't have a consensus, but I think the suggestion to distinguish
between single value vs. multiple semantics was good. Otherwise,
adding the default count could be very misleading. Normally, we try to
minimize surface area, but adding two methods for the single-value case
avoids ambiguity between the buffer and pointer semantics:

UMP (pointer)
--- func initialize(to:count:(=1))
+++ func initialize(to:)
+++ func initialize(repeating:count:) // no default count
+++ func assign(to:)
+++ func assign(repeating:count:) // no default count

UMRP (raw pointer):
--- func initializeMemory<T>(as:at:(=0)count:(1)to:)
+++ func initializeMemory<T>(as:repeating:count:) // remove default
count

still extremely suspicious of this but i’m willing to compromise. also
there should be an `initializeMemory<T>(at:to:)` to match the typed
methods

Do you mean initializeMemory<T>(as:to:)?

I don’t think it’s necessary since raw buffers are not normally used to
hold a single typed element. We don’t need symmetry between raw and typed
pointers and I wouldn’t want to add an API that will never be used. So, I
would be ok with it only if there’s some use case that I’m overlooking.

The only case i can think of is initializing a buffer header but then again
i rarely use raw pointers so i’m not sure how much this feature would be
used. However, this function is already
<https://developer.apple.com/documentation/swift/unsafemutablerawpointer/2427589-initializememory&gt;
in the API (with a default count of 1 and offset of 0) so there’s that.

···

On Tue, Sep 5, 2017 at 5:36 PM, Andrew Trick <atrick@apple.com> wrote:

-Andy

On Tue, Sep 5, 2017 at 11:31 AM, Andrew Trick <atrick@apple.com> wrote:

I think we’ve agreed to a few minor updates to this proposal. Since
there hasn’t been any other feedback on the thread it may be worth posting
an amended proposal so we all know what we’ve agreed on.

-Andy

On Sep 3, 2017, at 8:23 PM, Andrew Trick via swift-evolution < >>>> swift-evolution@swift.org> wrote:

On Sep 3, 2017, at 8:05 PM, Xiaodi Wu <xiaodi.wu@gmail.com> wrote:

If we use byte offset, then the at parameter in UnsafeMutableRawPointer

should be removed, since pointer arithmetic can be used instead
(just like with UnsafeMutablePointer).

I agree that it seems quite sensible to remove the ‘at’ parameter
altogether from the UMRP method.

No code in tree or on github is using the `at` argument. I think it can
be removed. A fixit should still be possible.

Not convinced moving the at: argument to come before the as: argument

is worth it in terms of source breakage.

Since much of this proposal involves shuffling and relabeling
arguments, I’d argue it’s better to break slight more source in one go for
the optimal API than to break slightly less for a slightly less optimal
API, no? (This is assuming there is agreement that ‘at:as:’ is less prone
to misinterpretation than ‘as:at:’.)

To be clear, we’re just talking about UnsafeMutableRawBufferPointer.initializeMemory
now, so this is purely additive.
I think the label needs to be `atByteOffset`, and placing it before
`as` makes a lot of sense because it no longer depends on the type’s
stride.

-Andy
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Andrew_Trick]

We do want the Swift memory model to be consistent with the reality that on most platforms, we need the runtime to track block size.

I don't know where this comes from. If you don't need to be malloc-compatible, you don't strictly "need" this information. (It would break tools like 'leaks', though.)

There are two distinct but related issues (1) malloc compatibility (2) malloc/free like functionality. I know developers sometimes expect or want #2. Realistically, we will always want the runtime to provide malloc_size, even if it’s not super fast, so we’re not giving up anything long term by providing #2. The fact the #1 is also a likely goal on major platforms just reinforces that position.

I don't understand why "realistically, we will always want the runtime to provide malloc_size". Could you explain why?

The standard library already makes good use of malloc_size. More to the point, I think “realistically" it needs to be possible to write tools for memory analysis and debugging. I meant “always” in the temporal sense. I don’t think all memory needs this, only memory that is directly under the user’s control.

But then given that, I don't understand why the 'capacity' parameter is necessary. Under what circumstances would it actually be faster than "just" calling malloc_size?

The runtime may need to hash the address or traverse a lookup table to find out which allocation pool the block resides in. Now, that’s only if some platform ditches full malloc compatibility for user allocations, so I’m not sure how realistic it is.

-Andy

···

On Sep 7, 2017, at 5:17 PM, Jordan Rose <jordan_rose@apple.com> wrote:

On Sep 7, 2017, at 17:09, Andrew Trick <atrick@apple.com <mailto:atrick@apple.com>> wrote:

On Sep 7, 2017, at 2:29 PM, Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:

[Andrew_Trick]

Presumably, in your use case, you’re working directly with buffers on both sides (please point us to the real code). With the slicing approach, that would be done as follows:

bufA[i ..< i + bufB.count].initialize(from: bufB)

That will enforce full initialization of the slice: precondition(self.count == source.count).

Your point stands that it is redundant. That point will need to be weighed against other points, which I think should be discussed in another thread.

-Andy

···

On Sep 29, 2017, at 4:03 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

On Sep 29, 2017, at 5:56 PM, Dave Abrahams <dabrahams@apple.com <mailto:dabrahams@apple.com>> wrote:

On Sep 29, 2017, at 3:48 PM, Taylor Swift <kelvin13ma@gmail.com <mailto:kelvin13ma@gmail.com>> wrote:

On Fri, Sep 29, 2017 at 4:13 PM, Andrew Trick <atrick@apple.com <mailto:atrick@apple.com>> wrote:

On Sep 29, 2017, at 1:23 PM, Taylor Swift <kelvin13ma@gmail.com <mailto:kelvin13ma@gmail.com>> wrote:

Instead of

  buf.intialize(at: i, from: source)

We want to force a more obvious idiom:

  buf[i..<n].intialize(from: source)

The problem with subscript notation is we currently get the n argument from the source argument. So what would really have to be written is

buf[i ..< i + source.count].initialize(from: source)

which is a lot more ugly and redundant. One option could be to decouple the count parameter from the length of the source buffer, but that opens up the whole can of worms in which length do we use? What happens if n - i is less than or longer than source.count? If we enforce the precondition that source.count == n - i, then this syntax seems horribly redundant.

Sorry, a better analogy would have been:

buf[i...].intialize(from: source)

Whether you specify the slice’s end point depends on whether you want to completely initialize that slice or whether you’re just filling up as much of the buffer as you can. It also depends on whether `source` is also a buffer (of known size) or some arbitrary Sequence.

Otherwise, point taken.

-Andy

After thinking about this more, one-sided ranges might provide just the expressivity we need. What if:

buf[offset...].initialize(from: source) // initializes source.count elements from source starting from offset

buf[offset ..< endIndex].initialize(from: source) // initializes up to source.count elements from source starting from offset

The one sided one does not give a full initialization guarantee. The two sided one guarantees the entire segment is initialized.

In every other context, x[i...] is equivalent to x[i..<x.endIndex]

I don't think breaking that precedent is a good idea.

For move operations, the one sided one will fully deinitialize the source buffer while the two sided one will only deinitialize endIndex - offset elements.

—
-Dave

well since people want to use subscript notation so much we need some way of expressing case 1. writing both bounds in the subscript seems to imply a full initialization (and thus partial movement) guarantee.

[dabrahams (Dave Abrahams)]

Yes, I understood your reasoning. Do you understand why I still don't want to proceed in that direction?

···

On Sep 29, 2017, at 4:03 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

On Sep 29, 2017, at 5:56 PM, Dave Abrahams <dabrahams@apple.com <mailto:dabrahams@apple.com>> wrote:

On Sep 29, 2017, at 3:48 PM, Taylor Swift <kelvin13ma@gmail.com <mailto:kelvin13ma@gmail.com>> wrote:

On Fri, Sep 29, 2017 at 4:13 PM, Andrew Trick <atrick@apple.com <mailto:atrick@apple.com>> wrote:

On Sep 29, 2017, at 1:23 PM, Taylor Swift <kelvin13ma@gmail.com <mailto:kelvin13ma@gmail.com>> wrote:

Instead of

  buf.intialize(at: i, from: source)

We want to force a more obvious idiom:

  buf[i..<n].intialize(from: source)

The problem with subscript notation is we currently get the n argument from the source argument. So what would really have to be written is

buf[i ..< i + source.count].initialize(from: source)

which is a lot more ugly and redundant. One option could be to decouple the count parameter from the length of the source buffer, but that opens up the whole can of worms in which length do we use? What happens if n - i is less than or longer than source.count? If we enforce the precondition that source.count == n - i, then this syntax seems horribly redundant.

Sorry, a better analogy would have been:

buf[i...].intialize(from: source)

Whether you specify the slice’s end point depends on whether you want to completely initialize that slice or whether you’re just filling up as much of the buffer as you can. It also depends on whether `source` is also a buffer (of known size) or some arbitrary Sequence.

Otherwise, point taken.

-Andy

After thinking about this more, one-sided ranges might provide just the expressivity we need. What if:

buf[offset...].initialize(from: source) // initializes source.count elements from source starting from offset

buf[offset ..< endIndex].initialize(from: source) // initializes up to source.count elements from source starting from offset

The one sided one does not give a full initialization guarantee. The two sided one guarantees the entire segment is initialized.

In every other context, x[i...] is equivalent to x[i..<x.endIndex]

I don't think breaking that precedent is a good idea.

For move operations, the one sided one will fully deinitialize the source buffer while the two sided one will only deinitialize endIndex - offset elements.

—
-Dave

well since people want to use subscript notation so much we need some way of expressing case 1. writing both bounds in the subscript seems to imply a full initialization (and thus partial movement) guarantee.

—
-Dave

[Joe_Groff]

It seems to me that you could still provide malloc/free compatibility with a zone that had to do a relatively expensive traversal on free() to recover the pool the memory came from; malloc/free just wouldn't be the ideal interface in that situation.

-Joe

···

On Sep 7, 2017, at 5:34 PM, Andrew Trick <atrick@apple.com> wrote:

On Sep 7, 2017, at 5:17 PM, Jordan Rose <jordan_rose@apple.com> wrote:

On Sep 7, 2017, at 17:09, Andrew Trick <atrick@apple.com> wrote:

On Sep 7, 2017, at 2:29 PM, Jordan Rose <jordan_rose@apple.com> wrote:

We do want the Swift memory model to be consistent with the reality that on most platforms, we need the runtime to track block size.

I don't know where this comes from. If you don't need to be malloc-compatible, you don't strictly "need" this information. (It would break tools like 'leaks', though.)

There are two distinct but related issues (1) malloc compatibility (2) malloc/free like functionality. I know developers sometimes expect or want #2. Realistically, we will always want the runtime to provide malloc_size, even if it’s not super fast, so we’re not giving up anything long term by providing #2. The fact the #1 is also a likely goal on major platforms just reinforces that position.

I don't understand why "realistically, we will always want the runtime to provide malloc_size". Could you explain why?

The standard library already makes good use of malloc_size. More to the point, I think “realistically" it needs to be possible to write tools for memory analysis and debugging. I meant “always” in the temporal sense. I don’t think all memory needs this, only memory that is directly under the user’s control.

But then given that, I don't understand why the 'capacity' parameter is necessary. Under what circumstances would it actually be faster than "just" calling malloc_size?

The runtime may need to hash the address or traverse a lookup table to find out which allocation pool the block resides in. Now, that’s only if some platform ditches full malloc compatibility for user allocations, so I’m not sure how realistic it is.

[taylorswift]

yeah, which is why I think the at:from: system is better than any subscript
alternative. I know everyone wants to use the square brackets but it just
doesn’t work very well for exactly the reasons you mentioned.

···

On Sat, Sep 30, 2017 at 6:07 PM, Dave Abrahams <dabrahams@apple.com> wrote:

On Sep 29, 2017, at 4:03 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

On Sep 29, 2017, at 5:56 PM, Dave Abrahams <dabrahams@apple.com> wrote:

On Sep 29, 2017, at 3:48 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

On Fri, Sep 29, 2017 at 4:13 PM, Andrew Trick <atrick@apple.com> wrote:

On Sep 29, 2017, at 1:23 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

Instead of

  buf.intialize(at: i, from: source)

We want to force a more obvious idiom:

  buf[i..<n].intialize(from: source)

The problem with subscript notation is we currently get the n argument
from the source argument. So what would really have to be written is

buf[i ..< i + source.count].initialize(from: source)

which is a lot more ugly and redundant. One option could be to decouple
the count parameter from the length of the source buffer, but that opens up
the whole can of worms in which length do we use? What happens if n - i is
less than or longer than source.count? If we enforce the precondition
that source.count == n - i, then this syntax seems horribly redundant.

Sorry, a better analogy would have been:

buf[i...].intialize(from: source)

Whether you specify the slice’s end point depends on whether you want to
completely initialize that slice or whether you’re just filling up as much
of the buffer as you can. It also depends on whether `source` is also a
buffer (of known size) or some arbitrary Sequence.

Otherwise, point taken.

-Andy

After thinking about this more, one-sided ranges might provide just the
expressivity we need. What if:

buf[offset...].initialize(from: source) // initializes source.count
elements from source starting from offset

buf[offset ..< endIndex].initialize(from: source) // initializes up to
source.count elements from source starting from offset

The one sided one does not give a full initialization guarantee. The two
sided one guarantees the entire segment is initialized.

In every other context, x[i...] is equivalent to x[i..<x.endIndex]

I don't think breaking that precedent is a good idea.

For move operations, the one sided one will fully deinitialize the source
buffer while the two sided one will only deinitialize endIndex - offset
elements.

—
-Dave

well since people want to use subscript notation so much we need some way
of expressing case 1. writing both bounds in the subscript seems to imply a
full initialization (and thus partial movement) guarantee.

Yes, I understood your reasoning. Do you understand why I still don't
want to proceed in that direction?

—
-Dave

[Andrew_Trick]

Joe is right, and I just learned how amazing malloc zones are.
-Andy

···

On Sep 7, 2017, at 5:40 PM, Joe Groff <jgroff@apple.com> wrote:

But then given that, I don't understand why the 'capacity' parameter is necessary. Under what circumstances would it actually be faster than "just" calling malloc_size?

The runtime may need to hash the address or traverse a lookup table to find out which allocation pool the block resides in. Now, that’s only if some platform ditches full malloc compatibility for user allocations, so I’m not sure how realistic it is.

It seems to me that you could still provide malloc/free compatibility with a zone that had to do a relatively expensive traversal on free() to recover the pool the memory came from; malloc/free just wouldn't be the ideal interface in that situation.

-Joe

[dabrahams (Dave Abrahams)]

I don't think I made any arguments that could be viewed as pointing out a problem with the slice approach, unless you take as given the idea that the slice approach should mean something novel and unprecedented. I don't see the whole/part implication that you see in the two notations, even though I understand why you want to read it that way, in particular because of the precedent I cited.

···

On Sep 30, 2017, at 4:23 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

yeah, which is why I think the at:from: system is better than any subscript alternative. I know everyone wants to use the square brackets but it just doesn’t work very well for exactly the reasons you mentioned.

On Sat, Sep 30, 2017 at 6:07 PM, Dave Abrahams <dabrahams@apple.com <mailto:dabrahams@apple.com>> wrote:

On Sep 29, 2017, at 4:03 PM, Taylor Swift <kelvin13ma@gmail.com <mailto:kelvin13ma@gmail.com>> wrote:

On Sep 29, 2017, at 5:56 PM, Dave Abrahams <dabrahams@apple.com <mailto:dabrahams@apple.com>> wrote:

On Sep 29, 2017, at 3:48 PM, Taylor Swift <kelvin13ma@gmail.com <mailto:kelvin13ma@gmail.com>> wrote:

On Fri, Sep 29, 2017 at 4:13 PM, Andrew Trick <atrick@apple.com <mailto:atrick@apple.com>> wrote:

On Sep 29, 2017, at 1:23 PM, Taylor Swift <kelvin13ma@gmail.com <mailto:kelvin13ma@gmail.com>> wrote:

Instead of

  buf.intialize(at: i, from: source)

We want to force a more obvious idiom:

  buf[i..<n].intialize(from: source)

The problem with subscript notation is we currently get the n argument from the source argument. So what would really have to be written is

buf[i ..< i + source.count].initialize(from: source)

which is a lot more ugly and redundant. One option could be to decouple the count parameter from the length of the source buffer, but that opens up the whole can of worms in which length do we use? What happens if n - i is less than or longer than source.count? If we enforce the precondition that source.count == n - i, then this syntax seems horribly redundant.

Sorry, a better analogy would have been:

buf[i...].intialize(from: source)

Whether you specify the slice’s end point depends on whether you want to completely initialize that slice or whether you’re just filling up as much of the buffer as you can. It also depends on whether `source` is also a buffer (of known size) or some arbitrary Sequence.

Otherwise, point taken.

-Andy

After thinking about this more, one-sided ranges might provide just the expressivity we need. What if:

buf[offset...].initialize(from: source) // initializes source.count elements from source starting from offset

buf[offset ..< endIndex].initialize(from: source) // initializes up to source.count elements from source starting from offset

The one sided one does not give a full initialization guarantee. The two sided one guarantees the entire segment is initialized.

In every other context, x[i...] is equivalent to x[i..<x.endIndex]

I don't think breaking that precedent is a good idea.

For move operations, the one sided one will fully deinitialize the source buffer while the two sided one will only deinitialize endIndex - offset elements.

—
-Dave

well since people want to use subscript notation so much we need some way of expressing case 1. writing both bounds in the subscript seems to imply a full initialization (and thus partial movement) guarantee.

Yes, I understood your reasoning. Do you understand why I still don't want to proceed in that direction?

—
-Dave

—
-Dave

[jrose (Jordan Rose)]

As long as you support multiple allocators (or hide everything behind malloc/free), there's already a cost of malloc_zone_from_ptr or equivalent. Without seeing a concrete use case, I wouldn't want to stay with the harder-to-use API in UnsafePointer itself. It might be a feature of a particular allocator that you need to keep the capacity around, but it isn't something generally true about Swift's memory model, and probably never will be.

(Interesting reference points: malloc/malloc.h and the implementation of malloc on macOS <https://opensource.apple.com/source/libmalloc/libmalloc-116.50.8/src/malloc.c.auto.html&gt; - search for "free(void *ptr)".)

Jordan

···

On Sep 7, 2017, at 17:46, Andrew Trick <atrick@apple.com> wrote:

On Sep 7, 2017, at 5:40 PM, Joe Groff <jgroff@apple.com> wrote:

But then given that, I don't understand why the 'capacity' parameter is necessary. Under what circumstances would it actually be faster than "just" calling malloc_size?

The runtime may need to hash the address or traverse a lookup table to find out which allocation pool the block resides in. Now, that’s only if some platform ditches full malloc compatibility for user allocations, so I’m not sure how realistic it is.

It seems to me that you could still provide malloc/free compatibility with a zone that had to do a relatively expensive traversal on free() to recover the pool the memory came from; malloc/free just wouldn't be the ideal interface in that situation.

-Joe

Joe is right, and I just learned how amazing malloc zones are.

[taylorswift]

okay, well until someone can come up with a subscript syntax that does what
we need it to do, I’m inclined to view at:from: in the function parameter
list as the clearest and most straightforward syntax. We shouldn’t use the
square brackets for the sake of using square brackets.

···

On Sat, Sep 30, 2017 at 6:51 PM, Dave Abrahams <dabrahams@apple.com> wrote:

I don't think I made any arguments that could be viewed as pointing out a
problem with the slice approach, unless you take as given the idea that the
slice approach should mean something novel and unprecedented. I don't see
the whole/part implication that you see in the two notations, even though I
understand why you want to read it that way, in particular *because* of
the precedent I cited.

On Sep 30, 2017, at 4:23 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

yeah, which is why I think the at:from: system is better than any
subscript alternative. I know everyone wants to use the square brackets but
it just doesn’t work very well for exactly the reasons you mentioned.

On Sat, Sep 30, 2017 at 6:07 PM, Dave Abrahams <dabrahams@apple.com> > wrote:

On Sep 29, 2017, at 4:03 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

On Sep 29, 2017, at 5:56 PM, Dave Abrahams <dabrahams@apple.com> wrote:

On Sep 29, 2017, at 3:48 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

On Fri, Sep 29, 2017 at 4:13 PM, Andrew Trick <atrick@apple.com> wrote:

On Sep 29, 2017, at 1:23 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

Instead of

  buf.intialize(at: i, from: source)

We want to force a more obvious idiom:

  buf[i..<n].intialize(from: source)

The problem with subscript notation is we currently get the n argument
from the source argument. So what would really have to be written is

buf[i ..< i + source.count].initialize(from: source)

which is a lot more ugly and redundant. One option could be to decouple
the count parameter from the length of the source buffer, but that opens up
the whole can of worms in which length do we use? What happens if n - i is
less than or longer than source.count? If we enforce the precondition
that source.count == n - i, then this syntax seems horribly redundant.

Sorry, a better analogy would have been:

buf[i...].intialize(from: source)

Whether you specify the slice’s end point depends on whether you want to
completely initialize that slice or whether you’re just filling up as much
of the buffer as you can. It also depends on whether `source` is also a
buffer (of known size) or some arbitrary Sequence.

Otherwise, point taken.

-Andy

After thinking about this more, one-sided ranges might provide just the
expressivity we need. What if:

buf[offset...].initialize(from: source) // initializes source.count
elements from source starting from offset

buf[offset ..< endIndex].initialize(from: source) // initializes up to
source.count elements from source starting from offset

The one sided one does not give a full initialization guarantee. The two
sided one guarantees the entire segment is initialized.

In every other context, x[i...] is equivalent to x[i..<x.endIndex]

I don't think breaking that precedent is a good idea.

For move operations, the one sided one will fully deinitialize the source
buffer while the two sided one will only deinitialize endIndex - offset
elements.

—
-Dave

well since people want to use subscript notation so much we need some way
of expressing case 1. writing both bounds in the subscript seems to imply a
full initialization (and thus partial movement) guarantee.

Yes, I understood your reasoning. Do you understand why I still don't
want to proceed in that direction?

—
-Dave

—
-Dave

[Andrew_Trick]

I’m primarily arguing from the point of view that UnsafeBufferPointer should pass it’s deallocation capacity and should be implementable in terms of UnsafePointer. But I’m fine hiding the capacity argument from the public API for now. We know what the proposal author wants to do, so unless Joe still feels strongly, we could accept the proposal as-is, put the API decision to rest and focus on better documentation and and assertions.

-Andy

···

On Sep 7, 2017, at 5:56 PM, Jordan Rose <jordan_rose@apple.com> wrote:

On Sep 7, 2017, at 17:46, Andrew Trick <atrick@apple.com <mailto:atrick@apple.com>> wrote:

On Sep 7, 2017, at 5:40 PM, Joe Groff <jgroff@apple.com <mailto:jgroff@apple.com>> wrote:

But then given that, I don't understand why the 'capacity' parameter is necessary. Under what circumstances would it actually be faster than "just" calling malloc_size?

The runtime may need to hash the address or traverse a lookup table to find out which allocation pool the block resides in. Now, that’s only if some platform ditches full malloc compatibility for user allocations, so I’m not sure how realistic it is.

It seems to me that you could still provide malloc/free compatibility with a zone that had to do a relatively expensive traversal on free() to recover the pool the memory came from; malloc/free just wouldn't be the ideal interface in that situation.

-Joe

Joe is right, and I just learned how amazing malloc zones are.

As long as you support multiple allocators (or hide everything behind malloc/free), there's already a cost of malloc_zone_from_ptr or equivalent. Without seeing a concrete use case, I wouldn't want to stay with the harder-to-use API in UnsafePointer itself. It might be a feature of a particular allocator that you need to keep the capacity around, but it isn't something generally true about Swift's memory model, and probably never will be.

(Interesting reference points: malloc/malloc.h and the implementation of malloc on macOS <https://opensource.apple.com/source/libmalloc/libmalloc-116.50.8/src/malloc.c.auto.html&gt; - search for "free(void *ptr)".)

Jordan

[dabrahams (Dave Abrahams)]

I’m sorry, but I think we disagree on the premise. In my opinion the square brackets do exactly what we need them to do, and it’s not for the sake of anything other than consistency with the rest of the library and incidentally, reducing API complexity that I want to use them.

···

Sent from my iPad

On Sep 30, 2017, at 6:04 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

okay, well until someone can come up with a subscript syntax that does what we need it to do, I’m inclined to view at:from: in the function parameter list as the clearest and most straightforward syntax. We shouldn’t use the square brackets for the sake of using square brackets.

On Sat, Sep 30, 2017 at 6:51 PM, Dave Abrahams <dabrahams@apple.com> wrote:
I don't think I made any arguments that could be viewed as pointing out a problem with the slice approach, unless you take as given the idea that the slice approach should mean something novel and unprecedented. I don't see the whole/part implication that you see in the two notations, even though I understand why you want to read it that way, in particular because of the precedent I cited.

On Sep 30, 2017, at 4:23 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

yeah, which is why I think the at:from: system is better than any subscript alternative. I know everyone wants to use the square brackets but it just doesn’t work very well for exactly the reasons you mentioned.

On Sat, Sep 30, 2017 at 6:07 PM, Dave Abrahams <dabrahams@apple.com> wrote:

On Sep 29, 2017, at 4:03 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

On Sep 29, 2017, at 5:56 PM, Dave Abrahams <dabrahams@apple.com> wrote:

On Sep 29, 2017, at 3:48 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

On Fri, Sep 29, 2017 at 4:13 PM, Andrew Trick <atrick@apple.com> wrote:

On Sep 29, 2017, at 1:23 PM, Taylor Swift <kelvin13ma@gmail.com> wrote:

Instead of

  buf.intialize(at: i, from: source)

We want to force a more obvious idiom:

  buf[i..<n].intialize(from: source)

The problem with subscript notation is we currently get the n argument from the source argument. So what would really have to be written is

buf[i ..< i + source.count].initialize(from: source)

which is a lot more ugly and redundant. One option could be to decouple the count parameter from the length of the source buffer, but that opens up the whole can of worms in which length do we use? What happens if n - i is less than or longer than source.count? If we enforce the precondition that source.count == n - i, then this syntax seems horribly redundant.

Sorry, a better analogy would have been:

buf[i...].intialize(from: source)

Whether you specify the slice’s end point depends on whether you want to completely initialize that slice or whether you’re just filling up as much of the buffer as you can. It also depends on whether `source` is also a buffer (of known size) or some arbitrary Sequence.

Otherwise, point taken.

-Andy

After thinking about this more, one-sided ranges might provide just the expressivity we need. What if:

buf[offset...].initialize(from: source) // initializes source.count elements from source starting from offset

buf[offset ..< endIndex].initialize(from: source) // initializes up to source.count elements from source starting from offset

The one sided one does not give a full initialization guarantee. The two sided one guarantees the entire segment is initialized.

In every other context, x[i...] is equivalent to x[i..<x.endIndex]

I don't think breaking that precedent is a good idea.

For move operations, the one sided one will fully deinitialize the source buffer while the two sided one will only deinitialize endIndex - offset elements.

—
-Dave

well since people want to use subscript notation so much we need some way of expressing case 1. writing both bounds in the subscript seems to imply a full initialization (and thus partial movement) guarantee.

Yes, I understood your reasoning. Do you understand why I still don't want to proceed in that direction?

—
-Dave

—
-Dave

[taylorswift]

let me summarize why i think deallocate should lose the capacity parameter
for now:

   - up to now, the capacity argument has been poorly, even misleadingly
   documented. this means it is unlikely that existing users of the API are
   using it correctly.
   - deallocate(capacity:) allows for optimizations that could improve the
   performance of the Swift heap. however, these optimizations are not in
   place yet, nor will they be any time soon, so dropping the parameter would
   result in no performance regressions.
   - users coming from C/C++ are used to the malloc(_:) – free() pattern.
   Swift should support this both for learnability, and for
malloc-compatibility.

   - completely departing from the malloc memory model might save a few
   bytes of headers in the heap, but as discussed in this thread, there are
   other upsides of Swift being malloc-compatible.
   - there is nothing preventing us from reintroducing
   deallocate(allocatedCapacity:) in the future, once the runtime actually
   supports this feature. informed users looking for a performance boost could
   opt-in to use this API instead.
   - deprecating, and then un-deprecating deallocate(capacity:) might sound
   extra, but keeping this method unbothered is actually a bad idea. whoever
   is using it right now, is probably not using it correctly. The best
   strategy is to reintroduce it later as a well-documented *opt-in*
   feature. Grandfathering in old incorrect code offers no benefits and many
   problems.
   - lying to users is never a good idea. for the forseeable future,
   Heap.cpp calls free(), and Swift shouldn’t pretend like it supports
   something it doesn’t support.

···

On Thu, Sep 7, 2017 at 8:18 PM, Andrew Trick <atrick@apple.com> wrote:

On Sep 7, 2017, at 5:56 PM, Jordan Rose <jordan_rose@apple.com> wrote:

On Sep 7, 2017, at 17:46, Andrew Trick <atrick@apple.com> wrote:

On Sep 7, 2017, at 5:40 PM, Joe Groff <jgroff@apple.com> wrote:

But then given that, I don't understand why the 'capacity' parameter is
necessary. Under what circumstances would it actually be faster than "just"
calling malloc_size?

The runtime may need to hash the address or traverse a lookup table to
find out which allocation pool the block resides in. Now, that’s only if
some platform ditches full malloc compatibility for user allocations, so
I’m not sure how realistic it is.

It seems to me that you could still provide malloc/free compatibility with
a zone that had to do a relatively expensive traversal on free() to recover
the pool the memory came from; malloc/free just wouldn't be the ideal
interface in that situation.

-Joe

Joe is right, and I just learned how amazing malloc zones are.

As long as you support multiple allocators (or hide everything behind
malloc/free), there's already a cost of malloc_zone_from_ptr or equivalent.
Without seeing a concrete use case, I wouldn't want to stay with the
harder-to-use API *in UnsafePointer itself.* It might be a feature of a
*particular* allocator that you need to keep the capacity around, but it
*isn't* something generally true about Swift's memory model, and probably
never will be.

(Interesting reference points: malloc/malloc.h and the implementation of
malloc on macOS
<https://opensource.apple.com/source/libmalloc/libmalloc-116.50.8/src/malloc.c.auto.html&gt; -
search for "free(void *ptr)".)

Jordan

I’m primarily arguing from the point of view that UnsafeBufferPointer
should pass it’s deallocation capacity and should be implementable in terms
of UnsafePointer. But I’m fine hiding the capacity argument from the public
API for now. We know what the proposal author wants to do, so unless Joe
still feels strongly, we could accept the proposal as-is, put the API
decision to rest and focus on better documentation and and assertions.

-Andy