SE-0138 UnsafeBytes

Does the `enumerateBytes` method (of Foundation.Data and DispatchData) also need an UnsafeRawBufferPointer version?

-- Ben

···

On 11 Sep 2016, at 01:53, Andrew Trick wrote:

https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsaferawbufferpointer.md

The review period has been extended until September 14. The UnsafeRawBufferPointer type name is settled, but we still need to come up with an answer for the name of the new closure taking functions:

withXyz() should normally reveal the closure argument type as Xyz. That's why I originally proposed UnsafeBytes as the type name. Now that we've decided to use the descriptive type instead we have a problem...

https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsaferawbufferpointer.md

The review period has been extended until September 14. The UnsafeRawBufferPointer type name is settled, but we still need to come up with an answer for the name of the new closure taking functions:

withXyz() should normally reveal the closure argument type as Xyz. That's why I originally proposed UnsafeBytes as the type name. Now that we've decided to use the descriptive type instead we have a problem…

I was totally wrong about this policy. In closure-taking “withXyz" functions, “Xyz" should reveal the role of the closure argument, not its type. We do not need to repeat type information.

We have strong agreement to leave the proposed `withUnsafeBytes {…}` name as it stands.

Note that `withRawBytes` was a strong contender, but at this time it's more important to consistently follow the convention for using `Unsafe` in the closure name whenever the closure argument is unsafe (e.g. you can't return it from the closure). We may want to revisit this logic later (in some sense Unsafe is redundant), but when we do that, we also need to reevaluate all of our withUnsafe APIs. Furthermore, we would want to change Foundation Data's API to be consistent. These are bigger debates that can be deferred.

-Andy

···

On Sep 10, 2016, at 5:53 PM, Andrew Trick <atrick@apple.com> wrote:

In this code, it's obvious that a sequence of bytes is being appended to an array.

var buffer = [UInt8]()
withUnsafeBytes(of: &header) {
buffer += $0
}

In the following version, the closure argument type is obvious, which is nice, but otherwise it's borderline unreadable, and doesn't describe what's actually happenning. How can we tell that a sequence of bytes will be appended?

var buffer = [UInt8]()
withUnsafeRawBufferPointer(to: &header) {
buffer += $0
}

The mutable version really stretches the limits of descriptively naming things, and still doesn't say anything about a byte sequence:

withUnsafeMutableRawBufferPointer(to: &header) {
readHeader(into: $0)
}

-Andy

On Sep 2, 2016, at 11:14 AM, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:

on Thu Sep 01 2016, Andrew Trick <swift-evolution@swift.org> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md&gt;

* What is your evaluation of the proposal?

I strongly support inclusion of the feature, but I have issues with the
name. It seems to me that in order to fit into the standard library, it
should be called Unsafe[Mutable]RawBufferPointer. Each part of the name
conveys something important, and for the same reasons we're using
Unsafe[Mutable]BufferPointer instead of UnsafeMutableElements, we should
stick to the scheme:

- “Unsafe,” because you can break memory safety with this tool

- “Raw,” because the fundamental model is that of “raw,” rather than
“typed,” memory.

- “Buffer,” because it works on a series of contiguous elements of known
length.

- “Pointer,” because it has reference semantics! When you pass one of
these things around by value, you're not passing the bytes; you're
passing a shared reference to the bytes.

* Is the problem being addressed significant enough to warrant a
change to Swift?

Yes, and it fills an important funcationality gap now that we have the
unsafe pointer model nailed down.

* Does this proposal fit well with the feel and direction of Swift?

Yes, except for the name.

* If you have used other languages or libraries with a similar
feature, how do you feel that this proposal compares to those?

I don't think any other language distinguishes raw from typed memory in
this way.

* How much effort did you put into your review? A glance, a quick
reading, or an in-depth study?

Enough ;-)

--
-Dave, posting as a reviewer, not a review manager

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

There is no mismatch between Data and UnsafeBytes. Data requires UnsafeBytes whenever the user is working with UnsafeRawPointers. UnsafeBytes is meant to enable interoperability with Data in these cases.

I included the Framework Interfaces section so that you could see what I think right long term solution is:
https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md#framework-interfaces

If you have any concrete concerns about that long-term solution, then that would be good raise.

-Andy

···

On Sep 2, 2016, at 1:34 PM, Tony Parker <anthony.parker@apple.com> wrote:

Arguments of being out of time do not compel me, personally. Introducing a new type is effectively permanent. We just got started with introducing Swift API. I don’t want to accidentally saddle ourselves with additional complexity forever without giving ourselves an adequate opportunity to fully consider what the right long-term solution is. The fact that you’ve already described the mismatch between struct Data and this type as “unfortunate” sounds, to me, like a big problem.

I support that sentiment. We should stick with existing solutions and not dilute them without reason.

Having said that, my personal experience with NSData is pretty bad. It never really fits my needs unless I am dealing with amorphous data that stay’s amorphous.
As soon as I need to figure out what is in the data, the code gets ugly fast and needs extensive commenting to stay maintainable (and we all know what happens with comments…)

I think Andy is right, and we need to differentiate between byte access and amorphous data blocks.

A minor point: many programmers new to the platform will know what to do with pointers, Data on the other hand is too much of an abstraction to grasp intuitively.

And a minor-minor point: Personally I hate the “Unsafe” part in the name. Seems a bit pedantic to me. I do not need subjective judgements in a language. (Is there really a SW-engineer out there that does not know about the inherent dangers of direct memory access?, does “unsafe” really add value to the language?)

Rien.

···

On 02 Sep 2016, at 22:34, Tony Parker via swift-evolution <swift-evolution@swift.org> wrote:

I would instead prefer to look at what we can do with adding API to the existing types to cover this use case. You point out in the proposal that it has become customary to use [UInt8] in API. I would prefer that we work towards a solution that makes it customary to use Data when you want to expose an API that uses Data.

I think instead handleMessages should take a Data argument.
The core issue here is that the problem solved here is not a "foundation problem", it is a "stdlib problem". They are superficially similar in that both of them involve an array of bytes but the comparison ends there.

The motivation for Data/NSData is we have a logical collection of bytes. That collection may be contiguous or discontiguous (speaking only of API; I'm unfamiliar with the implementation choice). It may be created from a file or even a URL, from a base64 representation, it may share the underlying memory with other NSData instance or not.

What we are considering here is a *physical* collection of bytes, e.g. a pointer and a length. By definition, they do not share memory with each other (unless they overlap, which is you can find out with public API). By definition, they are contiguous.

Data is the abstraction to choose when you don't care how the memory is laid out. UnsafeBytes is the abstraction to choose when the memory layout is the critical property. e.g., you are bitshifting between the IEEE754 fields to implement fastinvsqrt, or you are converting between sockaddr and sockaddr_in (same type but different sizes).

These are not I/O problems or array problems. They are C pointer problems, where we want to dispense with the traditional Swift abstractions and view the world as C arrays again like it's 1970.

Like all pointer problems in the language, they aren't foundation problems and we should not solve them there, whether we are under time pressure or with all the time in the world. They should be solved where we solve the other pointer problems, which is in the stdlib.

I think instead handleMessages should take a Data argument. The input driver code should be able to use API on Data (or elsewhere, API that returns Data) to populate it with the contents of the file.
The core API in my networking project is fairly similar to this example, and in that case, NSData was not the right choice, because it does not support

Uninitialized arrays
Explicitly managing zero-cost "views" of the underlying memory by creating instances that refer to the same location and sliding the start and end markers
Casting unsafely between arrays of different size
Working with memory regions where the size is not known at compile time but is discovered during a read such as pascal strings or msgpack
These are totally ridiculous additions to the Data API surface. Somebody who wants to load a URL should never see this garbage in their autocomplete. But they are things C programmers frequently do.

···

On September 2, 2016 at 3:34:56 PM, Tony Parker via swift-evolution (swift-evolution@swift.org) wrote:

I generally agree with what you said. I think the vague plan is later in Swift 4 to ship a bounds-checked variant of both UnsafeBufferPointer and UnsafeBytes (or UnsafeRawBufferPointer if you prefer).

I don’t want to eliminate the debug-mode checks though. I did try to make it clear in the comments that bounds-checking only applied to debug mode, so developers should not accidentally become too reliant on them.

So, the only question is whether the UnsafeBytes.copyBytes() API should have debug or release-mode checks. My decision to keep the stronger checks here was probabilistic—it seems unlikely to be a performance issue but likely to catch most buffer overruns. But I agree that it is inconsistent, especially if we plan to introduce a release bounds-checked variant later. We don’t want developers to begin relying on that check. I’m leaning toward dropping it down to a debug-mode check.

-Andy

···

On Sep 3, 2016, at 3:36 PM, Drew Crawford <drew@sealedabstract.com> wrote:

On September 2, 2016 at 2:36:43 AM, Andrew Trick (atrick@apple.com <mailto:atrick@apple.com>) wrote:

After thinking about this for a moment, I like the approach of extending UnsafeBytes with release-mode bounds checked versions of subscript, load, and storeBytes.

I agree with this, I think it's mostly a question of naming and defaults. My concern here is letting a swift developer accidentally write heartbleed, which we can't actually prevent, but we can make it harder.

IMO

1. There should be clear consistency in the checked-ness of the API surface. Agree that checked iterator makes no sense, but I think the most important thing is to avoid creating a job interview trivia game where `set` is checked but `store` is unchecked, spot the bug in this function.

2. For consistency with UnsafeBufferPointer it may make the most sense to just ship unchecked or ship an opt-in checked wrapper. I believe however that the existing precedent is all wrong on this point, and I'd like to see us revisit this question across both interfaces in Swift 4, but I don't want to lay out a whole case here that should be its own thread.

Coincidentally, I just wrote my first Swift code to use UnsafePointer<>. I was wrapping the LZMA API to decompress LZMA data. It's a C API that works by pointing to an input buffer and and output buffer, and then calling a function that decompresses what it can given those two buffers (and their lengths).

I treated them as UnsafePointer<UInt8>, but really they're raw, in the sense that they are not a collection of a single element, just a collection of bytes.

My wrapper's interface to LZMA uses Data instances. I don't see a way of getting from Data to UnsafeRawBufferPointer in Xcode 8 GM seed (which makes sense, given that this is still in progress). But I also didn't see a way to get to UnsafeRawPointer; should there be?

There should be and there isn't. It used to be Data.bytes, but it was just deprecated. In the current state of limbo, you just do this:

  return data.withUnsafeBytes { bytes: UnsafeBufferPointer<UInt8> in … }

and that binds Data’s memory to UInt8. It fine in practice as long as Data owns its memory (not using bytesNoCopy). Otherwise whoever else uses the memory should also view it as either raw or UInt8, or they should bind memory each time they access it.

Will something be added to Data when SE-0138 is finalized? I guess that's not for Swift 3 but 3.x?

Yes. It just takes a little more time to evolve the Data API.

-Andy

···

On Sep 10, 2016, at 6:23 PM, Rick Mann via swift-evolution <swift-evolution@swift.org> wrote:

Thanks, and sorry if I'm hijacking the thread a bit with this.

On Sep 10, 2016, at 17:53 , Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:

https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsaferawbufferpointer.md

The review period has been extended until September 14. The UnsafeRawBufferPointer type name is settled, but we still need to come up with an answer for the name of the new closure taking functions:

withXyz() should normally reveal the closure argument type as Xyz. That's why I originally proposed UnsafeBytes as the type name. Now that we've decided to use the descriptive type instead we have a problem...

In this code, it's obvious that a sequence of bytes is being appended to an array.

var buffer = [UInt8]()
withUnsafeBytes(of: &header) {
buffer += $0
}

In the following version, the closure argument type is obvious, which is nice, but otherwise it's borderline unreadable, and doesn't describe what's actually happenning. How can we tell that a sequence of bytes will be appended?

var buffer = [UInt8]()
withUnsafeRawBufferPointer(to: &header) {
buffer += $0
}

The mutable version really stretches the limits of descriptively naming things, and still doesn't say anything about a byte sequence:

withUnsafeMutableRawBufferPointer(to: &header) {
readHeader(into: $0)
}

-Andy

On Sep 2, 2016, at 11:14 AM, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:

on Thu Sep 01 2016, Andrew Trick <swift-evolution@swift.org> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md&gt;

* What is your evaluation of the proposal?

I strongly support inclusion of the feature, but I have issues with the
name. It seems to me that in order to fit into the standard library, it
should be called Unsafe[Mutable]RawBufferPointer. Each part of the name
conveys something important, and for the same reasons we're using
Unsafe[Mutable]BufferPointer instead of UnsafeMutableElements, we should
stick to the scheme:

- “Unsafe,” because you can break memory safety with this tool

- “Raw,” because the fundamental model is that of “raw,” rather than
“typed,” memory.

- “Buffer,” because it works on a series of contiguous elements of known
length.

- “Pointer,” because it has reference semantics! When you pass one of
these things around by value, you're not passing the bytes; you're
passing a shared reference to the bytes.

* Is the problem being addressed significant enough to warrant a
change to Swift?

Yes, and it fills an important funcationality gap now that we have the
unsafe pointer model nailed down.

* Does this proposal fit well with the feel and direction of Swift?

Yes, except for the name.

* If you have used other languages or libraries with a similar
feature, how do you feel that this proposal compares to those?

I don't think any other language distinguishes raw from typed memory in
this way.

* How much effort did you put into your review? A glance, a quick
reading, or an in-depth study?

Enough ;-)

--
-Dave, posting as a reviewer, not a review manager

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Rick Mann
rmann@latencyzero.com

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I think it should only have an UnsafeRawBufferPointer version. If the user wants to bind memory, they should do that explicitly. I’ve made the likely changes to Data on a branch:
https://github.com/atrick/swift/commit/19968405608fa326eb7ad5ffed5fcd9a78b0f0a5

There are enough changes to Data that I think it deserves a separate proposal and discussion thread. It’s useful to look ahead at how the Data API should look but I’m trying to get language-level changes accepted first (in some sense, Unsafe constructs are part of the language even if they don’t require compiler changes).

Also keep in mind, adding UnsafeRawBufferPointer does not make Data any less usable today. We just need to get core support in place so we can have a discussion about Foundation.

-Andy

···

On Sep 11, 2016, at 3:07 AM, Ben Rimmington <me@benrimmington.com> wrote:

On 11 Sep 2016, at 01:53, Andrew Trick wrote:

https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsaferawbufferpointer.md

The review period has been extended until September 14. The UnsafeRawBufferPointer type name is settled, but we still need to come up with an answer for the name of the new closure taking functions:

withXyz() should normally reveal the closure argument type as Xyz. That's why I originally proposed UnsafeBytes as the type name. Now that we've decided to use the descriptive type instead we have a problem...

Does the `enumerateBytes` method (of Foundation.Data and DispatchData) also need an UnsafeRawBufferPointer version?

An example without `Unsafe` in the function name:

<https://developer.apple.com/reference/swift/string/1538904-withcstring&gt;

-- Ben

···

On 14 Sep 2016, at 17:08, Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:

On Sep 10, 2016, at 5:53 PM, Andrew Trick <atrick@apple.com> wrote:

https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsaferawbufferpointer.md

The review period has been extended until September 14. The UnsafeRawBufferPointer type name is settled, but we still need to come up with an answer for the name of the new closure taking functions:

withXyz() should normally reveal the closure argument type as Xyz. That's why I originally proposed UnsafeBytes as the type name. Now that we've decided to use the descriptive type instead we have a problem…

I was totally wrong about this policy. In closure-taking “withXyz" functions, “Xyz" should reveal the role of the closure argument, not its type. We do not need to repeat type information.

We have strong agreement to leave the proposed `withUnsafeBytes {…}` name as it stands.

Note that `withRawBytes` was a strong contender, but at this time it's more important to consistently follow the convention for using `Unsafe` in the closure name whenever the closure argument is unsafe (e.g. you can't return it from the closure). We may want to revisit this logic later (in some sense Unsafe is redundant), but when we do that, we also need to reevaluate all of our withUnsafe APIs. Furthermore, we would want to change Foundation Data's API to be consistent. These are bigger debates that can be deferred.

Annotating that a parameter of a closure should not escape the closure is probably a worthwhile thing to have, anyway. Currently we only allow that for function-type parameters, but there are other parameters you might give (such as the pointer in `String.withCString()`’s closure) which should not escape their context.

···

On 14 Sep 2016, at 18:08, Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:

On Sep 10, 2016, at 5:53 PM, Andrew Trick <atrick@apple.com> wrote:

https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsaferawbufferpointer.md

The review period has been extended until September 14. The UnsafeRawBufferPointer type name is settled, but we still need to come up with an answer for the name of the new closure taking functions:

withXyz() should normally reveal the closure argument type as Xyz. That's why I originally proposed UnsafeBytes as the type name. Now that we've decided to use the descriptive type instead we have a problem…

I was totally wrong about this policy. In closure-taking “withXyz" functions, “Xyz" should reveal the role of the closure argument, not its type. We do not need to repeat type information.

We have strong agreement to leave the proposed `withUnsafeBytes {…}` name as it stands.

Note that `withRawBytes` was a strong contender, but at this time it's more important to consistently follow the convention for using `Unsafe` in the closure name whenever the closure argument is unsafe (e.g. you can't return it from the closure). We may want to revisit this logic later (in some sense Unsafe is redundant), but when we do that, we also need to reevaluate all of our withUnsafe APIs. Furthermore, we would want to change Foundation Data's API to be consistent. These are bigger debates that can be deferred.

Thanks, Andrew. In my case, the Data reads from a file, and since the raw access is wrapped around the LZMA decompression, I think it should be safe (no one else is accessing the data at that time).

I'll just wait for Foundation.Data to be updated and update my code then.

···

On Sep 10, 2016, at 19:33 , Andrew Trick <atrick@apple.com> wrote:

On Sep 10, 2016, at 6:23 PM, Rick Mann via swift-evolution <swift-evolution@swift.org> wrote:

Coincidentally, I just wrote my first Swift code to use UnsafePointer<>. I was wrapping the LZMA API to decompress LZMA data. It's a C API that works by pointing to an input buffer and and output buffer, and then calling a function that decompresses what it can given those two buffers (and their lengths).

I treated them as UnsafePointer<UInt8>, but really they're raw, in the sense that they are not a collection of a single element, just a collection of bytes.

My wrapper's interface to LZMA uses Data instances. I don't see a way of getting from Data to UnsafeRawBufferPointer in Xcode 8 GM seed (which makes sense, given that this is still in progress). But I also didn't see a way to get to UnsafeRawPointer; should there be?

There should be and there isn't. It used to be Data.bytes, but it was just deprecated. In the current state of limbo, you just do this:

return data.withUnsafeBytes { bytes: UnsafeBufferPointer<UInt8> in … }

and that binds Data’s memory to UInt8. It fine in practice as long as Data owns its memory (not using bytesNoCopy). Otherwise whoever else uses the memory should also view it as either raw or UInt8, or they should bind memory each time they access it.

Will something be added to Data when SE-0138 is finalized? I guess that's not for Swift 3 but 3.x?

Yes. It just takes a little more time to evolve the Data API.

-Andy

Thanks, and sorry if I'm hijacking the thread a bit with this.

On Sep 10, 2016, at 17:53 , Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:

https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsaferawbufferpointer.md

The review period has been extended until September 14. The UnsafeRawBufferPointer type name is settled, but we still need to come up with an answer for the name of the new closure taking functions:

withXyz() should normally reveal the closure argument type as Xyz. That's why I originally proposed UnsafeBytes as the type name. Now that we've decided to use the descriptive type instead we have a problem...

In this code, it's obvious that a sequence of bytes is being appended to an array.

var buffer = [UInt8]()
withUnsafeBytes(of: &header) {
buffer += $0
}

In the following version, the closure argument type is obvious, which is nice, but otherwise it's borderline unreadable, and doesn't describe what's actually happenning. How can we tell that a sequence of bytes will be appended?

var buffer = [UInt8]()
withUnsafeRawBufferPointer(to: &header) {
buffer += $0
}

The mutable version really stretches the limits of descriptively naming things, and still doesn't say anything about a byte sequence:

withUnsafeMutableRawBufferPointer(to: &header) {
readHeader(into: $0)
}

-Andy

On Sep 2, 2016, at 11:14 AM, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:

on Thu Sep 01 2016, Andrew Trick <swift-evolution@swift.org> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md&gt;

* What is your evaluation of the proposal?

I strongly support inclusion of the feature, but I have issues with the
name. It seems to me that in order to fit into the standard library, it
should be called Unsafe[Mutable]RawBufferPointer. Each part of the name
conveys something important, and for the same reasons we're using
Unsafe[Mutable]BufferPointer instead of UnsafeMutableElements, we should
stick to the scheme:

- “Unsafe,” because you can break memory safety with this tool

- “Raw,” because the fundamental model is that of “raw,” rather than
“typed,” memory.

- “Buffer,” because it works on a series of contiguous elements of known
length.

- “Pointer,” because it has reference semantics! When you pass one of
these things around by value, you're not passing the bytes; you're
passing a shared reference to the bytes.

* Is the problem being addressed significant enough to warrant a
change to Swift?

Yes, and it fills an important funcationality gap now that we have the
unsafe pointer model nailed down.

* Does this proposal fit well with the feel and direction of Swift?

Yes, except for the name.

* If you have used other languages or libraries with a similar
feature, how do you feel that this proposal compares to those?

I don't think any other language distinguishes raw from typed memory in
this way.

* How much effort did you put into your review? A glance, a quick
reading, or an in-depth study?

Enough ;-)

--
-Dave, posting as a reviewer, not a review manager

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Rick Mann
rmann@latencyzero.com

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Rick Mann
rmann@latencyzero.com

Annotating that a parameter of a closure should not escape the closure is probably a worthwhile thing to have, anyway. Currently we only allow that for function-type parameters, but there are other parameters you might give (such as the pointer in `String.withCString()`’s closure) which should not escape their context.

It certainly needs to be at least explicitly documented somewhere. I’ve seen people use it in published code to obtain a pointer to (fro example) the bytes in an array without realising that the storage could easily go away.

···

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Thanks, Andrew. In my case, the Data reads from a file, and since the raw access is wrapped around the LZMA decompression, I think it should be safe (no one else is accessing the data at that time).

I'll just wait for Foundation.Data to be updated and update my code then.

Sure, enumerateBytes is fine for you. To be clear, it would only be a problem if Data did not own the memory (bytesNoCopy:), allowing the same memory to be accessed as a non-UInt8 type.
-Andy

···

On Sep 11, 2016, at 2:29 PM, Rick Mann <rmann@latencyzero.com> wrote:

On Sep 10, 2016, at 19:33 , Andrew Trick <atrick@apple.com> wrote:

On Sep 10, 2016, at 6:23 PM, Rick Mann via swift-evolution <swift-evolution@swift.org> wrote:

Coincidentally, I just wrote my first Swift code to use UnsafePointer<>. I was wrapping the LZMA API to decompress LZMA data. It's a C API that works by pointing to an input buffer and and output buffer, and then calling a function that decompresses what it can given those two buffers (and their lengths).

I treated them as UnsafePointer<UInt8>, but really they're raw, in the sense that they are not a collection of a single element, just a collection of bytes.

My wrapper's interface to LZMA uses Data instances. I don't see a way of getting from Data to UnsafeRawBufferPointer in Xcode 8 GM seed (which makes sense, given that this is still in progress). But I also didn't see a way to get to UnsafeRawPointer; should there be?

There should be and there isn't. It used to be Data.bytes, but it was just deprecated. In the current state of limbo, you just do this:

return data.withUnsafeBytes { bytes: UnsafeBufferPointer<UInt8> in … }

and that binds Data’s memory to UInt8. It fine in practice as long as Data owns its memory (not using bytesNoCopy). Otherwise whoever else uses the memory should also view it as either raw or UInt8, or they should bind memory each time they access it.

Will something be added to Data when SE-0138 is finalized? I guess that's not for Swift 3 but 3.x?

Yes. It just takes a little more time to evolve the Data API.

-Andy

Thanks, and sorry if I'm hijacking the thread a bit with this.

On Sep 10, 2016, at 17:53 , Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:

https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsaferawbufferpointer.md

The review period has been extended until September 14. The UnsafeRawBufferPointer type name is settled, but we still need to come up with an answer for the name of the new closure taking functions:

withXyz() should normally reveal the closure argument type as Xyz. That's why I originally proposed UnsafeBytes as the type name. Now that we've decided to use the descriptive type instead we have a problem...

In this code, it's obvious that a sequence of bytes is being appended to an array.

var buffer = [UInt8]()
withUnsafeBytes(of: &header) {
buffer += $0
}

In the following version, the closure argument type is obvious, which is nice, but otherwise it's borderline unreadable, and doesn't describe what's actually happenning. How can we tell that a sequence of bytes will be appended?

var buffer = [UInt8]()
withUnsafeRawBufferPointer(to: &header) {
buffer += $0
}

The mutable version really stretches the limits of descriptively naming things, and still doesn't say anything about a byte sequence:

withUnsafeMutableRawBufferPointer(to: &header) {
readHeader(into: $0)
}

-Andy

On Sep 2, 2016, at 11:14 AM, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:

on Thu Sep 01 2016, Andrew Trick <swift-evolution@swift.org> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md&gt;

* What is your evaluation of the proposal?

I strongly support inclusion of the feature, but I have issues with the
name. It seems to me that in order to fit into the standard library, it
should be called Unsafe[Mutable]RawBufferPointer. Each part of the name
conveys something important, and for the same reasons we're using
Unsafe[Mutable]BufferPointer instead of UnsafeMutableElements, we should
stick to the scheme:

- “Unsafe,” because you can break memory safety with this tool

- “Raw,” because the fundamental model is that of “raw,” rather than
“typed,” memory.

- “Buffer,” because it works on a series of contiguous elements of known
length.

- “Pointer,” because it has reference semantics! When you pass one of
these things around by value, you're not passing the bytes; you're
passing a shared reference to the bytes.

* Is the problem being addressed significant enough to warrant a
change to Swift?

Yes, and it fills an important funcationality gap now that we have the
unsafe pointer model nailed down.

* Does this proposal fit well with the feel and direction of Swift?

Yes, except for the name.

* If you have used other languages or libraries with a similar
feature, how do you feel that this proposal compares to those?

I don't think any other language distinguishes raw from typed memory in
this way.

* How much effort did you put into your review? A glance, a quick
reading, or an in-depth study?

Enough ;-)

--
-Dave, posting as a reviewer, not a review manager

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Rick Mann
rmann@latencyzero.com

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Rick Mann
rmann@latencyzero.com