SE-0138 UnsafeBytes


(Andrew Trick) #1

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

Reviews are an important part of the Swift evolution process. All reviews
should be sent to the swift-evolution mailing list at

<https://lists.swift.org/mailman/listinfo/swift-evolution>

or, if you would like to keep your feedback private, directly to the
review manager. When replying, please try to keep the proposal link at
the top of the message:

Proposal link:
<https://lists.swift.org/mailman/listinfo/swift-evolution>

What goes into a review?

The goal of the review process is to improve the proposal under review
through constructive criticism and, eventually, determine the direction of
Swift. When writing your review, here are some questions you might want to
answer in your review:

* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a
   change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar
   feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick
   reading, or an in-depth study?

More information about the Swift evolution process is available at

<https://github.com/apple/swift-evolution/blob/master/process.md>

Thank you,

-Dave Abrahams
Review Manager


(Drew Crawford) #2

Proposal link:
<https://lists.swift.org/mailman/listinfo/swift-evolution>

I'm possibly one of the larger users of raw byte stuff in Swift as I maintain an entire client/server network protocol stack in Swift userspace, similar in spirit to one of the examples drawn out a lot longer. Grepping my code produces over 200 individual uses of unsafe byte accesses.

I definitely agree that the problem is significant enough to warrant a last-minute change.

To a first approximation I agree with all the implementation choices. The naming, the choice of UInt8, length tracking, and debug-bounds checking are all correct IMO. We have been using something similar for a long time internally [have you been reading my code? :slight_smile: ] so I can speak from experience that the basic plan here is sound.

One thing I would like to see is an (opt-in) release-mode-bounds-check. Networking is a core use case for this feature, but when you are reading from a socket, production is where you need a guard against out-of-bounds UB the most. If we can't solve it for Swift 3, affected users can write a wrapper to implement the boundscheck, but I think we should at very least take it up again for Swift 4.

Drew

···

On September 1, 2016 at 5:19:02 PM, Andrew Trick via swift-evolution (swift-evolution@swift.org) wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

Reviews are an important part of the Swift evolution process. All reviews
should be sent to the swift-evolution mailing list at

<https://lists.swift.org/mailman/listinfo/swift-evolution>

or, if you would like to keep your feedback private, directly to the
review manager. When replying, please try to keep the proposal link at
the top of the message:

Proposal link:
<https://lists.swift.org/mailman/listinfo/swift-evolution>

What goes into a review?

The goal of the review process is to improve the proposal under review
through constructive criticism and, eventually, determine the direction of
Swift. When writing your review, here are some questions you might want to
answer in your review:

* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a
change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar
feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick
reading, or an in-depth study?

More information about the Swift evolution process is available at

<https://github.com/apple/swift-evolution/blob/master/process.md>

Thank you,

-Dave Abrahams
Review Manager _______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Rien) #3

Please include this in Swift 3.

I just performed a migration of client/server software from Swift 2.3 to Swift 3 beta in Xcode 8 beta 6. And though I like the UnsafeRawPointer approach, I believe this proposal is more Swift-like and is easier to understand.
In fact I used to have my own UInt8 buffer definition that I cleaned out in the migration. Using the proposed solution would have made this process easier.

Rien.

···

On 02 Sep 2016, at 00:18, Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

Reviews are an important part of the Swift evolution process. All reviews
should be sent to the swift-evolution mailing list at

<https://lists.swift.org/mailman/listinfo/swift-evolution>

or, if you would like to keep your feedback private, directly to the
review manager. When replying, please try to keep the proposal link at
the top of the message:

Proposal link:
<https://lists.swift.org/mailman/listinfo/swift-evolution>

What goes into a review?

The goal of the review process is to improve the proposal under review
through constructive criticism and, eventually, determine the direction of
Swift. When writing your review, here are some questions you might want to
answer in your review:

* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a
   change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar
   feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick
   reading, or an in-depth study?

More information about the Swift evolution process is available at

<https://github.com/apple/swift-evolution/blob/master/process.md>

Thank you,

-Dave Abrahams
Review Manager
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Tony Parker) #4

Hi Andy, Dave,

I have two major objections to this proposal.

The first is timing. It is too late for us to evaluate this correctly for Swift 3.

The second is that this API overlaps too much with Foundation’s struct Data. We should be standardizing on a small number of common types, so that developers do not need to find ways to translate one API output into another API’s input.

I think we should instead focus on what needs to be added to struct Data (and other API, like Stream) to fill this role. The proposal touches only touches on this briefly, but in my opinion it is the direction we should pursue — and for Swift 4 or perhaps some Swift 3 update.

- Tony

···

On Sep 1, 2016, at 3:18 PM, Andrew Trick <atrick@apple.com> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

Reviews are an important part of the Swift evolution process. All reviews
should be sent to the swift-evolution mailing list at

<https://lists.swift.org/mailman/listinfo/swift-evolution>

or, if you would like to keep your feedback private, directly to the
review manager. When replying, please try to keep the proposal link at
the top of the message:

Proposal link:
<https://lists.swift.org/mailman/listinfo/swift-evolution>

What goes into a review?

The goal of the review process is to improve the proposal under review
through constructive criticism and, eventually, determine the direction of
Swift. When writing your review, here are some questions you might want to
answer in your review:

* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a
   change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar
   feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick
   reading, or an in-depth study?

More information about the Swift evolution process is available at

<https://github.com/apple/swift-evolution/blob/master/process.md>

Thank you,

-Dave Abrahams
Review Manager
_______________________________________________
swift-evolution-announce mailing list
swift-evolution-announce@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution-announce


(Dave Abrahams) #5

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

* What is your evaluation of the proposal?

I strongly support inclusion of the feature, but I have issues with the
name. It seems to me that in order to fit into the standard library, it
should be called Unsafe[Mutable]RawBufferPointer. Each part of the name
conveys something important, and for the same reasons we're using
Unsafe[Mutable]BufferPointer instead of UnsafeMutableElements, we should
stick to the scheme:

- “Unsafe,” because you can break memory safety with this tool

- “Raw,” because the fundamental model is that of “raw,” rather than
  “typed,” memory.

- “Buffer,” because it works on a series of contiguous elements of known
  length.

- “Pointer,” because it has reference semantics! When you pass one of
  these things around by value, you're not passing the bytes; you're
  passing a shared reference to the bytes.

* Is the problem being addressed significant enough to warrant a
   change to Swift?

Yes, and it fills an important funcationality gap now that we have the
unsafe pointer model nailed down.

* Does this proposal fit well with the feel and direction of Swift?

Yes, except for the name.

* If you have used other languages or libraries with a similar
feature, how do you feel that this proposal compares to those?

I don't think any other language distinguishes raw from typed memory in
this way.

* How much effort did you put into your review? A glance, a quick
reading, or an in-depth study?

Enough :wink:

···

on Thu Sep 01 2016, Andrew Trick <swift-evolution@swift.org> wrote:

--
-Dave, posting as a reviewer, not a review manager


(Andrew Trick) #6

In response to feedback, I've revised SE-0138, now named UnsafeRawBufferPointer, and extended the review period by one week, through September 14.

https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsaferawbufferpointer.md

-Andy

···

On Sep 1, 2016, at 3:18 PM, Andrew Trick <atrick@apple.com> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

Reviews are an important part of the Swift evolution process. All reviews
should be sent to the swift-evolution mailing list at

<https://lists.swift.org/mailman/listinfo/swift-evolution>

or, if you would like to keep your feedback private, directly to the
review manager. When replying, please try to keep the proposal link at
the top of the message:

Proposal link:
<https://lists.swift.org/mailman/listinfo/swift-evolution>

What goes into a review?

The goal of the review process is to improve the proposal under review
through constructive criticism and, eventually, determine the direction of
Swift. When writing your review, here are some questions you might want to
answer in your review:

* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a
   change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar
   feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick
   reading, or an in-depth study?

More information about the Swift evolution process is available at

<https://github.com/apple/swift-evolution/blob/master/process.md>

Thank you,

-Dave Abrahams
Review Manager
_______________________________________________
swift-evolution-announce mailing list
swift-evolution-announce@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution-announce


(Andrew Trick) #7

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

I'm possibly one of the larger users of raw byte stuff in Swift as I maintain an entire client/server network protocol stack in Swift userspace, similar in spirit to one of the examples drawn out a lot longer. Grepping my code produces over 200 individual uses of unsafe byte accesses.

I definitely agree that the problem is significant enough to warrant a last-minute change.

To a first approximation I agree with all the implementation choices. The naming, the choice of UInt8, length tracking, and debug-bounds checking are all correct IMO. We have been using something similar for a long time internally [have you been reading my code? :slight_smile: ] so I can speak from experience that the basic plan here is sound.

One thing I would like to see is an (opt-in) release-mode-bounds-check. Networking is a core use case for this feature, but when you are reading from a socket, production is where you need a guard against out-of-bounds UB the most. If we can't solve it for Swift 3, affected users can write a wrapper to implement the boundscheck, but I think we should at very least take it up again for Swift 4.

Drew

In my current implementation:
https://github.com/atrick/swift/blob/unsafebytes/stdlib/public/core/UnsafeBytes.swift.gyb

The bounds checks in `copyBytes(from:)` are release mode preconditions.

The bounds checks for `subscript`, `load(as:)`, and `storeBytes(of:as:)` are debug only because it’s likely they occur in some loop that could be covered by a single bounds check. By extension, the sequence iterator is only bounds checked in debug mode.

One possibility would be different names for the bounds checked forms of those methods: getByte(atOffset:), setByte(atOffset:), load(fromCheckedOffset:as:), storeBytes(of:toCheckedOffset:as:). Along with some kind of bounds checked Iterator.

I don’t think makes a lot of sense as generic Collection though. Alternatively, we just have an UnsafeBoundsCheckedBytes wrapper.

This would a good thing to experiment with in your project. We may be able to follow-up with a Swift 4 proposal. The important thing now is to determine whether the proposed Swift 3 design will make that wrapper difficult in any way.

-Andy

···

On Sep 1, 2016, at 4:59 PM, Drew Crawford <drew@sealedabstract.com> wrote:
On September 1, 2016 at 5:19:02 PM, Andrew Trick via swift-evolution (swift-evolution@swift.org <mailto:swift-evolution@swift.org>) wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

Reviews are an important part of the Swift evolution process. All reviews
should be sent to the swift-evolution mailing list at

<https://lists.swift.org/mailman/listinfo/swift-evolution>

or, if you would like to keep your feedback private, directly to the
review manager. When replying, please try to keep the proposal link at
the top of the message:

Proposal link:
<https://lists.swift.org/mailman/listinfo/swift-evolution>

What goes into a review?

The goal of the review process is to improve the proposal under review
through constructive criticism and, eventually, determine the direction of
Swift. When writing your review, here are some questions you might want to
answer in your review:

* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a
   change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar
   feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick
   reading, or an in-depth study?

More information about the Swift evolution process is available at

<https://github.com/apple/swift-evolution/blob/master/process.md>

Thank you,

-Dave Abrahams
Review Manager _______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Andrew Trick) #8

After thinking about this for a moment, I like the approach of extending UnsafeBytes with release-mode bounds checked versions of subscript, load, and storeBytes. It’s not actually meaningful to have a bounds checked iterator for UnsafeBytes. A wrapper would only be useful to guard against accidentally circumventing the bounds checks, but I’m not sure that’s really helpful in practice. It seems that a framework would want to provide more abstract Socket I/O or network message abstractions and those wrappers would just call the bounds checked version of the UnsafeBytes APIs.

-Andy

···

On Sep 1, 2016, at 5:37 PM, Andrew Trick <atrick@apple.com> wrote:

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

On Sep 1, 2016, at 4:59 PM, Drew Crawford <drew@sealedabstract.com <mailto:drew@sealedabstract.com>> wrote:
I'm possibly one of the larger users of raw byte stuff in Swift as I maintain an entire client/server network protocol stack in Swift userspace, similar in spirit to one of the examples drawn out a lot longer. Grepping my code produces over 200 individual uses of unsafe byte accesses.

I definitely agree that the problem is significant enough to warrant a last-minute change.

To a first approximation I agree with all the implementation choices. The naming, the choice of UInt8, length tracking, and debug-bounds checking are all correct IMO. We have been using something similar for a long time internally [have you been reading my code? :slight_smile: ] so I can speak from experience that the basic plan here is sound.

One thing I would like to see is an (opt-in) release-mode-bounds-check. Networking is a core use case for this feature, but when you are reading from a socket, production is where you need a guard against out-of-bounds UB the most. If we can't solve it for Swift 3, affected users can write a wrapper to implement the boundscheck, but I think we should at very least take it up again for Swift 4.

Drew

In my current implementation:
https://github.com/atrick/swift/blob/unsafebytes/stdlib/public/core/UnsafeBytes.swift.gyb

The bounds checks in `copyBytes(from:)` are release mode preconditions.

The bounds checks for `subscript`, `load(as:)`, and `storeBytes(of:as:)` are debug only because it’s likely they occur in some loop that could be covered by a single bounds check. By extension, the sequence iterator is only bounds checked in debug mode.

One possibility would be different names for the bounds checked forms of those methods: getByte(atOffset:), setByte(atOffset:), load(fromCheckedOffset:as:), storeBytes(of:toCheckedOffset:as:). Along with some kind of bounds checked Iterator.

I don’t think makes a lot of sense as generic Collection though. Alternatively, we just have an UnsafeBoundsCheckedBytes wrapper.

This would a good thing to experiment with in your project. We may be able to follow-up with a Swift 4 proposal. The important thing now is to determine whether the proposed Swift 3 design will make that wrapper difficult in any way.


(Félix Cloutier) #9

I'm +1 on the proposal. Not a lot to say about it; I don't expect the community to have a passionate argument either.

Félix

···

Le 1 sept. 2016 à 17:37:47, Andrew Trick via swift-evolution <swift-evolution@swift.org> a écrit :

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

On Sep 1, 2016, at 4:59 PM, Drew Crawford <drew@sealedabstract.com <mailto:drew@sealedabstract.com>> wrote:
I'm possibly one of the larger users of raw byte stuff in Swift as I maintain an entire client/server network protocol stack in Swift userspace, similar in spirit to one of the examples drawn out a lot longer. Grepping my code produces over 200 individual uses of unsafe byte accesses.

I definitely agree that the problem is significant enough to warrant a last-minute change.

To a first approximation I agree with all the implementation choices. The naming, the choice of UInt8, length tracking, and debug-bounds checking are all correct IMO. We have been using something similar for a long time internally [have you been reading my code? :slight_smile: ] so I can speak from experience that the basic plan here is sound.

One thing I would like to see is an (opt-in) release-mode-bounds-check. Networking is a core use case for this feature, but when you are reading from a socket, production is where you need a guard against out-of-bounds UB the most. If we can't solve it for Swift 3, affected users can write a wrapper to implement the boundscheck, but I think we should at very least take it up again for Swift 4.

Drew

In my current implementation:
https://github.com/atrick/swift/blob/unsafebytes/stdlib/public/core/UnsafeBytes.swift.gyb

The bounds checks in `copyBytes(from:)` are release mode preconditions.

The bounds checks for `subscript`, `load(as:)`, and `storeBytes(of:as:)` are debug only because it’s likely they occur in some loop that could be covered by a single bounds check. By extension, the sequence iterator is only bounds checked in debug mode.

One possibility would be different names for the bounds checked forms of those methods: getByte(atOffset:), setByte(atOffset:), load(fromCheckedOffset:as:), storeBytes(of:toCheckedOffset:as:). Along with some kind of bounds checked Iterator.

I don’t think makes a lot of sense as generic Collection though. Alternatively, we just have an UnsafeBoundsCheckedBytes wrapper.

This would a good thing to experiment with in your project. We may be able to follow-up with a Swift 4 proposal. The important thing now is to determine whether the proposed Swift 3 design will make that wrapper difficult in any way.

-Andy

On September 1, 2016 at 5:19:02 PM, Andrew Trick via swift-evolution (swift-evolution@swift.org <mailto:swift-evolution@swift.org>) wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

Reviews are an important part of the Swift evolution process. All reviews
should be sent to the swift-evolution mailing list at

<https://lists.swift.org/mailman/listinfo/swift-evolution>

or, if you would like to keep your feedback private, directly to the
review manager. When replying, please try to keep the proposal link at
the top of the message:

Proposal link:
<https://lists.swift.org/mailman/listinfo/swift-evolution>

What goes into a review?

The goal of the review process is to improve the proposal under review
through constructive criticism and, eventually, determine the direction of
Swift. When writing your review, here are some questions you might want to
answer in your review:

* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a
   change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar
   feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick
   reading, or an in-depth study?

More information about the Swift evolution process is available at

<https://github.com/apple/swift-evolution/blob/master/process.md>

Thank you,

-Dave Abrahams
Review Manager _______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Andrew Trick) #10

Hi Andy, Dave,

I have two major objections to this proposal.

The first is timing. It is too late for us to evaluate this correctly for Swift 3.

The second is that this API overlaps too much with Foundation’s struct Data. We should be standardizing on a small number of common types, so that developers do not need to find ways to translate one API output into another API’s input.

I think we should instead focus on what needs to be added to struct Data (and other API, like Stream) to fill this role. The proposal touches only touches on this briefly, but in my opinion it is the direction we should pursue — and for Swift 4 or perhaps some Swift 3 update.

- Tony

Foundation Data needs an interface to UnsafePointers. UnsafeBytes
perfectly fits that need. The same is true of any Stream or
BinaryFormat abstraction that we design in the future. In fact,
migrating Data itself to the raw pointer changes in Swift 3 is enough
motivation to add UnsafeBytes.

It's really unfortunate that Data's interface can't take advantage of
UnsafeBytes in Swift 3 because it would help with adoption of
Data. There is currently a design flaw whenever a client of Data
doesn't know the memory's original type. I'm not happy about that, but
I think we can live with it for a while.

The issue at hand is the large amount of Swift code out there working
with UnsafePointers. We urgently need to provide a migration strategy
for that code. Obviously, we don't have an urgent need to migrate code
already using Data, so improving it's interface can wait until Swift 4.

I've seen many attempts to migrate to Swift 3 (this has been my job
for the past month). I can claim with certainty that if we don't
give developers a natural way to replace their UnsafePointer<UInt8>,
we are going to be left with a lot of incorrect Swift code.

Let me make it clear that there is no overlap between Data and UnsafeBytes.

For public APIs, UnsafeBytes is meant to replace those functions that
currently take (UnsafePointer<UInt8>, Int) including Data's own
interfaces. Otherwise, we're strongly encouraging users to write
incorrect code on the client side. Won't NSStream, for example,
continue to to support UnsafePointer for those developers who need it?
If the developer does not need UnsafePointers, that's great, and
that's what we should continue striving for. But when developers are
using UnsafePointer, we need a natural way to use it correctly.

For general Swift code, Unsafe means something special and
important. The ultimate goal of the standard library and frameworks is
that application developers never need to do something Unsafe. If
they do, it needs to be explicitly marked Unsafe. If we are promoting
Foundation Data as the right way to solve problems for app developers,
then it needs to *not* be Unsafe.

UnsafeBytes simply provides a missing bridge between Unsafe pointers
and safe APIs like Data. By definition, Data and UnsafeBytes use cases
don't overlap. You either need to use Unsafe pointers because you're
programming at the systems level, or existing (safe) libraries do the
job. In practice, there are points at which these worlds meet.

So...

- Today we need UnsafeBytes so that we can migrate existing Swift
  code correctly to a well-defined memory model.

- In the future we need UnsafeBytes to safely implement the transitions
  between "systems code" and "application code”.

-Andy

···

On Sep 2, 2016, at 9:31 AM, Tony Parker via swift-evolution <swift-evolution@swift.org> wrote:

On Sep 1, 2016, at 3:18 PM, Andrew Trick <atrick@apple.com <mailto:atrick@apple.com>> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

Reviews are an important part of the Swift evolution process. All reviews
should be sent to the swift-evolution mailing list at

<https://lists.swift.org/mailman/listinfo/swift-evolution>

or, if you would like to keep your feedback private, directly to the
review manager. When replying, please try to keep the proposal link at
the top of the message:

Proposal link:
<https://lists.swift.org/mailman/listinfo/swift-evolution>

What goes into a review?

The goal of the review process is to improve the proposal under review
through constructive criticism and, eventually, determine the direction of
Swift. When writing your review, here are some questions you might want to
answer in your review:

* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a
   change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar
   feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick
   reading, or an in-depth study?

More information about the Swift evolution process is available at

<https://github.com/apple/swift-evolution/blob/master/process.md>

Thank you,

-Dave Abrahams
Review Manager
_______________________________________________
swift-evolution-announce mailing list
swift-evolution-announce@swift.org <mailto:swift-evolution-announce@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution-announce

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Andrew Trick) #11

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

* What is your evaluation of the proposal?

I strongly support inclusion of the feature, but I have issues with the

Clearly, otherwise you wouldn't have announced it 4 times :wink:

name. It seems to me that in order to fit into the standard library, it
should be called Unsafe[Mutable]RawBufferPointer. Each part of the name

Well, that's natural from a stdlib designer's viewpoint. It is almost
identical functionality, but it also exposes the UnsafeRawPointer API
for loading and storing arbitrary types. This naming issue was
discussed for a couple weeks on swift-evolution. Let's see if I can
recap inline with your comments.

conveys something important, and for the same reasons we're using
Unsafe[Mutable]BufferPointer instead of UnsafeMutableElements, we should
stick to the scheme:

- “Unsafe,” because you can break memory safety with this tool

OK. Let's not drop that one!

- “Raw,” because the fundamental model is that of “raw,” rather than
“typed,” memory.

To me, bytes only exist in memory. Accessing a byte, as opposed to some
in-memory type, is always a raw access.

- “Buffer,” because it works on a series of contiguous elements of known
length.

To me, bytes always represent a contiguous chunk of raw memory. The
term implies that we're dealing with memory layout, as opposed to just
some opaque chunk of data, which is I think what Foundation Data is for.

- “Pointer,” because it has reference semantics! When you pass one of
these things around by value, you're not passing the bytes; you're
passing a shared reference to the bytes.

Unsafe means that this value doesn't own the memory. I agree with you
that reference semantics are important, and we need to clearly
distinguished this from something like Data. I just think Unsafe is
enough for the name.

UnsafeMutableRawBufferPointer does not actually convey that it can be
viewed as a collection of 8-bit values, which is fairly important.

Now that I've satisfied my pedantic side, let's look at it from the developer's side.
To me it's a question of whether a longer or shorter name is more meaningful in
the natural setting of users' source code:

func foo(bytes: UnsafeMutableRawBufferPointer)

withUnsafeMutableRawBufferPointer(to: &header) {
  foo(bytes: $0)
}

···

On Sep 2, 2016, at 11:14 AM, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:
on Thu Sep 01 2016, Andrew Trick <swift-evolution@swift.org> wrote:

---
func foo(bytes: UnsafeMutableBytes)

withUnsafeBytes(of: &header) {
  write(bytes: $0)
}

I don't think the longer name is more descriptive. I do think the
shorter name is more intuitive and meaningful.

UnsafeMutableRawPointer is already too long to be recognizable to
users. A benefit of UnsafeBytes is that the most developers won't need
to know how to work directly with raw pointers. So the name doesn’t
need to evoke them.

-Andy


(Andrew Trick) #12

https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsaferawbufferpointer.md

The review period has been extended until September 14. The UnsafeRawBufferPointer type name is settled, but we still need to come up with an answer for the name of the new closure taking functions:

withXyz() should normally reveal the closure argument type as Xyz. That's why I originally proposed UnsafeBytes as the type name. Now that we've decided to use the descriptive type instead we have a problem...

In this code, it's obvious that a sequence of bytes is being appended to an array.

var buffer = [UInt8]()
withUnsafeBytes(of: &header) {
  buffer += $0
}

In the following version, the closure argument type is obvious, which is nice, but otherwise it's borderline unreadable, and doesn't describe what's actually happenning. How can we tell that a sequence of bytes will be appended?

var buffer = [UInt8]()
withUnsafeRawBufferPointer(to: &header) {
  buffer += $0
}

The mutable version really stretches the limits of descriptively naming things, and still doesn't say anything about a byte sequence:

withUnsafeMutableRawBufferPointer(to: &header) {
  readHeader(into: $0)
}

-Andy

···

On Sep 2, 2016, at 11:14 AM, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:

on Thu Sep 01 2016, Andrew Trick <swift-evolution@swift.org> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

* What is your evaluation of the proposal?

I strongly support inclusion of the feature, but I have issues with the
name. It seems to me that in order to fit into the standard library, it
should be called Unsafe[Mutable]RawBufferPointer. Each part of the name
conveys something important, and for the same reasons we're using
Unsafe[Mutable]BufferPointer instead of UnsafeMutableElements, we should
stick to the scheme:

- “Unsafe,” because you can break memory safety with this tool

- “Raw,” because the fundamental model is that of “raw,” rather than
“typed,” memory.

- “Buffer,” because it works on a series of contiguous elements of known
length.

- “Pointer,” because it has reference semantics! When you pass one of
these things around by value, you're not passing the bytes; you're
passing a shared reference to the bytes.

* Is the problem being addressed significant enough to warrant a
  change to Swift?

Yes, and it fills an important funcationality gap now that we have the
unsafe pointer model nailed down.

* Does this proposal fit well with the feel and direction of Swift?

Yes, except for the name.

* If you have used other languages or libraries with a similar
feature, how do you feel that this proposal compares to those?

I don't think any other language distinguishes raw from typed memory in
this way.

* How much effort did you put into your review? A glance, a quick
reading, or an in-depth study?

Enough :wink:

--
-Dave, posting as a reviewer, not a review manager

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Dave Abrahams) #13

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>>

I'm possibly one of the larger users of raw byte stuff in Swift as
I maintain an entire client/server network protocol stack in Swift
userspace, similar in spirit to one of the examples drawn out a lot
longer. Grepping my code produces over 200 individual uses of
unsafe byte accesses.

I definitely agree that the problem is significant enough to warrant a last-minute change.

To a first approximation I agree with all the implementation
choices. The naming, the choice of UInt8, length tracking, and
debug-bounds checking are all correct IMO. We have been using
something similar for a long time internally [have you been reading
my code? :slight_smile: ] so I can speak from experience that the basic plan
here is sound.

One thing I would like to see is an (opt-in)
release-mode-bounds-check. Networking is a core use case for this
feature, but when you are reading from a socket, production is
where you need a guard against out-of-bounds UB the most. If we
can't solve it for Swift 3, affected users can write a wrapper to
implement the boundscheck, but I think we should at very least take
it up again for Swift 4.

Drew

In my current implementation:
https://github.com/atrick/swift/blob/unsafebytes/stdlib/public/core/UnsafeBytes.swift.gyb
<https://github.com/atrick/swift/blob/unsafebytes/stdlib/public/core/UnsafeBytes.swift.gyb>

The bounds checks in `copyBytes(from:)` are release mode preconditions.

The bounds checks for `subscript`, `load(as:)`, and
`storeBytes(of:as:)` are debug only because it’s likely they occur
in some loop that could be covered by a single bounds check. By
extension, the sequence iterator is only bounds checked in debug
mode.

One possibility would be different names for the bounds checked
forms of those methods: getByte(atOffset:), setByte(atOffset:),
load(fromCheckedOffset:as:),
storeBytes(of:toCheckedOffset:as:). Along with some kind of bounds
checked Iterator.

I don’t think makes a lot of sense as generic Collection though. Alternatively, we just have an UnsafeBoundsCheckedBytes wrapper.

This would a good thing to experiment with in your project. We may
be able to follow-up with a Swift 4 proposal. The important thing
now is to determine whether the proposed Swift 3 design will make
that wrapper difficult in any way.

After thinking about this for a moment, I like the approach of
extending UnsafeBytes with release-mode bounds checked versions of
subscript, load, and storeBytes.

I agree, but would like to carefully design how we do this in
conjunction with Unsafe[Mutable]BufferPointer. For example, maybe

      p.boundsChecked

gives you an UnsafeBoundsChecked[Mutable][Raw]BufferPointer.

So I think this should be a separate proposal.

···

on Fri Sep 02 2016, Andrew Trick <swift-evolution@swift.org> wrote:

On Sep 1, 2016, at 5:37 PM, Andrew Trick <atrick@apple.com> wrote:

On Sep 1, 2016, at 4:59 PM, Drew Crawford >>> <drew@sealedabstract.com >>> <mailto:drew@sealedabstract.com>> >>> wrote:

It’s not actually meaningful to have a bounds checked iterator for
UnsafeBytes. A wrapper would only be useful to guard against
accidentally circumventing the bounds checks, but I’m not sure that’s
really helpful in practice. It seems that a framework would want to
provide more abstract Socket I/O or network message abstractions and
those wrappers would just call the bounds checked version of the
UnsafeBytes APIs.

-Andy

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
-Dave


(Tony Parker) #14

Hi Andy,

Hi Andy, Dave,

I have two major objections to this proposal.

The first is timing. It is too late for us to evaluate this correctly for Swift 3.

The second is that this API overlaps too much with Foundation’s struct Data. We should be standardizing on a small number of common types, so that developers do not need to find ways to translate one API output into another API’s input.

I think we should instead focus on what needs to be added to struct Data (and other API, like Stream) to fill this role. The proposal touches only touches on this briefly, but in my opinion it is the direction we should pursue — and for Swift 4 or perhaps some Swift 3 update.

- Tony

Foundation Data needs an interface to UnsafePointers. UnsafeBytes
perfectly fits that need. The same is true of any Stream or
BinaryFormat abstraction that we design in the future. In fact,
migrating Data itself to the raw pointer changes in Swift 3 is enough
motivation to add UnsafeBytes.

It's really unfortunate that Data's interface can't take advantage of
UnsafeBytes in Swift 3 because it would help with adoption of
Data. There is currently a design flaw whenever a client of Data
doesn't know the memory's original type. I'm not happy about that, but
I think we can live with it for a while.

The issue at hand is the large amount of Swift code out there working
with UnsafePointers. We urgently need to provide a migration strategy
for that code. Obviously, we don't have an urgent need to migrate code
already using Data, so improving it's interface can wait until Swift 4.

I've seen many attempts to migrate to Swift 3 (this has been my job
for the past month). I can claim with certainty that if we don't
give developers a natural way to replace their UnsafePointer<UInt8>,
we are going to be left with a lot of incorrect Swift code.

Let me make it clear that there is no overlap between Data and UnsafeBytes.

For public APIs, UnsafeBytes is meant to replace those functions that
currently take (UnsafePointer<UInt8>, Int) including Data's own
interfaces. Otherwise, we're strongly encouraging users to write
incorrect code on the client side. Won't NSStream, for example,
continue to to support UnsafePointer for those developers who need it?
If the developer does not need UnsafePointers, that's great, and
that's what we should continue striving for. But when developers are
using UnsafePointer, we need a natural way to use it correctly.

For general Swift code, Unsafe means something special and
important. The ultimate goal of the standard library and frameworks is
that application developers never need to do something Unsafe. If
they do, it needs to be explicitly marked Unsafe. If we are promoting
Foundation Data as the right way to solve problems for app developers,
then it needs to *not* be Unsafe.

UnsafeBytes simply provides a missing bridge between Unsafe pointers
and safe APIs like Data. By definition, Data and UnsafeBytes use cases
don't overlap. You either need to use Unsafe pointers because you're
programming at the systems level, or existing (safe) libraries do the
job. In practice, there are points at which these worlds meet.

So...

- Today we need UnsafeBytes so that we can migrate existing Swift
  code correctly to a well-defined memory model.

- In the future we need UnsafeBytes to safely implement the transitions
  between "systems code" and "application code”.

-Andy

If the goal is to simplify the story for developers, so they can understand the complicated topic of the way that binding memory works in Swift, then introducing a new intermediate type feels to me like working in the opposite direction.

I would instead prefer to look at what we can do with adding API to the existing types to cover this use case. You point out in the proposal that it has become customary to use [UInt8] in API. I would prefer that we work towards a solution that makes it customary to use Data when you want to expose an API that uses Data. The example of migrated code in the proposal illustrates my concern. It suggests that the conversion for handleMessages should end here:

func handleMessages(_ bytes: UnsafeBytes) -> Int

I think instead handleMessages should take a Data argument. The input driver code should be able to use API on Data (or elsewhere, API that returns Data) to populate it with the contents of the file. For example, the existing:

public init(contentsOf url: URL, options: Data.ReadingOptions = []) throws

Or by extending Stream, if necessary.

The JSON example advocates converting the struct Data to an NSData to access the bytes property. We specifically introduced the withUnsafeBytes argument to keep people from doing this. If there is an issue with it, we need to address it in struct Data, not by asking people to create another instance of a new type after bridging to the reference. Furthermore, with the lack of bridging on Linux plus the lack of the autoreleasepool that makes the bytes property safe in the first place, this example won’t even work there.

Arguments of being out of time do not compel me, personally. Introducing a new type is effectively permanent. We just got started with introducing Swift API. I don’t want to accidentally saddle ourselves with additional complexity forever without giving ourselves an adequate opportunity to fully consider what the right long-term solution is. The fact that you’ve already described the mismatch between struct Data and this type as “unfortunate” sounds, to me, like a big problem.

- Tony

···

On Sep 2, 2016, at 1:08 PM, Andrew Trick <atrick@apple.com> wrote:

On Sep 2, 2016, at 9:31 AM, Tony Parker via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Sep 1, 2016, at 3:18 PM, Andrew Trick <atrick@apple.com <mailto:atrick@apple.com>> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

Reviews are an important part of the Swift evolution process. All reviews
should be sent to the swift-evolution mailing list at

<https://lists.swift.org/mailman/listinfo/swift-evolution>

or, if you would like to keep your feedback private, directly to the
review manager. When replying, please try to keep the proposal link at
the top of the message:

Proposal link:
<https://lists.swift.org/mailman/listinfo/swift-evolution>

What goes into a review?

The goal of the review process is to improve the proposal under review
through constructive criticism and, eventually, determine the direction of
Swift. When writing your review, here are some questions you might want to
answer in your review:

* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a
   change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar
   feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick
   reading, or an in-depth study?

More information about the Swift evolution process is available at

<https://github.com/apple/swift-evolution/blob/master/process.md>

Thank you,

-Dave Abrahams
Review Manager
_______________________________________________
swift-evolution-announce mailing list
swift-evolution-announce@swift.org <mailto:swift-evolution-announce@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution-announce

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Rien) #15

Probably the biggest argument is that it will make the migration more palatable.
Most (??) apps today use some form of communication, and proving a better pointer solution is a big selling point.
I know that my migration would have been faster and more efficient…

Rien.

···

On 02 Sep 2016, at 16:45, Félix Cloutier via swift-evolution <swift-evolution@swift.org> wrote:

I'm +1 on the proposal. Not a lot to say about it; I don't expect the community to have a passionate argument either.

Félix

Le 1 sept. 2016 à 17:37:47, Andrew Trick via swift-evolution <swift-evolution@swift.org> a écrit :

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

On Sep 1, 2016, at 4:59 PM, Drew Crawford <drew@sealedabstract.com> wrote:
I'm possibly one of the larger users of raw byte stuff in Swift as I maintain an entire client/server network protocol stack in Swift userspace, similar in spirit to one of the examples drawn out a lot longer. Grepping my code produces over 200 individual uses of unsafe byte accesses.

I definitely agree that the problem is significant enough to warrant a last-minute change.

To a first approximation I agree with all the implementation choices. The naming, the choice of UInt8, length tracking, and debug-bounds checking are all correct IMO. We have been using something similar for a long time internally [have you been reading my code? :slight_smile: ] so I can speak from experience that the basic plan here is sound.

One thing I would like to see is an (opt-in) release-mode-bounds-check. Networking is a core use case for this feature, but when you are reading from a socket, production is where you need a guard against out-of-bounds UB the most. If we can't solve it for Swift 3, affected users can write a wrapper to implement the boundscheck, but I think we should at very least take it up again for Swift 4.

Drew

In my current implementation:
https://github.com/atrick/swift/blob/unsafebytes/stdlib/public/core/UnsafeBytes.swift.gyb

The bounds checks in `copyBytes(from:)` are release mode preconditions.

The bounds checks for `subscript`, `load(as:)`, and `storeBytes(of:as:)` are debug only because it’s likely they occur in some loop that could be covered by a single bounds check. By extension, the sequence iterator is only bounds checked in debug mode.

One possibility would be different names for the bounds checked forms of those methods: getByte(atOffset:), setByte(atOffset:), load(fromCheckedOffset:as:), storeBytes(of:toCheckedOffset:as:). Along with some kind of bounds checked Iterator.

I don’t think makes a lot of sense as generic Collection though. Alternatively, we just have an UnsafeBoundsCheckedBytes wrapper.

This would a good thing to experiment with in your project. We may be able to follow-up with a Swift 4 proposal. The important thing now is to determine whether the proposed Swift 3 design will make that wrapper difficult in any way.

-Andy

On September 1, 2016 at 5:19:02 PM, Andrew Trick via swift-evolution (swift-evolution@swift.org) wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

Reviews are an important part of the Swift evolution process. All reviews
should be sent to the swift-evolution mailing list at

<https://lists.swift.org/mailman/listinfo/swift-evolution>

or, if you would like to keep your feedback private, directly to the
review manager. When replying, please try to keep the proposal link at
the top of the message:

Proposal link:
<https://lists.swift.org/mailman/listinfo/swift-evolution>

What goes into a review?

The goal of the review process is to improve the proposal under review
through constructive criticism and, eventually, determine the direction of
Swift. When writing your review, here are some questions you might want to
answer in your review:

* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a
   change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar
   feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick
   reading, or an in-depth study?

More information about the Swift evolution process is available at

<https://github.com/apple/swift-evolution/blob/master/process.md>

Thank you,

-Dave Abrahams
Review Manager _______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Drew Crawford) #16

After thinking about this for a moment, I like the approach of extending UnsafeBytes with release-mode bounds checked versions of subscript, load, and storeBytes.
I agree with this, I think it's mostly a question of naming and defaults. My concern here is letting a swift developer accidentally write heartbleed, which we can't actually prevent, but we can make it harder.

IMO

1. There should be clear consistency in the checked-ness of the API surface. Agree that checked iterator makes no sense, but I think the most important thing is to avoid creating a job interview trivia game where `set` is checked but `store` is unchecked, spot the bug in this function.

2. For consistency with UnsafeBufferPointer it may make the most sense to just ship unchecked or ship an opt-in checked wrapper. I believe however that the existing precedent is all wrong on this point, and I'd like to see us revisit this question across both interfaces in Swift 4, but I don't want to lay out a whole case here that should be its own thread.

···

On September 2, 2016 at 2:36:43 AM, Andrew Trick (atrick@apple.com) wrote:


(Andrew Trick) #17

Update on the naming debate...

I've tentatively updated this proposal renaming UnsafeBytes to UnsafeRawBufferPointer. It's not bad as long as the `withUnsafeBytes` name remains:

https://github.com/atrick/swift-evolution/blob/unsafebytes/proposals/0138-unsaferawbufferpointer.md

I've heard a few compelling arguments to use a long type name. I'm the only person who's argued in favor of the short name. Here are the key points that have convinced me to give in:

- Although I would like "bytes" to only refer to raw, untyped memory,
  in reality it means different things to different people.

- We do not want to promote using this type in public API, except as
  an alternative to other UnsafePointer related types. APIs should
  really migrate to safe, managed types.

- In practice, we may end up with overloads that the more descriptive
  type can help clarify. For example, Data.withUnsafeBytes already
  passes an UnsafePointer to its closure. That API already shipped, but
  we may want a "raw" variant of it. Having the closure take
  `UnsafeRawBufferPointer` clarifies the distinction.

- What was more important to me was that the closure-taking functions
  are reasonably named and imply a collection of bytes over some
  value, which is not necessarilly a buffer to begin with. We can
  still do that. The function name can indicate a collection of bytes
  over some value's representation, while the argument type (which
  doesn't need to be spelled out) specifies that the collection is
  represented as a raw pointer with length:
  `withUnsafeBytes(of: &value) { p: UnsafeRawBufferPointer in ...}

Does anyone have a good argument to keep the short UnsafeBytes type name?

-Andy

···

On Sep 2, 2016, at 5:14 PM, Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:

On Sep 2, 2016, at 11:14 AM, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:

on Thu Sep 01 2016, Andrew Trick <swift-evolution@swift.org> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

* What is your evaluation of the proposal?

I strongly support inclusion of the feature, but I have issues with the

Clearly, otherwise you wouldn't have announced it 4 times :wink:

name. It seems to me that in order to fit into the standard library, it
should be called Unsafe[Mutable]RawBufferPointer. Each part of the name

Well, that's natural from a stdlib designer's viewpoint. It is almost
identical functionality, but it also exposes the UnsafeRawPointer API
for loading and storing arbitrary types. This naming issue was
discussed for a couple weeks on swift-evolution. Let's see if I can
recap inline with your comments.

conveys something important, and for the same reasons we're using
Unsafe[Mutable]BufferPointer instead of UnsafeMutableElements, we should
stick to the scheme:

- “Unsafe,” because you can break memory safety with this tool

OK. Let's not drop that one!

- “Raw,” because the fundamental model is that of “raw,” rather than
“typed,” memory.

To me, bytes only exist in memory. Accessing a byte, as opposed to some
in-memory type, is always a raw access.

- “Buffer,” because it works on a series of contiguous elements of known
length.

To me, bytes always represent a contiguous chunk of raw memory. The
term implies that we're dealing with memory layout, as opposed to just
some opaque chunk of data, which is I think what Foundation Data is for.

- “Pointer,” because it has reference semantics! When you pass one of
these things around by value, you're not passing the bytes; you're
passing a shared reference to the bytes.

Unsafe means that this value doesn't own the memory. I agree with you
that reference semantics are important, and we need to clearly
distinguished this from something like Data. I just think Unsafe is
enough for the name.

UnsafeMutableRawBufferPointer does not actually convey that it can be
viewed as a collection of 8-bit values, which is fairly important.

Now that I've satisfied my pedantic side, let's look at it from the developer's side.
To me it's a question of whether a longer or shorter name is more meaningful in
the natural setting of users' source code:

func foo(bytes: UnsafeMutableRawBufferPointer)

withUnsafeMutableRawBufferPointer(to: &header) {
foo(bytes: $0)
}
---
func foo(bytes: UnsafeMutableBytes)

withUnsafeBytes(of: &header) {
write(bytes: $0)
}

I don't think the longer name is more descriptive. I do think the
shorter name is more intuitive and meaningful.

UnsafeMutableRawPointer is already too long to be recognizable to
users. A benefit of UnsafeBytes is that the most developers won't need
to know how to work directly with raw pointers. So the name doesn’t
need to evoke them.

-Andy


(Andrew Trick) #18

Hi Andy, Dave,

I have two major objections to this proposal.

The first is timing. It is too late for us to evaluate this correctly for Swift 3.

The second is that this API overlaps too much with Foundation’s struct Data. We should be standardizing on a small number of common types, so that developers do not need to find ways to translate one API output into another API’s input.

I think we should instead focus on what needs to be added to struct Data (and other API, like Stream) to fill this role. The proposal touches only touches on this briefly, but in my opinion it is the direction we should pursue — and for Swift 4 or perhaps some Swift 3 update.

- Tony

Here's a quick follow up to the Data vs. UnsafeBytes discussion, merely because I think it's interesting and will help people understand both types. Some of the immediate concern will be alleviated by renaming UnsafeBytes to UnsafeRawBufferPointer.

Ignoring 'bytesNoCopy' for a moment:

- Both are abstractions over raw, untyped memory.

- Both can be viewed as a collection of UInt8 bytes.

- Data has value semantics. UnsafeRawBufferPointer has pointer semantics.

- Data is owned. UnsafeRawBufferPointer is unowned.

- Data is a self-contained object. UnsafeRawBufferPointer is a slice of memory.

- Data is growable.

'init(bytesNoCopy:)' is a special case that allows Data to circumvent normal behavior as an optimization. But getting data in via 'bytesNoCopy' requires first obtaining an unowned view of a slice of raw memory, and that is precisely what UnsafeRawBufferPointer is. Taking an unsafe, unowned view of memory should always be explicit in user code, not something that should be hidden behind the Data API. We need the UnsafeRawBufferPointer type in order to make that conversion explicit.

These data types represent distinct and individually important layers of the semantics of raw data. They do not compete for use cases at the same API level. UnsafeRawBufferPointer only supports the basic functionality of interoperating with UnsafePointers and accessing existing raw memory. Data has extensive API surface for constructing a buffer of serialized data and moving that data across library boundaries. Data is the common currency for framework APIs.

-Andy

···

On Sep 2, 2016, at 1:08 PM, Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:

On Sep 2, 2016, at 9:31 AM, Tony Parker via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Foundation Data needs an interface to UnsafePointers. UnsafeBytes
perfectly fits that need. The same is true of any Stream or
BinaryFormat abstraction that we design in the future. In fact,
migrating Data itself to the raw pointer changes in Swift 3 is enough
motivation to add UnsafeBytes.

It's really unfortunate that Data's interface can't take advantage of
UnsafeBytes in Swift 3 because it would help with adoption of
Data. There is currently a design flaw whenever a client of Data
doesn't know the memory's original type. I'm not happy about that, but
I think we can live with it for a while.

The issue at hand is the large amount of Swift code out there working
with UnsafePointers. We urgently need to provide a migration strategy
for that code. Obviously, we don't have an urgent need to migrate code
already using Data, so improving it's interface can wait until Swift 4.

I've seen many attempts to migrate to Swift 3 (this has been my job
for the past month). I can claim with certainty that if we don't
give developers a natural way to replace their UnsafePointer<UInt8>,
we are going to be left with a lot of incorrect Swift code.

Let me make it clear that there is no overlap between Data and UnsafeBytes.

For public APIs, UnsafeBytes is meant to replace those functions that
currently take (UnsafePointer<UInt8>, Int) including Data's own
interfaces. Otherwise, we're strongly encouraging users to write
incorrect code on the client side. Won't NSStream, for example,
continue to to support UnsafePointer for those developers who need it?
If the developer does not need UnsafePointers, that's great, and
that's what we should continue striving for. But when developers are
using UnsafePointer, we need a natural way to use it correctly.

For general Swift code, Unsafe means something special and
important. The ultimate goal of the standard library and frameworks is
that application developers never need to do something Unsafe. If
they do, it needs to be explicitly marked Unsafe. If we are promoting
Foundation Data as the right way to solve problems for app developers,
then it needs to *not* be Unsafe.

UnsafeBytes simply provides a missing bridge between Unsafe pointers
and safe APIs like Data. By definition, Data and UnsafeBytes use cases
don't overlap. You either need to use Unsafe pointers because you're
programming at the systems level, or existing (safe) libraries do the
job. In practice, there are points at which these worlds meet.

So...

- Today we need UnsafeBytes so that we can migrate existing Swift
  code correctly to a well-defined memory model.

- In the future we need UnsafeBytes to safely implement the transitions
  between "systems code" and "application code”.

-Andy

On Sep 1, 2016, at 3:18 PM, Andrew Trick <atrick@apple.com <mailto:atrick@apple.com>> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

Reviews are an important part of the Swift evolution process. All reviews
should be sent to the swift-evolution mailing list at

<https://lists.swift.org/mailman/listinfo/swift-evolution>

or, if you would like to keep your feedback private, directly to the
review manager. When replying, please try to keep the proposal link at
the top of the message:

Proposal link:
<https://lists.swift.org/mailman/listinfo/swift-evolution>

What goes into a review?

The goal of the review process is to improve the proposal under review
through constructive criticism and, eventually, determine the direction of
Swift. When writing your review, here are some questions you might want to
answer in your review:

* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a
   change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar
   feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick
   reading, or an in-depth study?

More information about the Swift evolution process is available at

<https://github.com/apple/swift-evolution/blob/master/process.md>

Thank you,

-Dave Abrahams
Review Manager
_______________________________________________
swift-evolution-announce mailing list
swift-evolution-announce@swift.org <mailto:swift-evolution-announce@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution-announce

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Rick M) #19

Coincidentally, I just wrote my first Swift code to use UnsafePointer<>. I was wrapping the LZMA API to decompress LZMA data. It's a C API that works by pointing to an input buffer and and output buffer, and then calling a function that decompresses what it can given those two buffers (and their lengths).

I treated them as UnsafePointer<UInt8>, but really they're raw, in the sense that they are not a collection of a single element, just a collection of bytes.

My wrapper's interface to LZMA uses Data instances. I don't see a way of getting from Data to UnsafeRawBufferPointer in Xcode 8 GM seed (which makes sense, given that this is still in progress). But I also didn't see a way to get to UnsafeRawPointer; should there be?

Will something be added to Data when SE-0138 is finalized? I guess that's not for Swift 3 but 3.x?

Thanks, and sorry if I'm hijacking the thread a bit with this.

···

On Sep 10, 2016, at 17:53 , Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:

https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsaferawbufferpointer.md

The review period has been extended until September 14. The UnsafeRawBufferPointer type name is settled, but we still need to come up with an answer for the name of the new closure taking functions:

withXyz() should normally reveal the closure argument type as Xyz. That's why I originally proposed UnsafeBytes as the type name. Now that we've decided to use the descriptive type instead we have a problem...

In this code, it's obvious that a sequence of bytes is being appended to an array.

var buffer = [UInt8]()
withUnsafeBytes(of: &header) {
buffer += $0
}

In the following version, the closure argument type is obvious, which is nice, but otherwise it's borderline unreadable, and doesn't describe what's actually happenning. How can we tell that a sequence of bytes will be appended?

var buffer = [UInt8]()
withUnsafeRawBufferPointer(to: &header) {
buffer += $0
}

The mutable version really stretches the limits of descriptively naming things, and still doesn't say anything about a byte sequence:

withUnsafeMutableRawBufferPointer(to: &header) {
readHeader(into: $0)
}

-Andy

On Sep 2, 2016, at 11:14 AM, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:

on Thu Sep 01 2016, Andrew Trick <swift-evolution@swift.org> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

* What is your evaluation of the proposal?

I strongly support inclusion of the feature, but I have issues with the
name. It seems to me that in order to fit into the standard library, it
should be called Unsafe[Mutable]RawBufferPointer. Each part of the name
conveys something important, and for the same reasons we're using
Unsafe[Mutable]BufferPointer instead of UnsafeMutableElements, we should
stick to the scheme:

- “Unsafe,” because you can break memory safety with this tool

- “Raw,” because the fundamental model is that of “raw,” rather than
“typed,” memory.

- “Buffer,” because it works on a series of contiguous elements of known
length.

- “Pointer,” because it has reference semantics! When you pass one of
these things around by value, you're not passing the bytes; you're
passing a shared reference to the bytes.

* Is the problem being addressed significant enough to warrant a
change to Swift?

Yes, and it fills an important funcationality gap now that we have the
unsafe pointer model nailed down.

* Does this proposal fit well with the feel and direction of Swift?

Yes, except for the name.

* If you have used other languages or libraries with a similar
feature, how do you feel that this proposal compares to those?

I don't think any other language distinguishes raw from typed memory in
this way.

* How much effort did you put into your review? A glance, a quick
reading, or an in-depth study?

Enough :wink:

--
-Dave, posting as a reviewer, not a review manager

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Rick Mann
rmann@latencyzero.com


(Rien) #20

Now that “Raw” is introduced, also using “Unsafe” seems redundant to me. So ditch the unsafe and just go for “withRawBytes” and “withMutableRawBytes”.
I expect that most code that uses this type wil already have a name indicating that it concerns a byte buffer pointer, so “withRawBytes” should give sufficient clue as to what is going on.

Rien.

···

On 11 Sep 2016, at 02:53, Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:

https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsaferawbufferpointer.md

The review period has been extended until September 14. The UnsafeRawBufferPointer type name is settled, but we still need to come up with an answer for the name of the new closure taking functions:

withXyz() should normally reveal the closure argument type as Xyz. That's why I originally proposed UnsafeBytes as the type name. Now that we've decided to use the descriptive type instead we have a problem...

In this code, it's obvious that a sequence of bytes is being appended to an array.

var buffer = [UInt8]()
withUnsafeBytes(of: &header) {
buffer += $0
}

In the following version, the closure argument type is obvious, which is nice, but otherwise it's borderline unreadable, and doesn't describe what's actually happenning. How can we tell that a sequence of bytes will be appended?

var buffer = [UInt8]()
withUnsafeRawBufferPointer(to: &header) {
buffer += $0
}

The mutable version really stretches the limits of descriptively naming things, and still doesn't say anything about a byte sequence:

withUnsafeMutableRawBufferPointer(to: &header) {
readHeader(into: $0)
}

-Andy

On Sep 2, 2016, at 11:14 AM, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:

on Thu Sep 01 2016, Andrew Trick <swift-evolution@swift.org> wrote:

I’m resending this for Review Manager Dave A. because the announce list is dropping his messages...

Hello Swift community,

The review of "UnsafeBytes" begins now and runs through September
7th. This late addition to Swift 3 is a follow-up to SE-0107:
UnsafeRawPointer. It addresses common use cases for UnsafeRawPointer,
allowing developers to continue working with collections of UInt8 values,
but now doing so via a type safe API. The UnsafeBytes API will not require
direct manipulation of raw pointers or reasoning about binding memory.

The proposal is available here:

<https://github.com/apple/swift-evolution/blob/master/proposals/0138-unsafebytes.md>

* What is your evaluation of the proposal?

I strongly support inclusion of the feature, but I have issues with the
name. It seems to me that in order to fit into the standard library, it
should be called Unsafe[Mutable]RawBufferPointer. Each part of the name
conveys something important, and for the same reasons we're using
Unsafe[Mutable]BufferPointer instead of UnsafeMutableElements, we should
stick to the scheme:

- “Unsafe,” because you can break memory safety with this tool

- “Raw,” because the fundamental model is that of “raw,” rather than
“typed,” memory.

- “Buffer,” because it works on a series of contiguous elements of known
length.

- “Pointer,” because it has reference semantics! When you pass one of
these things around by value, you're not passing the bytes; you're
passing a shared reference to the bytes.

* Is the problem being addressed significant enough to warrant a
change to Swift?

Yes, and it fills an important funcationality gap now that we have the
unsafe pointer model nailed down.

* Does this proposal fit well with the feel and direction of Swift?

Yes, except for the name.

* If you have used other languages or libraries with a similar
feature, how do you feel that this proposal compares to those?

I don't think any other language distinguishes raw from typed memory in
this way.

* How much effort did you put into your review? A glance, a quick
reading, or an in-depth study?

Enough :wink:

--
-Dave, posting as a reviewer, not a review manager

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution