or, if you would like to keep your feedback private, directly to the review manager.
What goes into a review?
The goal of the review process is to improve the proposal under review through constructive criticism and contribute to the direction of Swift. When writing your review, here are some questions you might want to answer in your review:
* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
More information about the Swift evolution process is available at
* What is your evaluation of the proposal?
-1. It seems like the author is trying to solve a non-problem. The pointer APIs already work very well with enough due warning that they are unsafe. This change will just make them even more cumbersome to work with.
* Is the problem being addressed significant enough to warrant a change to Swift?
No, I don’t think so, and if it were it would be too significant to be considered in scope for Swift 3.
* Does this proposal fit well with the feel and direction of Swift?
It’s a valid Swift API, I just feel it makes the UnsafePointer family more complex than it needs to be.
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
Swift’s approach to memory access has been unique in my programming experience and I like it for its power and simplicity.
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
I read most of the proposal and have done a lot of work with the existing pointer APIs. Again I feel this is adding needless complexity and doesn’t actually change what you can do with the API.
···
Hello Swift community,
The review of “SE-0107: UnsafeRawPointer API” begins now and runs through July 4, 2016. The proposal is available here:
or, if you would like to keep your feedback private, directly to the review manager.
What goes into a review?
The goal of the review process is to improve the proposal under review through constructive criticism and contribute to the direction of Swift. When writing your review, here are some questions you might want to answer in your review:
* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
More information about the Swift evolution process is available at
+1. This proposal looks really great to me. I think it will add a lot of clarity to code that needs to work with unsafe pointers. This is really important.
* Is the problem being addressed significant enough to warrant a change to Swift?
Yes.
* Does this proposal fit well with the feel and direction of Swift?
Very much. We should strive for clarity and prevent as much accidental misuse of unsafe constructs as possible while still enabling code that needs to work at low levels. This proposal does a great job of this.
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
This proposal introduces important distinctions in the type system that I haven’t seen elsewhere.
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
In depth study of the proposal and participation in the discussion.
···
More information about the Swift evolution process is available at
+1 This is an excellent change to make a fragile and tricky part of the language more explicit and correct by design.
* Is the problem being addressed significant enough to warrant a change to Swift?
Yes
* Does this proposal fit well with the feel and direction of Swift?
Yes
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
Raw memory access in languages with strong static type systems has typically come with some kind of marshalling framework or there is a virtual machine in place to ensure some measure of consistency with foreign pointers. Because we can make no such guarantees, this proposal provides an excellent middle ground: A new type to encapsulate and mark old semantics plus new typed means of access if desired.
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
Read the proposal a few times.
···
On Jun 28, 2016, at 9:05 PM, Chris Lattner <clattner@apple.com> wrote:
Hello Swift community,
The review of “SE-0107: UnsafeRawPointer API” begins now and runs through July 4, 2016. The proposal is available here:
or, if you would like to keep your feedback private, directly to the review manager.
What goes into a review?
The goal of the review process is to improve the proposal under review through constructive criticism and contribute to the direction of Swift. When writing your review, here are some questions you might want to answer in your review:
* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
More information about the Swift evolution process is available at
* Is there a reason there's a `load` that takes a byte offset, but not a `storeRaw`?
* I'm also a little nervous about the fact that `storeRaw` (and `load`?) is documented to only work properly on "trivial types", but it doesn't have any sort of constraints to ensure it's used correctly. (One could imagine, for instance, the compiler automatically conforming trivial types to a `Trivial` protocol.)
* I don't think I understand `initialize(toContiguous:atIndex:with:)`. Does it return a typed pointer to the whole buffer, or just the one instance it initialized? In the `stringFromBytes` example, shouldn't we either subscript the typed pointer from the previous `initialize(_:with:count:)` call, or call `storeRaw(toContiguous:atIndex:with:)`, rather than initializing memory twice? If this isn't a good use case for `initialize(toContiguous:atIndex:with:)`, what would be?
* * *
I'm quite concerned by the "moveInitialize should be more elegant" section at the bottom.
Since the types are so close, `moveInitialize` could require mutating arguments and actually swap the pointers. For instance:
uninitializedBuffer.swapPointersAfterMoving(from: &buffer, count: count)
// `buffer` now points to the new allocation, filled in with the Ints.
// `uninitializedBuffer` now points to the old allocation, which is deinitialized.
uninitializedBuffer.deallocate()
return buffer
}
This is *such* a strange semantic, however, that I'm not at all sure how to name this function.
`moveAssign(from:count:)` could do something much simpler, returning a raw version of `from`:
target.moveAssign(from: source).deallocate()
`move()`, on the other hand, I don't see a good way to fix like this.
One ridiculous thing we could do for `moveAssign(from:count:)` and perhaps `move()` is to deliberately make `self` invalid by setting it to address 0. If it were `Optional`, this would nil `self`. If it weren't...well, something would probably fail eventually.
* * *
I notice that many APIs require type parameters merely to force the user to explicitly state the types involved. I wonder if we could instead introduce an attribute which you could place on a parameter or return type indicating that there must be an explicit `as` cast specifying its type:
rawPointer.storeRaw(3 as Int)
rawPointer.load() as Int
rawPointer.cast() as UnsafePointer<Int>
This would also be useful on `unsafeBitCast`, and on user APIs which are prone to type inference issues.
* * *
In the long run, however, I wonder if we might end up removing `UnsafeRawPointer`. If `Never` becomes a subtype-of-all-types, then `UnsafePointer<Never>` would gain the basic properties of an `UnsafeRawPointer`:
* Because `Never` is a subtype of all types, `UnsafePointer<Never>` could alias any other pointer.
* Accessing `pointee` would be inherently invalid (it would either take or return a `Never`), and APIs which initialize or set `pointee` would be inherently uncallable.
* `Never` has no intrinsic size, so it could be treated as having a one-byte size, allowing APIs which normally allocate, deallocate, or do pointer arithmetic by instance size to automatically do so by byte size instead.
* APIs for casting an `UnsafePointer<T>` to `UnsafePointer<supertype of T>` or `<subtype of T>` would do the right thing with `UnsafePointer<Never>`.
Thus, I could imagine `Unsafe[Mutable]RawPointer` becoming `Unsafe[Mutable]Pointer<Never>` in the future, with some APIs being generalized and moving to all `UnsafePointer`s while others are in extensions on `UnsafePointer where Pointee == Never`.
It might be worth taking a look at the current API designs and thinking about how they would look in that world:
* Is `nsStringPtr.casting(to: UnsafePointer<NSObject>)` how you would want to write a pointee upcast? How about `UnsafePointer<NSString>(nsObjectPtr)` for a pointee downcast?
* Would you want `initialize<T>(_: T.Type, with: T, count: Int = 1) -> UnsafeMutablePointer<T>` in the `Never` extension, or (with a supertype-of-Pointee constraint on `T`) would it be something you'd put on other `UnsafeMutablePointer`s too? What does that mean for `UnsafeMutablePointer.initialize(with:)`?
* Are `load` or `storeRaw` things that might make sense on any `UnsafeMutablePointer` if they were constrained to supertypes only?
* Are there APIs which are basically the same on `Unsafe[Mutable]Pointer`s and their `Raw` equivalents, except that the `Raw` versions are "dumb" because they don't know what type they're operating on? If so, should they be given the same name?
* Is the problem being addressed significant enough to warrant a change to Swift?
Yes. Even in the realm of Unsafe types, we don't want undefined behavior with no way to define it.
* Does this proposal fit well with the feel and direction of Swift?
Yes. Nailing things down is important.
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
Need I mention how bizarre and arbitrary C pointers are?
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
I've put in a fair few hours on this post, but even so, there are parts of the proposal that I really haven't looked at very deeply; I will admit I rarely work at such a low level and don't fully understand all of the technicalities involved.
I've revised the proposal again based on extremely helpful feedback from DaveA and Jordan.
This revision expands on the concept of formally binding the memory type that I was recently working on with Dmitri. Now we can clearly define pre and post conditions on memory operations and pointer casts that can be used to prove the type safety. The model is now simpler, more complete, and easy to reason about locally. This will help developers reason about correctness and make it easy to implement a sanitizer that verifies the type safety of UnsafePointer operations.
Adding safety to pointer "casts" made it possible for me to actually simplify the allocation and initialization APIs. I think both camps, convenience and safety, will be happy.
You can see what changed in this pull request:
Brief summary:
- Memory is dynamically bound to a single type.
- All typed access to memory, whether via a typed pointer or regular
language construct, must be consistent with the memory's bound type
(the access type must be related to the bound type). Typed access
includes initialization, assignment, or deinitialization via a typed
pointer.
- Memory remains bound after being deinitialized.
- Memory is implicitly bound or rebound to a type by initializing it
via a raw pointer.
- A separate API now exists for explicity binding or rebinding memory
to a type. This allows binding to be decoupled from initialization
for convenience and efficiency. It also supports safe
interoperability between APIs that used different, but layout
compatible types.
- Using an API that accesses memory as a different type can now be
accomplished by rebinding the memory. This effectively changes the
type of any initialized values in memory. The compiler is still
permitted to assume strict aliasing for accesses on either side of
the operation that rebinds memory.
Andy
···
On Jun 28, 2016, at 11:05 PM, Chris Lattner <clattner@apple.com> wrote:
Hello Swift community,
The review of “SE-0107: UnsafeRawPointer API” begins now and runs through July 4, 2016. The proposal is available here:
I have not had time for a full review, but I have a question about the proposal:
When glancing at the examples, they strike me as mostly being marshalling, which in my opinion would be better served by a safe marshalling API followed by unsafe handling of the resulting buffer, and vice versa for unmarshalling. I think it is very important (in the long run) that code that doesn't interact with C directly has safe ways of doing inherently safe operations, and not take the unsafe route just because that is the only API available.
My question is, how does this API fit into the bigger picture of marshalling, and what are the benefits of using this API instead of marshalling with safe buffers?
I would consider "we don't have time, we have to do this for now" a perfectly valid answer.
* What is your evaluation of the proposal?
-1. It seems like the author is trying to solve a non-problem. The pointer APIs already work very well with enough due warning that they are unsafe. This change will just make them even more cumbersome to work with.
* Is the problem being addressed significant enough to warrant a change to Swift?
No, I don’t think so, and if it were it would be too significant to be considered in scope for Swift 3.
* Does this proposal fit well with the feel and direction of Swift?
It’s a valid Swift API, I just feel it makes the UnsafePointer family more complex than it needs to be.
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
Swift’s approach to memory access has been unique in my programming experience and I like it for its power and simplicity.
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
I read most of the proposal and have done a lot of work with the existing pointer APIs. Again I feel this is adding needless complexity and doesn’t actually change what you can do with the API.
Thanks for voicing an opinion because I suspect more people feel this way than are going to express openly, and I want to respond to all those people. I anticipate that a lot of users are going to feel this is more cumbersome and redundant. In fact, I don't expect the API to be easier in most cases, but I do expect it to be much harder to use incorrectly.
The proposal includes some use cases that developers are currently using the UnsafePointer API for but are in fact impossible to implement correctly unless we provide some alternate API. This proposed API evolved over a very long period and many attempts to propose some alternate API, none of which were easy to explain to users. Given my goal of eliminating undefined behavior from Swift code, I'd like to hear how this can be accomplished in a better way. In particular, should we allow UnsafePointer casts at all? How can the user communicate to the compiler that the same memory location is accessed as a different type, particularly when so many users are accustomed to getting away with this in C?
-Andy
···
On Jun 29, 2016, at 8:02 AM, Brad Hilton via swift-evolution <swift-evolution@swift.org> wrote:
Hello Swift community,
The review of “SE-0107: UnsafeRawPointer API” begins now and runs through July 4, 2016. The proposal is available here:
or, if you would like to keep your feedback private, directly to the review manager.
What goes into a review?
The goal of the review process is to improve the proposal under review through constructive criticism and contribute to the direction of Swift. When writing your review, here are some questions you might want to answer in your review:
* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
More information about the Swift evolution process is available at
This is an excellent proposal, and a step forward in balancing safety and un-safety in an understandable way.
I have no issues about the substance, but two documentation issues:
- the compiler’s strict aliasing rules: not clearly defined in this document.
I *think* I know mostly what that means, but I’m not completely sure.
- so-called “trivial” types:
Coming from physics, where trivial means “not worthy of consideration” (an English dictionary concurs.)
It made me wonder why integers deserved such a putdown; the word “simple” would be just fine.
[y’ = y has solutions y(x) = e^x and y = 0; the latter is trivial as it generally isn’t interesting, despite being valid.]
It’s probably meant as “easily proven”, but since no proofs are shown, it feels like an inappropriate use.
* Is the problem being addressed significant enough to warrant a change to Swift?
Absolutely.
* Does this proposal fit well with the feel and direction of Swift?
I think so.
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
I’ve only done the kind of things enabled by this in C/C++, where it just feels like gambling. [feels like testing can only show that something works on one particular C/C++ compiler]. Clarity is good.
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
I’ve followed this discussion from the start, and asked questions. My interest was informed by experimental attempts to push the memory model; those attempts would have been covered by this document.
This is excellent feedback. I can tell that you paid close attention to the API details.
I’m in the middle of working on a revision to the proposal. I'll address your questions after I finish writing it up so that my answers make more sense. I’ll include most of your naming improvements in a separate revision that we can bikeshed on the list.
-Andy
···
On Jul 2, 2016, at 8:10 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:
* What is your evaluation of the proposal?
I think this is basically a good design, but I have lots and lots of comments.
* Is there a reason there's a `load` that takes a byte offset, but not a `storeRaw`?
* I'm also a little nervous about the fact that `storeRaw` (and `load`?) is documented to only work properly on "trivial types", but it doesn't have any sort of constraints to ensure it's used correctly. (One could imagine, for instance, the compiler automatically conforming trivial types to a `Trivial` protocol.)
* I don't think I understand `initialize(toContiguous:atIndex:with:)`. Does it return a typed pointer to the whole buffer, or just the one instance it initialized? In the `stringFromBytes` example, shouldn't we either subscript the typed pointer from the previous `initialize(_:with:count:)` call, or call `storeRaw(toContiguous:atIndex:with:)`, rather than initializing memory twice? If this isn't a good use case for `initialize(toContiguous:atIndex:with:)`, what would be?
* * *
I'm quite concerned by the "moveInitialize should be more elegant" section at the bottom.
Since the types are so close, `moveInitialize` could require mutating arguments and actually swap the pointers. For instance:
uninitializedBuffer.swapPointersAfterMoving(from: &buffer, count: count)
// `buffer` now points to the new allocation, filled in with the Ints.
// `uninitializedBuffer` now points to the old allocation, which is deinitialized.
uninitializedBuffer.deallocate()
return buffer
}
This is *such* a strange semantic, however, that I'm not at all sure how to name this function.
`moveAssign(from:count:)` could do something much simpler, returning a raw version of `from`:
target.moveAssign(from: source).deallocate()
`move()`, on the other hand, I don't see a good way to fix like this.
One ridiculous thing we could do for `moveAssign(from:count:)` and perhaps `move()` is to deliberately make `self` invalid by setting it to address 0. If it were `Optional`, this would nil `self`. If it weren't...well, something would probably fail eventually.
* * *
I notice that many APIs require type parameters merely to force the user to explicitly state the types involved. I wonder if we could instead introduce an attribute which you could place on a parameter or return type indicating that there must be an explicit `as` cast specifying its type:
rawPointer.storeRaw(3 as Int)
rawPointer.load() as Int
rawPointer.cast() as UnsafePointer<Int>
This would also be useful on `unsafeBitCast`, and on user APIs which are prone to type inference issues.
* * *
In the long run, however, I wonder if we might end up removing `UnsafeRawPointer`. If `Never` becomes a subtype-of-all-types, then `UnsafePointer<Never>` would gain the basic properties of an `UnsafeRawPointer`:
* Because `Never` is a subtype of all types, `UnsafePointer<Never>` could alias any other pointer.
* Accessing `pointee` would be inherently invalid (it would either take or return a `Never`), and APIs which initialize or set `pointee` would be inherently uncallable.
* `Never` has no intrinsic size, so it could be treated as having a one-byte size, allowing APIs which normally allocate, deallocate, or do pointer arithmetic by instance size to automatically do so by byte size instead.
* APIs for casting an `UnsafePointer<T>` to `UnsafePointer<supertype of T>` or `<subtype of T>` would do the right thing with `UnsafePointer<Never>`.
Thus, I could imagine `Unsafe[Mutable]RawPointer` becoming `Unsafe[Mutable]Pointer<Never>` in the future, with some APIs being generalized and moving to all `UnsafePointer`s while others are in extensions on `UnsafePointer where Pointee == Never`.
It might be worth taking a look at the current API designs and thinking about how they would look in that world:
* Is `nsStringPtr.casting(to: UnsafePointer<NSObject>)` how you would want to write a pointee upcast? How about `UnsafePointer<NSString>(nsObjectPtr)` for a pointee downcast?
* Would you want `initialize<T>(_: T.Type, with: T, count: Int = 1) -> UnsafeMutablePointer<T>` in the `Never` extension, or (with a supertype-of-Pointee constraint on `T`) would it be something you'd put on other `UnsafeMutablePointer`s too? What does that mean for `UnsafeMutablePointer.initialize(with:)`?
* Are `load` or `storeRaw` things that might make sense on any `UnsafeMutablePointer` if they were constrained to supertypes only?
* Are there APIs which are basically the same on `Unsafe[Mutable]Pointer`s and their `Raw` equivalents, except that the `Raw` versions are "dumb" because they don't know what type they're operating on? If so, should they be given the same name?
* Is the problem being addressed significant enough to warrant a change to Swift?
Yes. Even in the realm of Unsafe types, we don't want undefined behavior with no way to define it.
* Does this proposal fit well with the feel and direction of Swift?
Yes. Nailing things down is important.
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
Need I mention how bizarre and arbitrary C pointers are?
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
I've put in a fair few hours on this post, but even so, there are parts of the proposal that I really haven't looked at very deeply; I will admit I rarely work at such a low level and don't fully understand all of the technicalities involved.
Let's bikeshed this easy one now... I’m curious what others think:
// In general, I think you "initialize to" a value, not
// "initialize with" a value. "with" is needlessly vacuous.
//
// func initialize<T>(_: T.Type, with: T, count: Int = 1)
// -> UnsafeMutablePointer<T>
func initialize<T>(_: T.Type, to: T, count: Int = 1)
-> UnsafeMutablePointer<T>
`initialize` was recently renamed to `initialized(with:)`.
commit d96b051d28b6042adcc8b8692a918abddf211aec
Author: Dave Abrahams <dabrahams@apple.com>
stdlib: initializePointee(_) => initialize(with:)
Tacking "Pointee" on just for unary operations (and especially
operations with an optional count) created inconsistency.
So Swift 3 users have already migrated to this “better” name.
I agree that initialize(to:) is consistent with the language we use for assigning values. But grammatically, I think initialize(with:) also makes perfect sense and is just as common.
In general, if there’s controversy, I’ll stick with the existing conventions because there’s already enough to debate in this proposal.
-Andy
···
On Jul 2, 2016, at 8:10 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:
This is the first version of this proposal which I've had time to read. I
like it a lot overall. If I have some more time, I may try pulling the
branch and writing some code with it to see how it feels. (If we could get
a toolchain built from the branch, that might help others review it.)
Here are a handful of minor comments:
- Naming: "bindMemory(to:capacity:)", being "verb-ish", seems incongruous
with "assumingMemoryBound(to:)" and "withMemoryRebound(to:capacity:_:)".
How about "bindingMemory(to:capacity:)" ?
- Would it be possible for "+(UnsafeRawPointer, Int) -> UnsafeRawPointer"
to accept any Integer or FixedWidthInteger, rather than only Int?
- Why allow/encourage multiple calls to bindMemory on the same RawPointer?
These APIs do a good job of making aliasing explicit by introducing data
dependencies between pointers (you can only use a typed pointer after
obtaining it from a raw pointer, and you can recover the raw pointer for
later use by deinitializing the typed pointer). So, I would think the
guidelines should prefer bindMemory to be used only once on a particular
RawPointer value.
And minor notes about the proposal itself:
- strideof(Int.self) is used in most examples, but sizeof(Int.self) appears
in one of them (the "normalLifetime()" example).
- I think there must be a mistake in this example, because pA is already
bound and bindMemory was only defined for untyped RawPointers:
func testInitAB() {
// Get a raw pointer to (A, B).
let p = initAB()
let pA = p.bindMemory(to: A.self, capacity: 1)
printA(pA)
printB((pA + 1).bindMemory(to: B.self, capacity: 1)) //<<< should
this be (p+1) rather than (pA+1)?
}
Jacob
···
On Mon, Jul 4, 2016 at 3:32 PM, Andrew Trick via swift-evolution < swift-evolution@swift.org> wrote:
On Jun 28, 2016, at 11:05 PM, Chris Lattner <clattner@apple.com> wrote:
Hello Swift community,
The review of “SE-0107: UnsafeRawPointer API” begins now and runs through
July 4, 2016. The proposal is available here:
I've revised the proposal again based on extremely helpful feedback from
DaveA and Jordan.
This revision expands on the concept of formally binding the memory type
that I was recently working on with Dmitri. Now we can clearly define pre
and post conditions on memory operations and pointer casts that can be used
to prove the type safety. The model is now simpler, more complete, and easy
to reason about locally. This will help developers reason about correctness
and make it easy to implement a sanitizer that verifies the type safety of
UnsafePointer operations.
Adding safety to pointer "casts" made it possible for me to actually
simplify the allocation and initialization APIs. I think both camps,
convenience and safety, will be happy.
- All typed access to memory, whether via a typed pointer or regular
language construct, must be consistent with the memory's bound type
(the access type must be related to the bound type). Typed access
includes initialization, assignment, or deinitialization via a typed
pointer.
- Memory remains bound after being deinitialized.
- Memory is implicitly bound or rebound to a type by initializing it
via a raw pointer.
- A separate API now exists for explicity binding or rebinding memory
to a type. This allows binding to be decoupled from initialization
for convenience and efficiency. It also supports safe
interoperability between APIs that used different, but layout
compatible types.
- Using an API that accesses memory as a different type can now be
accomplished by rebinding the memory. This effectively changes the
type of any initialized values in memory. The compiler is still
permitted to assume strict aliasing for accesses on either side of
the operation that rebinds memory.
I want to call this out separately because it’s not specific to my proposal and changes the existing UnsafePointer API.
Brent’s suggestion is to change `initialize(from:count:)` to `initialize(from:forwardToCount:)` for symmetry with the backward operations. It makes perfect sense to me, and it does a better job of conveying that a sequence of elements will be read out of “from”.
Any objections?
// I'm not happy with the asymmetry of these forwards/backwards
// pairs.
//
// func initialize<T>(from: UnsafePointer<T>, count: Int)
// -> UnsafeMutablePointer<T>
// func initializeBackward<T>(from: UnsafePointer<T>, count: Int)
// -> UnsafeMutablePointer<T>
func initialize<T>(from: UnsafePointer<T>, forwardToCount: Int)
-> UnsafeMutablePointer<T>
func initialize<T>(from: UnsafePointer<T>, backwardFromCount: Int)
-> UnsafeMutablePointer<T>
// More detailed thoughts on redesigining `move` methods in the
// email, but for now:
//
// func moveInitialize<T>(from: UnsafePointer<T>, count: Int)
// -> UnsafeMutablePointer<T>
// func moveInitializeBackward<T>(from: UnsafePointer<T>, count: Int)
// -> UnsafeMutablePointer<T>
func moveInitialize<T>(from: UnsafePointer<T>, forwardToCount: Int)
-> UnsafeMutablePointer<T>
func moveInitialize<T>(from: UnsafePointer<T>, backwardFromCount: Int)
-> UnsafeMutablePointer<T>
Andy
···
On Jul 2, 2016, at 10:10 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:
"bind" is active. It has important side effects.
"assuming" is passive; it's just a "cast".
"withMemoryRebound" temporarily rebinds the type, but leaves it in its original state. "with" is sufficient to imply that the closure may mutate state.
···
On Jul 2, 2016, at 10:10 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:
---
// In addition to the above point, I don't think "to" belongs in
// front of "contiguous" here. It makes sense with the verbs in
// `load` and `storeRaw` below, but not with this verb.
//
// I also think `contiguous` basically implies `Index`, so we can
// just use `at`. Yes, I know about the `load` method with
// `atByteOffset`.
//
// func initialize<T>(toContiguous: T.Type, atIndex: Int, with: T)
// -> UnsafeMutablePointer<T>
func initialize<T>(contiguous: T.Type, at: Int, to: T)
-> UnsafeMutablePointer<T>
This is a purely additive API that I'm personally in favor of dropping if it's confusing, but it was requested as a convenience...
The "toContiguous" was to avoid implying that a sequence of elements is being initialized, as opposed to a single element. "atIndex" was also suggested as a clarification. In hindsight I don't think either are likely to improve clarity in practice. I'm inclined to go with your suggestion, and if anyone thinks it's confusing we should drop the API for now:
func initialize<T>(contiguous: T.Type, at: Int, to: T)
-> UnsafeMutablePointer<T>
---
// A little backwards, but `as` here is to support a more fluent
// `storeRaw` equivalent.
//
// func load<T>(_: T.Type) -> T
// func load<T>(_: T.Type, atByteOffset: Int) -> T
func load<T>(as: T.Type) -> T
func load<T>(fromByteOffset: Int, as: T.Type) -> T
// func load<T>(fromContiguous: T.Type, atIndex: Int) -> T
func load<T>(fromContiguous: T.Type, at: Int) -> T
// I'm reversing the arguments here because the thing being stored is
// the obvious direct object of the verb "store".
//
// func storeRaw<T>(_: T.Type, with: T)
// func storeRaw<T>(toContiguous: T.Type, at: Int, with: T)
func storeRaw<T>(_: T, as: T.Type)
func storeRaw<T>(_: T, toContiguous: T.Type, at: Int)
I agree. Giving `load` and "as" label makes sense because the in-memory type may be different than the loaded type.
* Is there a reason there's a `load` that takes a byte offset, but not a `storeRaw`?
A couple of those methods were added by request, but I was reluctant to add more methods just because they looked like they should be there.
But since you mention it, if we decide keep the purely additive `load(fromByteOffset:as)` method, then I’ll also add this for symmetry:
func storeRaw<T>(_: T, toByteOffset: Int)
Should it be "toByteOffset" or "atByteOffset”?
* I'm also a little nervous about the fact that `storeRaw` (and `load`?) is documented to only work properly on "trivial types", but it doesn't have any sort of constraints to ensure it's used correctly. (One could imagine, for instance, the compiler automatically conforming trivial types to a `Trivial` protocol.)
Noted. There's absolutely no way to enforce that any overwritten value is trivial, but a sanitizer could catch it. When we have a "trivial" protocol we can add a debugPrecondition on the destination type.
`load` simply does not need to take a trivial type. Although it reads from raw memory, it is not a "raw" operation. It knows how to retain things.
I'd rather not introduce a symmetric `store` that handles nontrivial types, until we really need it, because of the serious potential for misuse. People should try to use typed pointers for assignment semantics.
There's a discussion on this in the proposal now:
* I don't think I understand `initialize(toContiguous:atIndex:with:)`. Does it return a typed pointer to the whole buffer, or just the one instance it initialized? In the `stringFromBytes` example, shouldn't we either subscript the typed pointer from the previous `initialize(_:with:count:)` call, or call `storeRaw(toContiguous:atIndex:with:)`, rather than initializing memory twice? If this isn't a good use case for `initialize(toContiguous:atIndex:with:)`, what would be?
The latest proposal has this example, which actually ignores the returned value:
This was requested as a convenience. As mentioned in a previous email, I'm happy to drop it for now.
`initialize(toContiguous:with:count:)` returns a typed pointer to all the initialized elements.
Subscripting the typed pointer to write the null terminator would be wrong because that memory has never been bound to a type.
I do agree that the example should just be:
let rawBuffer = UnsafeMutableRawPointer.allocate(bytes: size + 1)
rawBuffer.initialize(UInt8.self, with: value, count: size)
rawBuffer.initialize(contiguous: UInt8.self, at: size, to: 0)
But the easy, common way to initialize a C string will simply be:
let cstr = UnsafeMutablePointer<CChar>.allocate(capacity: size + 1)
// The whole string is now bound to CChar
for i in 0..<size { cstr[i] = … }
cstr[size] = 0
I'm quite concerned by the "moveInitialize should be more elegant" section at the bottom.
Since the types are so close, `moveInitialize` could require mutating arguments and actually swap the pointers. For instance:
uninitializedBuffer.swapPointersAfterMoving(from: &buffer, count: count)
// `buffer` now points to the new allocation, filled in with the Ints.
// `uninitializedBuffer` now points to the old allocation, which is deinitialized.
uninitializedBuffer.deallocate()
return buffer
}
This is *such* a strange semantic, however, that I'm not at all sure how to name this function.
`moveAssign(from:count:)` could do something much simpler, returning a raw version of `from`:
target.moveAssign(from: source).deallocate()
`move()`, on the other hand, I don't see a good way to fix like this.
One ridiculous thing we could do for `moveAssign(from:count:)` and perhaps `move()` is to deliberately make `self` invalid by setting it to address 0. If it were `Optional`, this would nil `self`. If it weren't...well, something would probably fail eventually.
For a while, I was trying to force a convention where deinitialization always returned a raw pointer because it's safer to initialize that
raw pointer. With the latest proposal I'm not as concerned about that. The majority of the time, it will be fine to reinitialize using the typed pointer. If the user wants a raw pointer back after the move, it is trivial just to cast the typed pointer into a raw pointer.
So, while move-semantics would be cool, I really don’t think it’s necessary or even desired in this case. Clear doc comments should be sufficient.
* * *
I notice that many APIs require type parameters merely to force the user to explicitly state the types involved. I wonder if we could instead introduce an attribute which you could place on a parameter or return type indicating that there must be an explicit `as` cast specifying its type:
rawPointer.storeRaw(3 as Int)
rawPointer.load() as Int
rawPointer.cast() as UnsafePointer<Int>
This would also be useful on `unsafeBitCast`, and on user APIs which are prone to type inference issues.
I'm not as irrated by explicit type arguments as some, but the feeling I get is that we really want a language feature that forces certain
generic paramters to be explicit. When that happens, we'll likely phase out the old-style type arguments in favor of angle brackets, and
I'll be sad because I dislike angle brackets.
* * *
In the long run, however, I wonder if we might end up removing `UnsafeRawPointer`. If `Never` becomes a subtype-of-all-types, then `UnsafePointer<Never>` would gain the basic properties of an `UnsafeRawPointer`:
* Because `Never` is a subtype of all types, `UnsafePointer<Never>` could alias any other pointer.
* Accessing `pointee` would be inherently invalid (it would either take or return a `Never`), and APIs which initialize or set `pointee` would be inherently uncallable.
* `Never` has no intrinsic size, so it could be treated as having a one-byte size, allowing APIs which normally allocate, deallocate, or do pointer arithmetic by instance size to automatically do so by byte size instead.
* APIs for casting an `UnsafePointer<T>` to `UnsafePointer<supertype of T>` or `<subtype of T>` would do the right thing with `UnsafePointer<Never>`.
Thus, I could imagine `Unsafe[Mutable]RawPointer` becoming `Unsafe[Mutable]Pointer<Never>` in the future, with some APIs being generalized and moving to all `UnsafePointer`s while others are in extensions on `UnsafePointer where Pointee == Never`.
It might be worth taking a look at the current API designs and thinking about how they would look in that world:
* Is `nsStringPtr.casting(to: UnsafePointer<NSObject>)` how you would want to write a pointee upcast? How about `UnsafePointer<NSString>(nsObjectPtr)` for a pointee downcast?
* Would you want `initialize<T>(_: T.Type, with: T, count: Int = 1) -> UnsafeMutablePointer<T>` in the `Never` extension, or (with a supertype-of-Pointee constraint on `T`) would it be something you'd put on other `UnsafeMutablePointer`s too? What does that mean for `UnsafeMutablePointer.initialize(with:)`?
* Are `load` or `storeRaw` things that might make sense on any `UnsafeMutablePointer` if they were constrained to supertypes only?
* Are there APIs which are basically the same on `Unsafe[Mutable]Pointer`s and their `Raw` equivalents, except that the `Raw` versions are "dumb" because they don't know what type they're operating on? If so, should they be given the same name?
Very early on I considered a special `Never` element type for all of the excellent reasons that you laid out (nice job explaining that), but the pointer conversion rules that we want are not implementable.
Since then, the proposal has evolved so much that it makes sense to have a nominal raw pointer type. The pointer type itself is distinctly different, not just the element type, and the type system needs to be aware of that. It's also critical that the raw and typed pointers have a distinct API. Moving both of their functionality into extensions would just be a workaround. In reality, since the semantics are different, there's almost no shared implementation.
In short, raw pointers are deliberately a different types and we want developers and APIs to be cognizant of that.
-Andy
···
On Jul 2, 2016, at 10:10 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:
It wouldn't be ridiculous at all to do this in debug builds. I would
consider whether there's a better “trap” value than 0 that we could
write in there.
···
on Sat Jul 02 2016, Brent Royal-Gordon <swift-evolution@swift.org> wrote:
One ridiculous thing we could do for `moveAssign(from:count:)` and
perhaps `move()` is to deliberately make `self` invalid by setting it
to address 0. If it were `Optional`, this would nil `self`. If it
weren't...well, something would probably fail eventually.
The examples in the "Motivation" section are contrived to demonstrate safety holes as tersely as possible. The examples in "Expected use
cases" are prototypical use cases that I expect most uses to fall under.
I think APIs for marshalling should evolve toward higher abstraction and safety. For starters, they should probably use UnsafeBufferPointer more often than UnsafePointer. But those APIs need time to evolve. The important thing for this release is to settle on the source-breaking changes to UnsafePointer and establish clear, verifiable rules for safety at the lowest level of the API.
One of the conundrums of this proposal is that well designed APIs should not need to use UnsafeRawPointer, but it does need to exist as the sanctioned way to work around limitations of UnsafePointer. If UnsafePointer is the *only* way to directly access a memory buffer, then it's natural for users to assume that they can use it as a valid workaround for all the messy interoperability, marshalling, and low-level buffer access problem that they face in practice. The situation today is that the UnsafePointer API encourages that misuse, and we possibly miscompile the code without warning.
-Andy
···
On Jul 5, 2016, at 10:09 AM, Magnus Ahltorp via swift-evolution <swift-evolution@swift.org> wrote:
The review of “SE-0107: UnsafeRawPointer API” begins now and runs through July 4, 2016. The proposal is available here:
I have not had time for a full review, but I have a question about the proposal:
When glancing at the examples, they strike me as mostly being marshalling, which in my opinion would be better served by a safe marshalling API followed by unsafe handling of the resulting buffer, and vice versa for unmarshalling. I think it is very important (in the long run) that code that doesn't interact with C directly has safe ways of doing inherently safe operations, and not take the unsafe route just because that is the only API available.
My question is, how does this API fit into the bigger picture of marshalling, and what are the benefits of using this API instead of marshalling with safe buffers?
I would consider "we don't have time, we have to do this for now" a perfectly valid answer.
I'm revising this proposal based on last week's feedback. A few of the
additive APIs are removed and a number of UnsafePointer and
UnsafeRawPointer methods are renamed.
Here is a PR for the revision. Note that the examples in the proposal
text still need to be updated:
---
I should preemptively answer the question "why do UnsafeRawPointer
methods take an explicit type argument when it can be inferred?". Such
as:
rawPtr.initializeMemory(as: Int.self, ...)
These methods don't simply operate on values of some type, they
actively bind memory to that type. At the point of use, that type
needs to be explicit to convey that fact. It's important for
readability, comprehension, and correctness. We cannot rely on type
inferrence on some expression which can change without the original
author's intervention resulting in subtle miscompiles.
---
The only concern I have about this version of the proposal is this method name:
* What is your evaluation of the proposal?
-1. It seems like the author is trying to solve a non-problem. The pointer APIs already work very well with enough due warning that they are unsafe. This change will just make them even more cumbersome to work with.
* Is the problem being addressed significant enough to warrant a change to Swift?
No, I don’t think so, and if it were it would be too significant to be considered in scope for Swift 3.
* Does this proposal fit well with the feel and direction of Swift?
It’s a valid Swift API, I just feel it makes the UnsafePointer family more complex than it needs to be.
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
Swift’s approach to memory access has been unique in my programming experience and I like it for its power and simplicity.
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
I read most of the proposal and have done a lot of work with the existing pointer APIs. Again I feel this is adding needless complexity and doesn’t actually change what you can do with the API.
Thanks for voicing an opinion because I suspect more people feel this way than are going to express openly, and I want to respond to all those people. I anticipate that a lot of users are going to feel this is more cumbersome and redundant. In fact, I don't expect the API to be easier in most cases, but I do expect it to be much harder to use incorrectly.
The proposal includes some use cases that developers are currently using the UnsafePointer API for but are in fact impossible to implement correctly unless we provide some alternate API. This proposed API evolved over a very long period and many attempts to propose some alternate API, none of which were easy to explain to users. Given my goal of eliminating undefined behavior from Swift code, I'd like to hear how this can be accomplished in a better way. In particular, should we allow UnsafePointer casts at all? How can the user communicate to the compiler that the same memory location is accessed as a different type, particularly when so many users are accustomed to getting away with this in C?
One more point here. The proposed low-level UnsafeRawPointer API is painfully explicit and seemingly redundant when it comes to writing type names in many places. This is intentional. The initial goal is to make the primitive API's difficult to misuse, but that is not the end goal.
Eventually we will provide convenience APIs on top of these primitive APIs. For example, I expect to provide convenience APIs for safely allocating and initializing memory in one step. I also expect to make better use of the UnsafeBufferPointer API in the future.
However, these convenience APIs are purely additive and it's important that we move forward with source-breaking changes to the primitive API even if some common idioms become more verbose.
-Andy
···
On Jun 29, 2016, at 11:38 AM, Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:
On Jun 29, 2016, at 8:02 AM, Brad Hilton via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
Hello Swift community,
The review of “SE-0107: UnsafeRawPointer API” begins now and runs through July 4, 2016. The proposal is available here:
or, if you would like to keep your feedback private, directly to the review manager.
What goes into a review?
The goal of the review process is to improve the proposal under review through constructive criticism and contribute to the direction of Swift. When writing your review, here are some questions you might want to answer in your review:
* What is your evaluation of the proposal?
* Is the problem being addressed significant enough to warrant a change to Swift?
* Does this proposal fit well with the feel and direction of Swift?
* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
More information about the Swift evolution process is available at