Proposal: [stdlib] Remove withUnsafe[Mutable]Pointer[s]()

Is there a reason that this couldn't be made to work?

    let ptr: UnsafePointer<Void> = &x

Jacob

···

On Wed, Dec 16, 2015 at 2:16 PM, Michael Gottesman via swift-evolution < swift-evolution@swift.org> wrote:

> On Dec 16, 2015, at 4:07 PM, Michael Gottesman via swift-evolution < > swift-evolution@swift.org> wrote:
>
>
>> On Dec 16, 2015, at 2:22 PM, Dave Abrahams <dabrahams@apple.com> wrote:
>>
>>
>>> On Dec 16, 2015, at 11:54 AM, Michael Gottesman via swift-evolution < > swift-evolution@swift.org> wrote:
>>>
>>>>
>>>> On Dec 16, 2015, at 1:49 PM, Kevin Ballard via swift-evolution < > swift-evolution@swift.org> wrote:
>>>>
>>>> Another replacement for withUnsafe[Mutable]Pointer is declaring a
nested function of the appropriate type (this is equivalent to the
anonymous closure, but perhaps more readable):
>>>>
>>>> func foo(ptr: UnsafePointer<Int>) {
>>>> // ...
>>>> }
>>>> foo(&x)
>>>>
>>>> -Kevin
>>>>
>>>> On Wed, Dec 16, 2015, at 11:38 AM, Kevin Ballard wrote:
>>>>> # Introduction
>>>>>
>>>>> The stdlib provides functions withUnsafePointer() and
withUnsafeMutablePointer() (and plural variants) that take an inout
reference and call a block with the UnsafePointer/UnsafeMutablePointer
created from the reference.
>>>>>
>>>>> # Problem
>>>>>
>>>>> withUnsafePointer() can only be used with mutable variables, because
those are the only things that can be used with inout &refs. Both functions
are also fairly useless, as &x refs can be passed directly to functions
taking an UnsafePointer or UnsafeMutablePointer. The existence of the
functions mostly just causes people to think they're necessary when they're
not. The provide no functionality that passing &x refs directly to the
functions taking a pointer doesn't already fulfill.
>>>>>
>>>>> # Solution
>>>>>
>>>>> Remove the functions from the stdlib. The Swift Book should also be
updated to talk about passing an &x ref to a function that takes an
UnsafePointer or UnsafeMutablePointer (but of course changes to the book
are not covered by the open-source project). Most uses of these functions
can probably be replaced with a &x ref directly. If any can't, they could
be replaced with the following equivalent expressions:
>>>>>
>>>>> { (ptr: UnsafePointer<Int>) in
>>>>> // ...
>>>>> }(&x)
>>>>>
>>>>> or:
>>>>>
>>>>> withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
>>>>> // ...
>>>>> }
>>>
>>> One thing to keep in mind here is that with*Pointer and friends is
also meant to enable one to work around issues with the optimizer if they
come up in a convenient manner. I.e. imagine if one is attempting to
process an image using a 5d array and for whatever reason, you are not
getting the performance you need. Hopefully you would file a bug report and
then use with*Pointer for your image processing loop.
>>>
>>> My fear about withExtendedLifetime is that the name is a misnomer. You
are not extending the lifetime.
>>
>> What makes you say that?
>
> Let me be more specific.
>
> My issue with the name 'withExtendedLifetime' is that it is suggestive
that the lifetime of &x is being extended in a way that is different from
if one just passed off &x to any old function. In reality though, nothing
special is happening here implying that the name is misleading. A better
name IMO would be something that drops any such implication.

For example, we could just use 'with' (something that has been suggested
for this use case in various blog posts). I.e.:

with(&x) { (ptr: UnsafePointer<Int>) in
   ...
}

Michael

>
> Michael
>
>>
>> If in fact it is true, shouldn't you file a bug report?
>>
>> -Dave
>>
>>
>>
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

'x' may not have storage at all, either because it fits entirely in registers or because it's actually a computed property. So this would have to synthesize storage, presumably on the stack. It's unclear when it's safe to destroy that storage, too.

Limiting inout-to-pointer conversions to a single call gives a clear start and end for where the pointer is valid, which allows the compiler to do correct cleanup.

Jordan

···

On Dec 16, 2015, at 14:25 , Jacob Bandes-Storch via swift-evolution <swift-evolution@swift.org> wrote:

Is there a reason that this couldn't be made to work?
    let ptr: UnsafePointer<Void> = &x

Jacob

On Wed, Dec 16, 2015 at 2:16 PM, Michael Gottesman via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

> On Dec 16, 2015, at 4:07 PM, Michael Gottesman via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
>
>
>> On Dec 16, 2015, at 2:22 PM, Dave Abrahams <dabrahams@apple.com <mailto:dabrahams@apple.com>> wrote:
>>
>>
>>> On Dec 16, 2015, at 11:54 AM, Michael Gottesman via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
>>>
>>>>
>>>> On Dec 16, 2015, at 1:49 PM, Kevin Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
>>>>
>>>> Another replacement for withUnsafe[Mutable]Pointer is declaring a nested function of the appropriate type (this is equivalent to the anonymous closure, but perhaps more readable):
>>>>
>>>> func foo(ptr: UnsafePointer<Int>) {
>>>> // ...
>>>> }
>>>> foo(&x)
>>>>
>>>> -Kevin
>>>>
>>>> On Wed, Dec 16, 2015, at 11:38 AM, Kevin Ballard wrote:
>>>>> # Introduction
>>>>>
>>>>> The stdlib provides functions withUnsafePointer() and withUnsafeMutablePointer() (and plural variants) that take an inout reference and call a block with the UnsafePointer/UnsafeMutablePointer created from the reference.
>>>>>
>>>>> # Problem
>>>>>
>>>>> withUnsafePointer() can only be used with mutable variables, because those are the only things that can be used with inout &refs. Both functions are also fairly useless, as &x refs can be passed directly to functions taking an UnsafePointer or UnsafeMutablePointer. The existence of the functions mostly just causes people to think they're necessary when they're not. The provide no functionality that passing &x refs directly to the functions taking a pointer doesn't already fulfill.
>>>>>
>>>>> # Solution
>>>>>
>>>>> Remove the functions from the stdlib. The Swift Book should also be updated to talk about passing an &x ref to a function that takes an UnsafePointer or UnsafeMutablePointer (but of course changes to the book are not covered by the open-source project). Most uses of these functions can probably be replaced with a &x ref directly. If any can't, they could be replaced with the following equivalent expressions:
>>>>>
>>>>> { (ptr: UnsafePointer<Int>) in
>>>>> // ...
>>>>> }(&x)
>>>>>
>>>>> or:
>>>>>
>>>>> withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
>>>>> // ...
>>>>> }
>>>
>>> One thing to keep in mind here is that with*Pointer and friends is also meant to enable one to work around issues with the optimizer if they come up in a convenient manner. I.e. imagine if one is attempting to process an image using a 5d array and for whatever reason, you are not getting the performance you need. Hopefully you would file a bug report and then use with*Pointer for your image processing loop.
>>>
>>> My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.
>>
>> What makes you say that?
>
> Let me be more specific.
>
> My issue with the name 'withExtendedLifetime' is that it is suggestive that the lifetime of &x is being extended in a way that is different from if one just passed off &x to any old function. In reality though, nothing special is happening here implying that the name is misleading. A better name IMO would be something that drops any such implication.

For example, we could just use 'with' (something that has been suggested for this use case in various blog posts). I.e.:

with(&x) { (ptr: UnsafePointer<Int>) in
   ...
}

Michael

>
> Michael
>
>>
>> If in fact it is true, shouldn't you file a bug report?
>>
>> -Dave
>>
>>
>>
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org <mailto:swift-evolution@swift.org>
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Yes. That breaks several parts of the language:

1. Swift semantics allow you to use computed properties and stored
   properties interchangeably. That expression there can't work with
   computed properties unless Swift silently creates a temporary that
   lives for the entire scope, which would be surprising behavior.
2. Swift also aggressively discards values that are no longer used even
   if they're still in scope (at least under optimization; it likely
   keeps them around in debug builds so they can be inspected). But
   there's no way for Swift to know how long that pointer is going to be
   used for (it could track that variable itself, but what about derived
   values?). So any value that's used in that expression would also have
   to have its lifetime extended for the entire scope, which is
   surprising behavior.
3. The pointer definitely cannot be valid outside of the current scope,
   but there's nothing in that statement to imply this.
   withUnsafePointer() taking a scope tells the user that the pointer is
   only valid for the scope (otherwise it would just return the
   pointer), and similarly the current &x behavior can only be used as a
   parameter to a function, which tells the user that it's only valid
   for the duration of that function call.

-Kevin Ballard

···

On Wed, Dec 16, 2015, at 02:25 PM, Jacob Bandes-Storch via swift-evolution wrote:

Is there a reason that this couldn't be made to work? let ptr:
UnsafePointer<Void> = &x

Yeah, that could be made to work, and give an error if `x` isn't known to be a stored property.

-Joe

···

On Dec 16, 2015, at 2:25 PM, Jacob Bandes-Storch via swift-evolution <swift-evolution@swift.org> wrote:

Is there a reason that this couldn't be made to work?
    let ptr: UnsafePointer<Void> = &x

If 'ptr' is a local variable, then the scope of 'ptr' could potentially be used as the lifetime of the referenced memory. That's still not perfect if 'ptr' is escaped and you expect it to remain valid for the longer duration of `x`, though.

-Joe

···

On Dec 16, 2015, at 2:38 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:

'x' may not have storage at all, either because it fits entirely in registers or because it's actually a computed property. So this would have to synthesize storage, presumably on the stack. It's unclear when it's safe to destroy that storage, too.

Limiting inout-to-pointer conversions to a single call gives a clear start and end for where the pointer is valid, which allows the compiler to do correct cleanup.

Jordan

On Dec 16, 2015, at 14:25 , Jacob Bandes-Storch via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Is there a reason that this couldn't be made to work?
    let ptr: UnsafePointer<Void> = &x

Jacob

On Wed, Dec 16, 2015 at 2:16 PM, Michael Gottesman via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

> On Dec 16, 2015, at 4:07 PM, Michael Gottesman via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
>
>
>> On Dec 16, 2015, at 2:22 PM, Dave Abrahams <dabrahams@apple.com <mailto:dabrahams@apple.com>> wrote:
>>
>>
>>> On Dec 16, 2015, at 11:54 AM, Michael Gottesman via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
>>>
>>>>
>>>> On Dec 16, 2015, at 1:49 PM, Kevin Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
>>>>
>>>> Another replacement for withUnsafe[Mutable]Pointer is declaring a nested function of the appropriate type (this is equivalent to the anonymous closure, but perhaps more readable):
>>>>
>>>> func foo(ptr: UnsafePointer<Int>) {
>>>> // ...
>>>> }
>>>> foo(&x)
>>>>
>>>> -Kevin
>>>>
>>>> On Wed, Dec 16, 2015, at 11:38 AM, Kevin Ballard wrote:
>>>>> # Introduction
>>>>>
>>>>> The stdlib provides functions withUnsafePointer() and withUnsafeMutablePointer() (and plural variants) that take an inout reference and call a block with the UnsafePointer/UnsafeMutablePointer created from the reference.
>>>>>
>>>>> # Problem
>>>>>
>>>>> withUnsafePointer() can only be used with mutable variables, because those are the only things that can be used with inout &refs. Both functions are also fairly useless, as &x refs can be passed directly to functions taking an UnsafePointer or UnsafeMutablePointer. The existence of the functions mostly just causes people to think they're necessary when they're not. The provide no functionality that passing &x refs directly to the functions taking a pointer doesn't already fulfill.
>>>>>
>>>>> # Solution
>>>>>
>>>>> Remove the functions from the stdlib. The Swift Book should also be updated to talk about passing an &x ref to a function that takes an UnsafePointer or UnsafeMutablePointer (but of course changes to the book are not covered by the open-source project). Most uses of these functions can probably be replaced with a &x ref directly. If any can't, they could be replaced with the following equivalent expressions:
>>>>>
>>>>> { (ptr: UnsafePointer<Int>) in
>>>>> // ...
>>>>> }(&x)
>>>>>
>>>>> or:
>>>>>
>>>>> withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
>>>>> // ...
>>>>> }
>>>
>>> One thing to keep in mind here is that with*Pointer and friends is also meant to enable one to work around issues with the optimizer if they come up in a convenient manner. I.e. imagine if one is attempting to process an image using a 5d array and for whatever reason, you are not getting the performance you need. Hopefully you would file a bug report and then use with*Pointer for your image processing loop.
>>>
>>> My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.
>>
>> What makes you say that?
>
> Let me be more specific.
>
> My issue with the name 'withExtendedLifetime' is that it is suggestive that the lifetime of &x is being extended in a way that is different from if one just passed off &x to any old function. In reality though, nothing special is happening here implying that the name is misleading. A better name IMO would be something that drops any such implication.

For example, we could just use 'with' (something that has been suggested for this use case in various blog posts). I.e.:

with(&x) { (ptr: UnsafePointer<Int>) in
   ...
}

Michael

>
> Michael
>
>>
>> If in fact it is true, shouldn't you file a bug report?
>>
>> -Dave
>>
>>
>>
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org <mailto:swift-evolution@swift.org>
> https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
// ...
}

withExtendedLifetime doesn't have this overload. Adding it would essentially be just renaming withUnsafePointer.

No overload necessary. As I said, it already works today (tested in Swift 2.1). The generic type T is resolved to UnsafePointer<Int>, and so you can pass &x into it just fine because you're passing it to a parameter of type UnsafePointer.

That's interesting emergent behavior, but it's not something we anticipated withUnsafePointer being used for.

# Detailed Solution

The functions would be marked as unavailable with a message saying to pass &x directly to the function taking a pointer. If it's feasible to do so, Fix-Its would be introduced for the trivial case of the pointer being used once in the closure as a parameter to a function.

The documentation of UnsafePointer and UnsafeMutablePointer would be edited to include a mention of how &x refs can be passed to functions expecting an UnsafePointer or UnsafeMutablePointer. This way anyone new to the language who encounters those types in the wild will be able to easily see how to create one from a variable.

I'm against removing the functionality. The magic pointer conversions were only ever intended to make interop with well-behaved C functions easier; it was in my mind that we would eventually constrain the magic pointer conversions to only apply to imported APIs. withUnsafePointer is necessary more often than you think, since the current semantics of the conversion only keep the pointer valid for one call—you can't call a conversion constructor or otherwise touch the pointer in between. We should fix that, but it'll still be necessary to persist a pointer across multiple C calls that expect pointer identity to be maintained.

Why constrain the magic pointer conversions?

In general, implicit conversions are bad. These particular conversions lead to lots of gross knock-on effects, things like [1] - [2] working by taking the difference of two transient pointers.

We should discourage people from writing Swift functions that take UnsafePointers, but sometimes it is appropriate (e.g. when wrapping imported C functions, if you need a stable pointer or need to do pointer math, then your function should be taking Unsafe[Mutable]Pointer instead of taking an inout parameter).

If you need a stable pointer internally in your implementation, I don't see a reason to expose that implementation detail to users. You could use withUnsafeMutablePointer(&inoutParameter) internally.

And if you really need to preserve identity across multiple function calls, I'd argue that after a certain point, it's better to allocate the memory yourself with UnsafeMutablePointer.alloc/dealloc (with optimizer support for promoting small, scoped .alloc/.dealloc pairs to stack allocations), since interfacing compiler-managed properties with manual memory manipulation is always going to be finicky.

I would suggest the use of ManagedBuffer over UnsafeMutablePointer.alloc/dealloc for safety reasons, FWIW.

-Joe
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

-Dave

···

On Dec 16, 2015, at 1:24 PM, Joe Groff via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 16, 2015, at 1:20 PM, Joe Groff <jgroff@apple.com <mailto:jgroff@apple.com>> wrote:

On Dec 16, 2015, at 1:11 PM, Kevin Ballard <kevin@sb.org <mailto:kevin@sb.org>> wrote:
On Wed, Dec 16, 2015, at 11:58 AM, Joe Groff wrote:

On Dec 16, 2015, at 11:38 AM, Kevin Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

However, in the face of resilience, that could only be known if x were explicitly declared to be fragile, or within the current module.

-Chris

···

On Dec 16, 2015, at 2:40 PM, Joe Groff via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 16, 2015, at 2:25 PM, Jacob Bandes-Storch via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Is there a reason that this couldn't be made to work?
    let ptr: UnsafePointer<Void> = &x

Yeah, that could be made to work, and give an error if `x` isn't known to be a stored property.

The pointer conversions are already poorly discoverable, and people don't know they exist since code completion doesn't make it apparent. Showing the APIs as overloaded taking 'inout's or something similar would make it more apparent how you can use them.

I'm hoping that adding documentation to UnsafePointer/UnsafeMutablePointer that mentions the conversions will help a lot, and ideally the book would mention this too (presumably as a note in the section on in-out parameters). If I didn't already know about the conversion and I encountered a function expecting an UnsafePointer/UnsafeMutablePointer, my first inclination is to type "UnsafePointer" (or "UnsafeMutablePointer") into the documentation browser and read the results. This is why I don't like withUnsafe[Mutable]Pointer[s]() existing, because it makes people think that's what you have to use, when in most cases the &x conversion is simpler.

It's my hope that better SDK annotations can make these conversions less necessary too, by letting us eventually import well-behaved in/inout/out arguments directly as value arguments/inouts/extra returns.

That would be nice.

>> We added them because they *are* necessary.
>
> You still haven't demonstrated that. Every single use of withUnsafe[Mutable]Pointer[s] that can't be handled with the implicit &x conversion can be replaced with a nested function, a call to an anonymous closure, or an (ab)use of withExtendedLifetime. And the existence of the functions encourages people to call them when a simple &x ref would have sufficed.

I guess it's a difference of perspective. Your alternatives hinge on blessing the pointer conversions as a core part of the language, rather than an affordance to make many C APIs not totally awful, which I'm not comfortable with.

I guess it is. From my perspective they're already a core part of the language.

-Kevin

···

On Wed, Dec 16, 2015, at 01:38 PM, Joe Groff wrote:

They are documented fairly extensively in "Using Swift with Cocoa and Objective-C" (since they exist to interop with C):

-Joe

···

On Dec 16, 2015, at 1:51 PM, Kevin Ballard <kevin@sb.org> wrote:

On Wed, Dec 16, 2015, at 01:38 PM, Joe Groff wrote:

The pointer conversions are already poorly discoverable, and people don't know they exist since code completion doesn't make it apparent. Showing the APIs as overloaded taking 'inout's or something similar would make it more apparent how you can use them.

I'm hoping that adding documentation to UnsafePointer/UnsafeMutablePointer that mentions the conversions will help a lot, and ideally the book would mention this too (presumably as a note in the section on in-out parameters). If I didn't already know about the conversion and I encountered a function expecting an UnsafePointer/UnsafeMutablePointer, my first inclination is to type "UnsafePointer" (or "UnsafeMutablePointer") into the documentation browser and read the results. This is why I don't like withUnsafe[Mutable]Pointer[s]() existing, because it makes people think that's what you have to use, when in most cases the &x conversion is simpler.