Proposal: [stdlib] Remove withUnsafe[Mutable]Pointer[s]()

[Lily_Ballard]

# Introduction

The stdlib provides functions withUnsafePointer() and withUnsafeMutablePointer() (and plural variants) that take an inout reference and call a block with the UnsafePointer/UnsafeMutablePointer created from the reference.

# Problem

withUnsafePointer() can only be used with mutable variables, because those are the only things that can be used with inout &refs. Both functions are also fairly useless, as &x refs can be passed directly to functions taking an UnsafePointer or UnsafeMutablePointer. The existence of the functions mostly just causes people to think they're necessary when they're not. The provide no functionality that passing &x refs directly to the functions taking a pointer doesn't already fulfill.

# Solution

Remove the functions from the stdlib. The Swift Book should also be updated to talk about passing an &x ref to a function that takes an UnsafePointer or UnsafeMutablePointer (but of course changes to the book are not covered by the open-source project). Most uses of these functions can probably be replaced with a &x ref directly. If any can't, they could be replaced with the following equivalent expressions:

{ (ptr: UnsafePointer<Int>) in
    // ...
}(&x)

or:

withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
    // ...
}

# Detailed Solution

The functions would be marked as unavailable with a message saying to pass &x directly to the function taking a pointer. If it's feasible to do so, Fix-Its would be introduced for the trivial case of the pointer being used once in the closure as a parameter to a function.

The documentation of UnsafePointer and UnsafeMutablePointer would be edited to include a mention of how &x refs can be passed to functions expecting an UnsafePointer or UnsafeMutablePointer. This way anyone new to the language who encounters those types in the wild will be able to easily see how to create one from a variable.

[Lily_Ballard]

Another replacement for withUnsafe[Mutable]Pointer is declaring a nested function of the appropriate type (this is equivalent to the anonymous closure, but perhaps more readable):

func foo(ptr: UnsafePointer<Int>) {
    // ...
}
foo(&x)

-Kevin

···

On Wed, Dec 16, 2015, at 11:38 AM, Kevin Ballard wrote:

# Introduction

The stdlib provides functions withUnsafePointer() and withUnsafeMutablePointer() (and plural variants) that take an inout reference and call a block with the UnsafePointer/UnsafeMutablePointer created from the reference.

# Problem

withUnsafePointer() can only be used with mutable variables, because those are the only things that can be used with inout &refs. Both functions are also fairly useless, as &x refs can be passed directly to functions taking an UnsafePointer or UnsafeMutablePointer. The existence of the functions mostly just causes people to think they're necessary when they're not. The provide no functionality that passing &x refs directly to the functions taking a pointer doesn't already fulfill.

# Solution

Remove the functions from the stdlib. The Swift Book should also be updated to talk about passing an &x ref to a function that takes an UnsafePointer or UnsafeMutablePointer (but of course changes to the book are not covered by the open-source project). Most uses of these functions can probably be replaced with a &x ref directly. If any can't, they could be replaced with the following equivalent expressions:

{ (ptr: UnsafePointer<Int>) in
    // ...
}(&x)

or:

withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
    // ...
}

# Detailed Solution

The functions would be marked as unavailable with a message saying to pass &x directly to the function taking a pointer. If it's feasible to do so, Fix-Its would be introduced for the trivial case of the pointer being used once in the closure as a parameter to a function.

The documentation of UnsafePointer and UnsafeMutablePointer would be edited to include a mention of how &x refs can be passed to functions expecting an UnsafePointer or UnsafeMutablePointer. This way anyone new to the language who encounters those types in the wild will be able to easily see how to create one from a variable.

[Joe_Groff]

# Introduction

The stdlib provides functions withUnsafePointer() and withUnsafeMutablePointer() (and plural variants) that take an inout reference and call a block with the UnsafePointer/UnsafeMutablePointer created from the reference.

# Problem

withUnsafePointer() can only be used with mutable variables, because those are the only things that can be used with inout &refs. Both functions are also fairly useless, as &x refs can be passed directly to functions taking an UnsafePointer or UnsafeMutablePointer. The existence of the functions mostly just causes people to think they're necessary when they're not. The provide no functionality that passing &x refs directly to the functions taking a pointer doesn't already fulfill.

# Solution

Remove the functions from the stdlib. The Swift Book should also be updated to talk about passing an &x ref to a function that takes an UnsafePointer or UnsafeMutablePointer (but of course changes to the book are not covered by the open-source project). Most uses of these functions can probably be replaced with a &x ref directly. If any can't, they could be replaced with the following equivalent expressions:

{ (ptr: UnsafePointer<Int>) in
   // ...
}(&x)

or:

withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
   // ...
}

withExtendedLifetime doesn't have this overload. Adding it would essentially be just renaming withUnsafePointer.

# Detailed Solution

The functions would be marked as unavailable with a message saying to pass &x directly to the function taking a pointer. If it's feasible to do so, Fix-Its would be introduced for the trivial case of the pointer being used once in the closure as a parameter to a function.

The documentation of UnsafePointer and UnsafeMutablePointer would be edited to include a mention of how &x refs can be passed to functions expecting an UnsafePointer or UnsafeMutablePointer. This way anyone new to the language who encounters those types in the wild will be able to easily see how to create one from a variable.

I'm against removing the functionality. The magic pointer conversions were only ever intended to make interop with well-behaved C functions easier; it was in my mind that we would eventually constrain the magic pointer conversions to only apply to imported APIs. withUnsafePointer is necessary more often than you think, since the current semantics of the conversion only keep the pointer valid for one call—you can't call a conversion constructor or otherwise touch the pointer in between. We should fix that, but it'll still be necessary to persist a pointer across multiple C calls that expect pointer identity to be maintained. Your proposed alternatives all involve essentially reimplementing withUnsafePointer in a different way.

-Joe

···

On Dec 16, 2015, at 11:38 AM, Kevin Ballard via swift-evolution <swift-evolution@swift.org> wrote:

[Lily_Ballard]

>
> withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
> // ...
> }

withExtendedLifetime doesn't have this overload. Adding it would essentially be just renaming withUnsafePointer.

No overload necessary. As I said, it already works today (tested in Swift 2.1). The generic type T is resolved to UnsafePointer<Int>, and so you can pass &x into it just fine because you're passing it to a parameter of type UnsafePointer.

> # Detailed Solution
>
> The functions would be marked as unavailable with a message saying to pass &x directly to the function taking a pointer. If it's feasible to do so, Fix-Its would be introduced for the trivial case of the pointer being used once in the closure as a parameter to a function.
>
> The documentation of UnsafePointer and UnsafeMutablePointer would be edited to include a mention of how &x refs can be passed to functions expecting an UnsafePointer or UnsafeMutablePointer. This way anyone new to the language who encounters those types in the wild will be able to easily see how to create one from a variable.

I'm against removing the functionality. The magic pointer conversions were only ever intended to make interop with well-behaved C functions easier; it was in my mind that we would eventually constrain the magic pointer conversions to only apply to imported APIs. withUnsafePointer is necessary more often than you think, since the current semantics of the conversion only keep the pointer valid for one call—you can't call a conversion constructor or otherwise touch the pointer in between. We should fix that, but it'll still be necessary to persist a pointer across multiple C calls that expect pointer identity to be maintained.

Why constrain the magic pointer conversions? We should discourage people from writing Swift functions that take UnsafePointers, but sometimes it is appropriate (e.g. when wrapping imported C functions, if you need a stable pointer or need to do pointer math, then your function should be taking Unsafe[Mutable]Pointer instead of taking an inout parameter).

Your proposed alternatives all involve essentially reimplementing withUnsafePointer in a different way.

Well yeah, but without requiring anything in the stdlib. swiftdoc.org lists 36 global Swift functions. 4 of them are unsafe[Mutable]Pointer[s]. That's 11% of the global functions in the stdlib, for functionality that can already be accomplished without any stdlib support and which is relatively rare.

The way I see it, if the stdlib didn't already have these functions, we wouldn't add them. So why keep them around? All they do is make people think they're necessary.

-Kevin Ballard

···

On Wed, Dec 16, 2015, at 11:58 AM, Joe Groff wrote:

> On Dec 16, 2015, at 11:38 AM, Kevin Ballard via swift-evolution <swift-evolution@swift.org> wrote:

[Michael_Gottesman]

Another replacement for withUnsafe[Mutable]Pointer is declaring a nested function of the appropriate type (this is equivalent to the anonymous closure, but perhaps more readable):

func foo(ptr: UnsafePointer<Int>) {
   // ...
}
foo(&x)

-Kevin

# Introduction

The stdlib provides functions withUnsafePointer() and withUnsafeMutablePointer() (and plural variants) that take an inout reference and call a block with the UnsafePointer/UnsafeMutablePointer created from the reference.

# Problem

withUnsafePointer() can only be used with mutable variables, because those are the only things that can be used with inout &refs. Both functions are also fairly useless, as &x refs can be passed directly to functions taking an UnsafePointer or UnsafeMutablePointer. The existence of the functions mostly just causes people to think they're necessary when they're not. The provide no functionality that passing &x refs directly to the functions taking a pointer doesn't already fulfill.

# Solution

Remove the functions from the stdlib. The Swift Book should also be updated to talk about passing an &x ref to a function that takes an UnsafePointer or UnsafeMutablePointer (but of course changes to the book are not covered by the open-source project). Most uses of these functions can probably be replaced with a &x ref directly. If any can't, they could be replaced with the following equivalent expressions:

{ (ptr: UnsafePointer<Int>) in
   // ...
}(&x)

or:

withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
   // ...
}

One thing to keep in mind here is that with*Pointer and friends is also meant to enable one to work around issues with the optimizer if they come up in a convenient manner. I.e. imagine if one is attempting to process an image using a 5d array and for whatever reason, you are not getting the performance you need. Hopefully you would file a bug report and then use with*Pointer for your image processing loop.

My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.

···

On Dec 16, 2015, at 1:49 PM, Kevin Ballard via swift-evolution <swift-evolution@swift.org> wrote:
On Wed, Dec 16, 2015, at 11:38 AM, Kevin Ballard wrote:

# Detailed Solution

The functions would be marked as unavailable with a message saying to pass &x directly to the function taking a pointer. If it's feasible to do so, Fix-Its would be introduced for the trivial case of the pointer being used once in the closure as a parameter to a function.

The documentation of UnsafePointer and UnsafeMutablePointer would be edited to include a mention of how &x refs can be passed to functions expecting an UnsafePointer or UnsafeMutablePointer. This way anyone new to the language who encounters those types in the wild will be able to easily see how to create one from a variable.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[dabrahams (Dave Abrahams)]

What makes you say that?

If in fact it is true, shouldn't you file a bug report?

-Dave

···

On Dec 16, 2015, at 11:54 AM, Michael Gottesman via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 16, 2015, at 1:49 PM, Kevin Ballard via swift-evolution <swift-evolution@swift.org> wrote:

Another replacement for withUnsafe[Mutable]Pointer is declaring a nested function of the appropriate type (this is equivalent to the anonymous closure, but perhaps more readable):

func foo(ptr: UnsafePointer<Int>) {
  // ...
}
foo(&x)

-Kevin

On Wed, Dec 16, 2015, at 11:38 AM, Kevin Ballard wrote:

# Introduction

The stdlib provides functions withUnsafePointer() and withUnsafeMutablePointer() (and plural variants) that take an inout reference and call a block with the UnsafePointer/UnsafeMutablePointer created from the reference.

# Problem

withUnsafePointer() can only be used with mutable variables, because those are the only things that can be used with inout &refs. Both functions are also fairly useless, as &x refs can be passed directly to functions taking an UnsafePointer or UnsafeMutablePointer. The existence of the functions mostly just causes people to think they're necessary when they're not. The provide no functionality that passing &x refs directly to the functions taking a pointer doesn't already fulfill.

# Solution

Remove the functions from the stdlib. The Swift Book should also be updated to talk about passing an &x ref to a function that takes an UnsafePointer or UnsafeMutablePointer (but of course changes to the book are not covered by the open-source project). Most uses of these functions can probably be replaced with a &x ref directly. If any can't, they could be replaced with the following equivalent expressions:

{ (ptr: UnsafePointer<Int>) in
  // ...
}(&x)

or:

withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
  // ...
}

One thing to keep in mind here is that with*Pointer and friends is also meant to enable one to work around issues with the optimizer if they come up in a convenient manner. I.e. imagine if one is attempting to process an image using a 5d array and for whatever reason, you are not getting the performance you need. Hopefully you would file a bug report and then use with*Pointer for your image processing loop.

My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.

[Lily_Ballard]

One thing to keep in mind here is that with*Pointer and friends is also meant to enable one to work around issues with the optimizer if they come up in a convenient manner. I.e. imagine if one is attempting to process an image using a 5d array and for whatever reason, you are not getting the performance you need. Hopefully you would file a bug report and then use with*Pointer for your image processing loop.

The cited workarounds should all behave identically to withUnsafePointer() and friends. There's no special optimization done in withUnsafePointer() that's relevant. For the array example you cited, you'd actually want to be calling Array.withUnsafeMutableBufferPointer() anyway, as withUnsafePointer() doesn't do anything useful there.

My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.

I assume you mean that you don't need to extend the lifetime, rather than that it's not actually extending anything (because it is, it's just technically extending the lifetime of the UnsafePointer).

But in a sense it's actually correct anyway. In the rare case where you need one of these workarounds (e.g. you need to use the pointer several times without it changing, or you need to use a value derived from the pointer while the pointer is still valid), you actually do need to extend the lifetime of the value that was passed by-ref into the function. Swift already ensures the lifetime is valid for the duration of the call because it's an inout parameter and the function itself is only going to actually keep the pointer alive, but in a broader sense you are saying "keep this value alive for the lifetime of this closure, and give me a pointer to it".

···

On Wed, Dec 16, 2015, at 11:54 AM, Michael Gottesman wrote:

[Joe_Groff]

withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
  // ...
}

withExtendedLifetime doesn't have this overload. Adding it would essentially be just renaming withUnsafePointer.

No overload necessary. As I said, it already works today (tested in Swift 2.1). The generic type T is resolved to UnsafePointer<Int>, and so you can pass &x into it just fine because you're passing it to a parameter of type UnsafePointer.

That's interesting emergent behavior, but it's not something we anticipated withUnsafePointer being used for.

# Detailed Solution

The functions would be marked as unavailable with a message saying to pass &x directly to the function taking a pointer. If it's feasible to do so, Fix-Its would be introduced for the trivial case of the pointer being used once in the closure as a parameter to a function.

The documentation of UnsafePointer and UnsafeMutablePointer would be edited to include a mention of how &x refs can be passed to functions expecting an UnsafePointer or UnsafeMutablePointer. This way anyone new to the language who encounters those types in the wild will be able to easily see how to create one from a variable.

I'm against removing the functionality. The magic pointer conversions were only ever intended to make interop with well-behaved C functions easier; it was in my mind that we would eventually constrain the magic pointer conversions to only apply to imported APIs. withUnsafePointer is necessary more often than you think, since the current semantics of the conversion only keep the pointer valid for one call—you can't call a conversion constructor or otherwise touch the pointer in between. We should fix that, but it'll still be necessary to persist a pointer across multiple C calls that expect pointer identity to be maintained.

Why constrain the magic pointer conversions?

In general, implicit conversions are bad. These particular conversions lead to lots of gross knock-on effects, things like [1] - [2] working by taking the difference of two transient pointers.

We should discourage people from writing Swift functions that take UnsafePointers, but sometimes it is appropriate (e.g. when wrapping imported C functions, if you need a stable pointer or need to do pointer math, then your function should be taking Unsafe[Mutable]Pointer instead of taking an inout parameter).

If you need a stable pointer internally in your implementation, I don't see a reason to expose that implementation detail to users. You could use withUnsafeMutablePointer(&inoutParameter) internally.

Your proposed alternatives all involve essentially reimplementing withUnsafePointer in a different way.

Well yeah, but without requiring anything in the stdlib. swiftdoc.org lists 36 global Swift functions. 4 of them are unsafe[Mutable]Pointer[s]. That's 11% of the global functions in the stdlib, for functionality that can already be accomplished without any stdlib support and which is relatively rare.

I agree, it's unfortunate that these functions pollute the global namespace. Maybe they should be static methods of Unsafe[Mutable]Pointer instead?

The way I see it, if the stdlib didn't already have these functions, we wouldn't add them. So why keep them around? All they do is make people think they're necessary.

We added them because they *are* necessary.

-Joe

···

On Dec 16, 2015, at 1:11 PM, Kevin Ballard <kevin@sb.org> wrote:
On Wed, Dec 16, 2015, at 11:58 AM, Joe Groff wrote:

On Dec 16, 2015, at 11:38 AM, Kevin Ballard via swift-evolution <swift-evolution@swift.org> wrote:

[Michael_Gottesman]

Another replacement for withUnsafe[Mutable]Pointer is declaring a nested function of the appropriate type (this is equivalent to the anonymous closure, but perhaps more readable):

func foo(ptr: UnsafePointer<Int>) {
  // ...
}
foo(&x)

-Kevin

# Introduction

The stdlib provides functions withUnsafePointer() and withUnsafeMutablePointer() (and plural variants) that take an inout reference and call a block with the UnsafePointer/UnsafeMutablePointer created from the reference.

# Problem

withUnsafePointer() can only be used with mutable variables, because those are the only things that can be used with inout &refs. Both functions are also fairly useless, as &x refs can be passed directly to functions taking an UnsafePointer or UnsafeMutablePointer. The existence of the functions mostly just causes people to think they're necessary when they're not. The provide no functionality that passing &x refs directly to the functions taking a pointer doesn't already fulfill.

# Solution

Remove the functions from the stdlib. The Swift Book should also be updated to talk about passing an &x ref to a function that takes an UnsafePointer or UnsafeMutablePointer (but of course changes to the book are not covered by the open-source project). Most uses of these functions can probably be replaced with a &x ref directly. If any can't, they could be replaced with the following equivalent expressions:

{ (ptr: UnsafePointer<Int>) in
  // ...
}(&x)

or:

withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
  // ...
}

One thing to keep in mind here is that with*Pointer and friends is also meant to enable one to work around issues with the optimizer if they come up in a convenient manner. I.e. imagine if one is attempting to process an image using a 5d array and for whatever reason, you are not getting the performance you need. Hopefully you would file a bug report and then use with*Pointer for your image processing loop.

My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.

It sounds to me like what you really want is something like:

with (&x) {

}

which has been spoken about before internally. I do not remember what the result of it is.

···

On Dec 16, 2015, at 1:54 PM, Michael Gottesman via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 16, 2015, at 1:49 PM, Kevin Ballard via swift-evolution <swift-evolution@swift.org> wrote:
On Wed, Dec 16, 2015, at 11:38 AM, Kevin Ballard wrote:

# Detailed Solution

The functions would be marked as unavailable with a message saying to pass &x directly to the function taking a pointer. If it's feasible to do so, Fix-Its would be introduced for the trivial case of the pointer being used once in the closure as a parameter to a function.

The documentation of UnsafePointer and UnsafeMutablePointer would be edited to include a mention of how &x refs can be passed to functions expecting an UnsafePointer or UnsafeMutablePointer. This way anyone new to the language who encounters those types in the wild will be able to easily see how to create one from a variable.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

[Michael_Gottesman]

Another replacement for withUnsafe[Mutable]Pointer is declaring a nested function of the appropriate type (this is equivalent to the anonymous closure, but perhaps more readable):

func foo(ptr: UnsafePointer<Int>) {
// ...
}
foo(&x)

-Kevin

# Introduction

The stdlib provides functions withUnsafePointer() and withUnsafeMutablePointer() (and plural variants) that take an inout reference and call a block with the UnsafePointer/UnsafeMutablePointer created from the reference.

# Problem

withUnsafePointer() can only be used with mutable variables, because those are the only things that can be used with inout &refs. Both functions are also fairly useless, as &x refs can be passed directly to functions taking an UnsafePointer or UnsafeMutablePointer. The existence of the functions mostly just causes people to think they're necessary when they're not. The provide no functionality that passing &x refs directly to the functions taking a pointer doesn't already fulfill.

# Solution

Remove the functions from the stdlib. The Swift Book should also be updated to talk about passing an &x ref to a function that takes an UnsafePointer or UnsafeMutablePointer (but of course changes to the book are not covered by the open-source project). Most uses of these functions can probably be replaced with a &x ref directly. If any can't, they could be replaced with the following equivalent expressions:

{ (ptr: UnsafePointer<Int>) in
// ...
}(&x)

or:

withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
// ...
}

One thing to keep in mind here is that with*Pointer and friends is also meant to enable one to work around issues with the optimizer if they come up in a convenient manner. I.e. imagine if one is attempting to process an image using a 5d array and for whatever reason, you are not getting the performance you need. Hopefully you would file a bug report and then use with*Pointer for your image processing loop.

My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.

What makes you say that?

Let me be more specific.

My issue with the name 'withExtendedLifetime' is that it is suggestive that the lifetime of &x is being extended in a way that is different from if one just passed off &x to any old function. In reality though, nothing special is happening here implying that the name is misleading. A better name IMO would be something that drops any such implication.

Michael

···

On Dec 16, 2015, at 2:22 PM, Dave Abrahams <dabrahams@apple.com> wrote:

On Dec 16, 2015, at 11:54 AM, Michael Gottesman via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 16, 2015, at 1:49 PM, Kevin Ballard via swift-evolution <swift-evolution@swift.org> wrote:
On Wed, Dec 16, 2015, at 11:38 AM, Kevin Ballard wrote:

If in fact it is true, shouldn't you file a bug report?

-Dave

[Michael_Gottesman]

One thing to keep in mind here is that with*Pointer and friends is also meant to enable one to work around issues with the optimizer if they come up in a convenient manner. I.e. imagine if one is attempting to process an image using a 5d array and for whatever reason, you are not getting the performance you need. Hopefully you would file a bug report and then use with*Pointer for your image processing loop.

The cited workarounds should all behave identically to withUnsafePointer() and friends. There's no special optimization done in withUnsafePointer() that's relevant. For the array example you cited, you'd actually want to be calling Array.withUnsafeMutableBufferPointer() anyway, as withUnsafePointer() doesn't do anything useful there.

The interesting example is that by using withUnsafePointer one can avoid ARC, COW, or Uniqueness Check overhead.

My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.

I assume you mean that you don't need to extend the lifetime, rather than that it's not actually extending anything (because it is, it's just technically extending the lifetime of the UnsafePointer).

No, what I am saying is that withExtendedLifetime implies some sort of special lifetime extension (otherwise, why would one specifically call it out). It is not performing any sort of special lifetime extension, it is just using the normal lifetime extension rules.

But in a sense it's actually correct anyway. In the rare case where you need one of these workarounds (e.g. you need to use the pointer several times without it changing, or you need to use a value derived from the pointer while the pointer is still valid), you actually do need to extend the lifetime of the value that was passed by-ref into the function. Swift already ensures the lifetime is valid for the duration of the call because it's an inout parameter and the function itself is only going to actually keep the pointer alive, but in a broader sense you are saying "keep this value alive for the lifetime of this closure, and give me a pointer to it".

Michael

···

On Dec 16, 2015, at 3:05 PM, Kevin Ballard <kevin@sb.org> wrote:
On Wed, Dec 16, 2015, at 11:54 AM, Michael Gottesman wrote:

[Joe_Groff]

And if you really need to preserve identity across multiple function calls, I'd argue that after a certain point, it's better to allocate the memory yourself with UnsafeMutablePointer.alloc/dealloc (with optimizer support for promoting small, scoped .alloc/.dealloc pairs to stack allocations), since interfacing compiler-managed properties with manual memory manipulation is always going to be finicky.

-Joe

···

On Dec 16, 2015, at 1:20 PM, Joe Groff <jgroff@apple.com> wrote:

On Dec 16, 2015, at 1:11 PM, Kevin Ballard <kevin@sb.org> wrote:

On Wed, Dec 16, 2015, at 11:58 AM, Joe Groff wrote:

On Dec 16, 2015, at 11:38 AM, Kevin Ballard via swift-evolution <swift-evolution@swift.org> wrote:

withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
// ...
}

withExtendedLifetime doesn't have this overload. Adding it would essentially be just renaming withUnsafePointer.

No overload necessary. As I said, it already works today (tested in Swift 2.1). The generic type T is resolved to UnsafePointer<Int>, and so you can pass &x into it just fine because you're passing it to a parameter of type UnsafePointer.

That's interesting emergent behavior, but it's not something we anticipated withUnsafePointer being used for.

# Detailed Solution

The functions would be marked as unavailable with a message saying to pass &x directly to the function taking a pointer. If it's feasible to do so, Fix-Its would be introduced for the trivial case of the pointer being used once in the closure as a parameter to a function.

The documentation of UnsafePointer and UnsafeMutablePointer would be edited to include a mention of how &x refs can be passed to functions expecting an UnsafePointer or UnsafeMutablePointer. This way anyone new to the language who encounters those types in the wild will be able to easily see how to create one from a variable.

I'm against removing the functionality. The magic pointer conversions were only ever intended to make interop with well-behaved C functions easier; it was in my mind that we would eventually constrain the magic pointer conversions to only apply to imported APIs. withUnsafePointer is necessary more often than you think, since the current semantics of the conversion only keep the pointer valid for one call—you can't call a conversion constructor or otherwise touch the pointer in between. We should fix that, but it'll still be necessary to persist a pointer across multiple C calls that expect pointer identity to be maintained.

Why constrain the magic pointer conversions?

In general, implicit conversions are bad. These particular conversions lead to lots of gross knock-on effects, things like [1] - [2] working by taking the difference of two transient pointers.

We should discourage people from writing Swift functions that take UnsafePointers, but sometimes it is appropriate (e.g. when wrapping imported C functions, if you need a stable pointer or need to do pointer math, then your function should be taking Unsafe[Mutable]Pointer instead of taking an inout parameter).

If you need a stable pointer internally in your implementation, I don't see a reason to expose that implementation detail to users. You could use withUnsafeMutablePointer(&inoutParameter) internally.

[Lily_Ballard]

>> withExtendedLifetime doesn't have this overload. Adding it would essentially be just renaming withUnsafePointer.
>
> No overload necessary. As I said, it already works today (tested in Swift 2.1). The generic type T is resolved to UnsafePointer<Int>, and so you can pass &x into it just fine because you're passing it to a parameter of type UnsafePointer.

That's interesting emergent behavior, but it's not something we anticipated withUnsafePointer being used for.

Sure, which is why there's 2 other alternatives that work just fine as well (nested function, or anonymous closure). And I listed that mostly because I find it neat that it actually works and it has the nice property of keeping the &x ref above the scoped pointer instead of putting it below. But at this point I'm inclined to encourage the nested function approach instead as it's simpler.

> Why constrain the magic pointer conversions?

In general, implicit conversions are bad. These particular conversions lead to lots of gross knock-on effects, things like [1] - [2] working by taking the difference of two transient pointers.

I meant why constrain them to just imported functions. I fully agree with the current constraint of only doing the conversion when the &x ref is passed as a parameter to a function expecting Unsafe[Mutable]Pointer. I don't see any compelling reason to constrain it further to only imported functions, and that would be rather confusing (looking at the function signature, there'd be no way to tell whether the conversion would work or not).

> We should discourage people from writing Swift functions that take UnsafePointers, but sometimes it is appropriate (e.g. when wrapping imported C functions, if you need a stable pointer or need to do pointer math, then your function should be taking Unsafe[Mutable]Pointer instead of taking an inout parameter).

If you need a stable pointer internally in your implementation, I don't see a reason to expose that implementation detail to users. You could use withUnsafeMutablePointer(&inoutParameter) internally.

You could, and in many cases that may be appropriate, but it really depends on the function. If I'm writing a private function that wraps a few calls to a C library, I may still want it to accept a pointer because e.g. the caller may need to pass a stable pointer to several functions including my Swift wrapper function.

>> Your proposed alternatives all involve essentially reimplementing withUnsafePointer in a different way.
>
> Well yeah, but without requiring anything in the stdlib. swiftdoc.org lists 36 global Swift functions. 4 of them are unsafe[Mutable]Pointer[s]. That's 11% of the global functions in the stdlib, for functionality that can already be accomplished without any stdlib support and which is relatively rare.

I agree, it's unfortunate that these functions pollute the global namespace. Maybe they should be static methods of Unsafe[Mutable]Pointer instead?

> The way I see it, if the stdlib didn't already have these functions, we wouldn't add them. So why keep them around? All they do is make people think they're necessary.

We added them because they *are* necessary.

You still haven't demonstrated that. Every single use of withUnsafe[Mutable]Pointer[s] that can't be handled with the implicit &x conversion can be replaced with a nested function, a call to an anonymous closure, or an (ab)use of withExtendedLifetime. And the existence of the functions encourages people to call them when a simple &x ref would have sufficed.

-Kevin Ballard

···

On Wed, Dec 16, 2015, at 01:20 PM, Joe Groff wrote:

[Michael_Gottesman]

... *was.

Michael

···

On Dec 16, 2015, at 1:57 PM, Michael Gottesman <mgottesman@apple.com> wrote:

On Dec 16, 2015, at 1:54 PM, Michael Gottesman via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Dec 16, 2015, at 1:49 PM, Kevin Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Another replacement for withUnsafe[Mutable]Pointer is declaring a nested function of the appropriate type (this is equivalent to the anonymous closure, but perhaps more readable):

func foo(ptr: UnsafePointer<Int>) {
  // ...
}
foo(&x)

-Kevin

On Wed, Dec 16, 2015, at 11:38 AM, Kevin Ballard wrote:

# Introduction

The stdlib provides functions withUnsafePointer() and withUnsafeMutablePointer() (and plural variants) that take an inout reference and call a block with the UnsafePointer/UnsafeMutablePointer created from the reference.

# Problem

withUnsafePointer() can only be used with mutable variables, because those are the only things that can be used with inout &refs. Both functions are also fairly useless, as &x refs can be passed directly to functions taking an UnsafePointer or UnsafeMutablePointer. The existence of the functions mostly just causes people to think they're necessary when they're not. The provide no functionality that passing &x refs directly to the functions taking a pointer doesn't already fulfill.

# Solution

Remove the functions from the stdlib. The Swift Book should also be updated to talk about passing an &x ref to a function that takes an UnsafePointer or UnsafeMutablePointer (but of course changes to the book are not covered by the open-source project). Most uses of these functions can probably be replaced with a &x ref directly. If any can't, they could be replaced with the following equivalent expressions:

{ (ptr: UnsafePointer<Int>) in
  // ...
}(&x)

or:

withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
  // ...
}

One thing to keep in mind here is that with*Pointer and friends is also meant to enable one to work around issues with the optimizer if they come up in a convenient manner. I.e. imagine if one is attempting to process an image using a 5d array and for whatever reason, you are not getting the performance you need. Hopefully you would file a bug report and then use with*Pointer for your image processing loop.

My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.

It sounds to me like what you really want is something like:

with (&x) {

}

which has been spoken about before internally. I do not remember what the result of it is.

# Detailed Solution

The functions would be marked as unavailable with a message saying to pass &x directly to the function taking a pointer. If it's feasible to do so, Fix-Its would be introduced for the trivial case of the pointer being used once in the closure as a parameter to a function.

The documentation of UnsafePointer and UnsafeMutablePointer would be edited to include a mention of how &x refs can be passed to functions expecting an UnsafePointer or UnsafeMutablePointer. This way anyone new to the language who encounters those types in the wild will be able to easily see how to create one from a variable.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

[jtbandes (Jacob Bandes-Storch)]

withExtendedLifetime(&x) does extend the lifetime of &x, it just doesn't
extend the lifetime of x.

However, withExtendedLifetime(x) does extend the lifetime of x by holding a
strong reference (if it's a reference type), which can be useful. Maybe
withExtendedLifetime should be changed to <T: AnyObject> rather than <T> ?

Jacob

···

On Wed, Dec 16, 2015 at 2:07 PM, Michael Gottesman via swift-evolution < swift-evolution@swift.org> wrote:

> On Dec 16, 2015, at 2:22 PM, Dave Abrahams <dabrahams@apple.com> wrote:
>
>
>> On Dec 16, 2015, at 11:54 AM, Michael Gottesman via swift-evolution < > swift-evolution@swift.org> wrote:
>>
>>>
>>> On Dec 16, 2015, at 1:49 PM, Kevin Ballard via swift-evolution < > swift-evolution@swift.org> wrote:
>>>
>>> Another replacement for withUnsafe[Mutable]Pointer is declaring a
nested function of the appropriate type (this is equivalent to the
anonymous closure, but perhaps more readable):
>>>
>>> func foo(ptr: UnsafePointer<Int>) {
>>> // ...
>>> }
>>> foo(&x)
>>>
>>> -Kevin
>>>
>>> On Wed, Dec 16, 2015, at 11:38 AM, Kevin Ballard wrote:
>>>> # Introduction
>>>>
>>>> The stdlib provides functions withUnsafePointer() and
withUnsafeMutablePointer() (and plural variants) that take an inout
reference and call a block with the UnsafePointer/UnsafeMutablePointer
created from the reference.
>>>>
>>>> # Problem
>>>>
>>>> withUnsafePointer() can only be used with mutable variables, because
those are the only things that can be used with inout &refs. Both functions
are also fairly useless, as &x refs can be passed directly to functions
taking an UnsafePointer or UnsafeMutablePointer. The existence of the
functions mostly just causes people to think they're necessary when they're
not. The provide no functionality that passing &x refs directly to the
functions taking a pointer doesn't already fulfill.
>>>>
>>>> # Solution
>>>>
>>>> Remove the functions from the stdlib. The Swift Book should also be
updated to talk about passing an &x ref to a function that takes an
UnsafePointer or UnsafeMutablePointer (but of course changes to the book
are not covered by the open-source project). Most uses of these functions
can probably be replaced with a &x ref directly. If any can't, they could
be replaced with the following equivalent expressions:
>>>>
>>>> { (ptr: UnsafePointer<Int>) in
>>>> // ...
>>>> }(&x)
>>>>
>>>> or:
>>>>
>>>> withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
>>>> // ...
>>>> }
>>
>> One thing to keep in mind here is that with*Pointer and friends is also
meant to enable one to work around issues with the optimizer if they come
up in a convenient manner. I.e. imagine if one is attempting to process an
image using a 5d array and for whatever reason, you are not getting the
performance you need. Hopefully you would file a bug report and then use
with*Pointer for your image processing loop.
>>
>> My fear about withExtendedLifetime is that the name is a misnomer. You
are not extending the lifetime.
>
> What makes you say that?

Let me be more specific.

My issue with the name 'withExtendedLifetime' is that it is suggestive
that the lifetime of &x is being extended in a way that is different from
if one just passed off &x to any old function. In reality though, nothing
special is happening here implying that the name is misleading. A better
name IMO would be something that drops any such implication.

Michael

>
> If in fact it is true, shouldn't you file a bug report?
>
> -Dave
>
>
>

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[jrose (Jordan Rose)]

withExtendedLifetime does extend lifetime: it guarantees that the object will stay alive for the entire body of the closure, rather than being released early by ARC. It's similar to the 'objc_precise_lifetime' attribute in Clang.

Jordan

···

On Dec 16, 2015, at 14:24 , Michael Gottesman via swift-evolution <swift-evolution@swift.org> wrote:

My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.

I assume you mean that you don't need to extend the lifetime, rather than that it's not actually extending anything (because it is, it's just technically extending the lifetime of the UnsafePointer).

No, what I am saying is that withExtendedLifetime implies some sort of special lifetime extension (otherwise, why would one specifically call it out). It is not performing any sort of special lifetime extension, it is just using the normal lifetime extension rules.

[jrose (Jordan Rose)]

Michael explained to me that his comments are referring to Kevin's proposed (?) new overload of withExtendedLifetime, not the existing overload which takes a value.

Jordan

···

On Dec 16, 2015, at 14:27 , Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 16, 2015, at 14:24 , Michael Gottesman via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.

I assume you mean that you don't need to extend the lifetime, rather than that it's not actually extending anything (because it is, it's just technically extending the lifetime of the UnsafePointer).

No, what I am saying is that withExtendedLifetime implies some sort of special lifetime extension (otherwise, why would one specifically call it out). It is not performing any sort of special lifetime extension, it is just using the normal lifetime extension rules.

withExtendedLifetime does extend lifetime: it guarantees that the object will stay alive for the entire body of the closure, rather than being released early by ARC. It's similar to the 'objc_precise_lifetime' attribute in Clang.

[Michael_Gottesman]

Another replacement for withUnsafe[Mutable]Pointer is declaring a nested function of the appropriate type (this is equivalent to the anonymous closure, but perhaps more readable):

func foo(ptr: UnsafePointer<Int>) {
// ...
}
foo(&x)

-Kevin

# Introduction

The stdlib provides functions withUnsafePointer() and withUnsafeMutablePointer() (and plural variants) that take an inout reference and call a block with the UnsafePointer/UnsafeMutablePointer created from the reference.

# Problem

withUnsafePointer() can only be used with mutable variables, because those are the only things that can be used with inout &refs. Both functions are also fairly useless, as &x refs can be passed directly to functions taking an UnsafePointer or UnsafeMutablePointer. The existence of the functions mostly just causes people to think they're necessary when they're not. The provide no functionality that passing &x refs directly to the functions taking a pointer doesn't already fulfill.

# Solution

Remove the functions from the stdlib. The Swift Book should also be updated to talk about passing an &x ref to a function that takes an UnsafePointer or UnsafeMutablePointer (but of course changes to the book are not covered by the open-source project). Most uses of these functions can probably be replaced with a &x ref directly. If any can't, they could be replaced with the following equivalent expressions:

{ (ptr: UnsafePointer<Int>) in
// ...
}(&x)

or:

withExtendedLifetime(&x) { (ptr: UnsafePointer<Int>) in
// ...
}

One thing to keep in mind here is that with*Pointer and friends is also meant to enable one to work around issues with the optimizer if they come up in a convenient manner. I.e. imagine if one is attempting to process an image using a 5d array and for whatever reason, you are not getting the performance you need. Hopefully you would file a bug report and then use with*Pointer for your image processing loop.

My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.

What makes you say that?

Let me be more specific.

My issue with the name 'withExtendedLifetime' is that it is suggestive that the lifetime of &x is being extended in a way that is different from if one just passed off &x to any old function. In reality though, nothing special is happening here implying that the name is misleading. A better name IMO would be something that drops any such implication.

For example, we could just use 'with' (something that has been suggested for this use case in various blog posts). I.e.:

with(&x) { (ptr: UnsafePointer<Int>) in
   ...
}

Michael

···

On Dec 16, 2015, at 4:07 PM, Michael Gottesman via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 16, 2015, at 2:22 PM, Dave Abrahams <dabrahams@apple.com> wrote:

On Dec 16, 2015, at 11:54 AM, Michael Gottesman via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 16, 2015, at 1:49 PM, Kevin Ballard via swift-evolution <swift-evolution@swift.org> wrote:
On Wed, Dec 16, 2015, at 11:38 AM, Kevin Ballard wrote:

Michael

If in fact it is true, shouldn't you file a bug report?

-Dave

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Michael_Gottesman]

One thing to keep in mind here is that with*Pointer and friends is also meant to enable one to work around issues with the optimizer if they come up in a convenient manner. I.e. imagine if one is attempting to process an image using a 5d array and for whatever reason, you are not getting the performance you need. Hopefully you would file a bug report and then use with*Pointer for your image processing loop.

The cited workarounds should all behave identically to withUnsafePointer() and friends. There's no special optimization done in withUnsafePointer() that's relevant. For the array example you cited, you'd actually want to be calling Array.withUnsafeMutableBufferPointer() anyway, as withUnsafePointer() doesn't do anything useful there.

The interesting example is that by using withUnsafePointer one can avoid ARC, COW, or Uniqueness Check overhead.

And to be clear, I am just saying that we need some form of this API. I do not care the form that it takes.

···

On Dec 16, 2015, at 4:24 PM, Michael Gottesman via swift-evolution <swift-evolution@swift.org> wrote:

On Dec 16, 2015, at 3:05 PM, Kevin Ballard <kevin@sb.org> wrote:
On Wed, Dec 16, 2015, at 11:54 AM, Michael Gottesman wrote:

My fear about withExtendedLifetime is that the name is a misnomer. You are not extending the lifetime.

I assume you mean that you don't need to extend the lifetime, rather than that it's not actually extending anything (because it is, it's just technically extending the lifetime of the UnsafePointer).

No, what I am saying is that withExtendedLifetime implies some sort of special lifetime extension (otherwise, why would one specifically call it out). It is not performing any sort of special lifetime extension, it is just using the normal lifetime extension rules.

But in a sense it's actually correct anyway. In the rare case where you need one of these workarounds (e.g. you need to use the pointer several times without it changing, or you need to use a value derived from the pointer while the pointer is still valid), you actually do need to extend the lifetime of the value that was passed by-ref into the function. Swift already ensures the lifetime is valid for the duration of the call because it's an inout parameter and the function itself is only going to actually keep the pointer alive, but in a broader sense you are saying "keep this value alive for the lifetime of this closure, and give me a pointer to it".

Michael
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Joe_Groff]

withExtendedLifetime doesn't have this overload. Adding it would essentially be just renaming withUnsafePointer.

No overload necessary. As I said, it already works today (tested in Swift 2.1). The generic type T is resolved to UnsafePointer<Int>, and so you can pass &x into it just fine because you're passing it to a parameter of type UnsafePointer.

That's interesting emergent behavior, but it's not something we anticipated withUnsafePointer being used for.

Sure, which is why there's 2 other alternatives that work just fine as well (nested function, or anonymous closure). And I listed that mostly because I find it neat that it actually works and it has the nice property of keeping the &x ref above the scoped pointer instead of putting it below. But at this point I'm inclined to encourage the nested function approach instead as it's simpler.

Why constrain the magic pointer conversions?

In general, implicit conversions are bad. These particular conversions lead to lots of gross knock-on effects, things like [1] - [2] working by taking the difference of two transient pointers.

I meant why constrain them to just imported functions. I fully agree with the current constraint of only doing the conversion when the &x ref is passed as a parameter to a function expecting Unsafe[Mutable]Pointer. I don't see any compelling reason to constrain it further to only imported functions, and that would be rather confusing (looking at the function signature, there'd be no way to tell whether the conversion would work or not).

The pointer conversions are already poorly discoverable, and people don't know they exist since code completion doesn't make it apparent. Showing the APIs as overloaded taking 'inout's or something similar would make it more apparent how you can use them. It's my hope that better SDK annotations can make these conversions less necessary too, by letting us eventually import well-behaved in/inout/out arguments directly as value arguments/inouts/extra returns.

We should discourage people from writing Swift functions that take UnsafePointers, but sometimes it is appropriate (e.g. when wrapping imported C functions, if you need a stable pointer or need to do pointer math, then your function should be taking Unsafe[Mutable]Pointer instead of taking an inout parameter).

If you need a stable pointer internally in your implementation, I don't see a reason to expose that implementation detail to users. You could use withUnsafeMutablePointer(&inoutParameter) internally.

You could, and in many cases that may be appropriate, but it really depends on the function. If I'm writing a private function that wraps a few calls to a C library, I may still want it to accept a pointer because e.g. the caller may need to pass a stable pointer to several functions including my Swift wrapper function.

Your proposed alternatives all involve essentially reimplementing withUnsafePointer in a different way.

Well yeah, but without requiring anything in the stdlib. swiftdoc.org lists 36 global Swift functions. 4 of them are unsafe[Mutable]Pointer[s]. That's 11% of the global functions in the stdlib, for functionality that can already be accomplished without any stdlib support and which is relatively rare.

I agree, it's unfortunate that these functions pollute the global namespace. Maybe they should be static methods of Unsafe[Mutable]Pointer instead?

The way I see it, if the stdlib didn't already have these functions, we wouldn't add them. So why keep them around? All they do is make people think they're necessary.

We added them because they *are* necessary.

You still haven't demonstrated that. Every single use of withUnsafe[Mutable]Pointer[s] that can't be handled with the implicit &x conversion can be replaced with a nested function, a call to an anonymous closure, or an (ab)use of withExtendedLifetime. And the existence of the functions encourages people to call them when a simple &x ref would have sufficed.

I guess it's a difference of perspective. Your alternatives hinge on blessing the pointer conversions as a core part of the language, rather than an affordance to make many C APIs not totally awful, which I'm not comfortable with.

-Joe

···

On Dec 16, 2015, at 1:30 PM, Kevin Ballard <kevin@sb.org> wrote:
On Wed, Dec 16, 2015, at 01:20 PM, Joe Groff wrote: