`withUnsafePointer` mutates `self`

I've been playing around with a Swift wrapper for the FFmpeg C libraries
(e.g., libavutil, libavcodec, libavformat, etc...). While providing some
extensions to some of the core C structures I've run into something that
doesn't feel quite right to me.

Before I provide a discussion here is a Gist [0] that I hope illustrates
the problem.

When inside of a Swift function declaration if I take `self` and send it
to `withUnsafePointer` Swift requires the function to mark the function
as `mutating`. This surprised me. Thanks to Swift being open source (:tada:)
I was able to go look at the implementation of `withUnsafePointer` [1].
And I'm no longer surprised that its required by the compiler; the
definition marks the argument as `inout`.

However, this seems wrong to me. In the "Pointers" section of "Using
Swift with Cocoa and Objective-C (Swift 2.1)" [2] it says that `cost
Type *` (pointer to a constant value) is equivalent to
`UnsafePointer<Type>`. My understanding of C says that `const Type *`
means that the instance of `Type` that the pointer points to cannot be
modified. This corresponds with my understanding of the difference
between `UnsafePointer<Type>` and `UnsafeMutablePointer<Type>`.

Therefore, from my perspective I feel like there is a bug here. It also
seems that marking the `arg` of `withUnsafePointer` to `var` instead of
`inout` would allow it to compile and would more closely model the
`UnsafePointer`. I'm still new to all this and I don't want to make a
fool of myself. So I'd rather ask here before I move forward and file a
bug and MR.

[0] Allow `withUnsafePointer` to not mark `struct` function as `mutating` · GitHub
[1]
swift/LifetimeManager.swift at 8d9ef80304d7b36e13619ea50e6e76f3ec9221ba · apple/swift · GitHub
[2]
Apple Developer Documentation

Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.

-Joe

···

On Dec 15, 2015, at 8:12 AM, Ryan Lovelett via swift-dev <swift-dev@swift.org> wrote:

I've been playing around with a Swift wrapper for the FFmpeg C libraries
(e.g., libavutil, libavcodec, libavformat, etc...). While providing some
extensions to some of the core C structures I've run into something that
doesn't feel quite right to me.

Before I provide a discussion here is a Gist [0] that I hope illustrates
the problem.

When inside of a Swift function declaration if I take `self` and send it
to `withUnsafePointer` Swift requires the function to mark the function
as `mutating`. This surprised me. Thanks to Swift being open source (:tada:)
I was able to go look at the implementation of `withUnsafePointer` [1].
And I'm no longer surprised that its required by the compiler; the
definition marks the argument as `inout`.

However, this seems wrong to me. In the "Pointers" section of "Using
Swift with Cocoa and Objective-C (Swift 2.1)" [2] it says that `cost
Type *` (pointer to a constant value) is equivalent to
`UnsafePointer<Type>`. My understanding of C says that `const Type *`
means that the instance of `Type` that the pointer points to cannot be
modified. This corresponds with my understanding of the difference
between `UnsafePointer<Type>` and `UnsafeMutablePointer<Type>`.

Therefore, from my perspective I feel like there is a bug here. It also
seems that marking the `arg` of `withUnsafePointer` to `var` instead of
`inout` would allow it to compile and would more closely model the
`UnsafePointer`. I'm still new to all this and I don't want to make a
fool of myself. So I'd rather ask here before I move forward and file a
bug and MR.

A change like that is going to break any code that relies on the inout optimization (where it uses call-by-reference instead of copy-in copy-out when possible). Yes, such code is in violation of Swift semantics today, but it does work.

-Kevin Ballard

···

On Tue, Dec 15, 2015, at 03:03 PM, Joe Groff via swift-dev wrote:

Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.

Two questions:

1. Don’t we want a withUnsafeMutablePointer for the mutating cases (where the inout optimization can take effect) anyway?

2. Joe, these APIs predate many of your changes that make &x transparently convert to Unsafe[Mutable]Pointer arguments. Are they obsolete? Can we replace them with { x: Unsafe[Mutable]Pointer in … }(&y) ?

-Dave

···

On Dec 15, 2015, at 6:33 PM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:

On Tue, Dec 15, 2015, at 03:03 PM, Joe Groff via swift-dev wrote:

Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.

A change like that is going to break any code that relies on the inout optimization (where it uses call-by-reference instead of copy-in copy-out when possible). Yes, such code is in violation of Swift semantics today, but it does work.

If a value is immutable, it should be easy for the compiler to reuse the address of memory it may already be using for it.

-Joe

···

On Dec 15, 2015, at 6:33 PM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:

On Tue, Dec 15, 2015, at 03:03 PM, Joe Groff via swift-dev wrote:

Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.

A change like that is going to break any code that relies on the inout optimization (where it uses call-by-reference instead of copy-in copy-out when possible). Yes, such code is in violation of Swift semantics today, but it does work.

>
>>
>> Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.
>
> A change like that is going to break any code that relies on the inout optimization (where it uses call-by-reference instead of copy-in copy-out when possible). Yes, such code is in violation of Swift semantics today, but it does work.

Two questions:

1. Don’t we want a withUnsafeMutablePointer for the mutating cases (where the inout optimization can take effect) anyway?

I'm thinking here of cases like passing a context pointer to KVO. You're not actually mutating it, you just need a pointer that's the same every time you call the code.

2. Joe, these APIs predate many of your changes that make &x transparently convert to Unsafe[Mutable]Pointer arguments. Are they obsolete? Can we replace them with { x: Unsafe[Mutable]Pointer in … }(&y) ?

Interesting idea. I rarely use withUnsafePointer/withUnsafeMutablePointer because the &x shorthand exists. If withUnsafePointer() is modified to take its argument by-value and create a pointer from it, I expect people will start using it in place of &x shorthand so they can use it with immutable values, without realizing that they're getting a different pointer back each time (whether they notice this depends on what they're using the pointer for).

If { x: UnsafePointer<Int> in ... }(&y) works, that's a cute replacement. Although it does move the &y reference to after the scope instead of before, which is a little unfortunate.

...huh. On a whim I tested the following and it shocked me by working.

var x: Int = 42
withExtendedLifetime(&x) { (x: UnsafeMutablePointer<Int>) in
    x.memory += 1
}
print(x) // prints 43

If we want to replace withUnsafe[Mutable]Pointer() with a common idiom, I'd recommend using withExtendedLifetime like that, because a) it guarantees the pointed-to value exists for the duration of the scope, and b) it already works.

-Kevin Ballard

···

On Tue, Dec 15, 2015, at 06:39 PM, Dave Abrahams wrote:

> On Dec 15, 2015, at 6:33 PM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:
> On Tue, Dec 15, 2015, at 03:03 PM, Joe Groff via swift-dev wrote:

Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.

A change like that is going to break any code that relies on the inout optimization (where it uses call-by-reference instead of copy-in copy-out when possible). Yes, such code is in violation of Swift semantics today, but it does work.

Two questions:

1. Don’t we want a withUnsafeMutablePointer for the mutating cases (where the inout optimization can take effect) anyway?

Yeah, a withUnsafeMutablePointer variant that is inout would be necessary.

2. Joe, these APIs predate many of your changes that make &x transparently convert to Unsafe[Mutable]Pointer arguments. Are they obsolete? Can we replace them with { x: Unsafe[Mutable]Pointer in … }(&y) ?

Not if you need the same pointer across multiple calls, or you need to convert or adjust the pointer before the call. A call like foo(&x) that involves a pointer conversion only guarantees that pointer for that exact call, as if you'd written withUnsafe[Mutable]Pointer(&x) { foo($0) }.

-Joe

···

On Dec 15, 2015, at 6:39 PM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:

On Dec 15, 2015, at 6:33 PM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:
On Tue, Dec 15, 2015, at 03:03 PM, Joe Groff via swift-dev wrote:

Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.

A change like that is going to break any code that relies on the inout optimization (where it uses call-by-reference instead of copy-in copy-out when possible). Yes, such code is in violation of Swift semantics today, but it does work.

Two questions:

1. Don’t we want a withUnsafeMutablePointer for the mutating cases (where the inout optimization can take effect) anyway?

I'm thinking here of cases like passing a context pointer to KVO. You're not actually mutating it, you just need a pointer that's the same every time you call the code.

Well, it is not possible to code a version of withUnsafePointer that makes that guarantee in Swift.

2. Joe, these APIs predate many of your changes that make &x transparently convert to Unsafe[Mutable]Pointer arguments. Are they obsolete? Can we replace them with { x: Unsafe[Mutable]Pointer in … }(&y) ?

Interesting idea. I rarely use withUnsafePointer/withUnsafeMutablePointer because the &x shorthand exists. If withUnsafePointer() is modified to take its argument by-value and create a pointer from it, I expect people will start using it in place of &x shorthand so they can use it with immutable values, without realizing that they're getting a different pointer back each time (whether they notice this depends on what they're using the pointer for).

If { x: UnsafePointer<Int> in ... }(&y) works, that's a cute replacement. Although it does move the &y reference to after the scope instead of before, which is a little unfortunate.

...huh. On a whim I tested the following and it shocked me by working.

var x: Int = 42
withExtendedLifetime(&x) { (x: UnsafeMutablePointer<Int>) in
   x.memory += 1
}
print(x) // prints 43

If we want to replace withUnsafe[Mutable]Pointer() with a common idiom, I'd recommend using withExtendedLifetime like that, because a) it guarantees the pointed-to value exists for the duration of the scope, and b) it already works.

-Kevin Ballard

-Dave

···

On Dec 15, 2015, at 6:46 PM, Kevin Ballard <kevin@sb.org> wrote:
On Tue, Dec 15, 2015, at 06:39 PM, Dave Abrahams wrote:

On Dec 15, 2015, at 6:33 PM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:
On Tue, Dec 15, 2015, at 03:03 PM, Joe Groff via swift-dev wrote:

Yeah but we want to move in the direction of making that more reliable, not less. I forget who but someone said in another thread that global variables can be reliably passed by-ref to functions that take pointers already (even though the Swift documentation does not guarantee this).

Come to think of it, what's the actual use-case for withUnsafePointer()? If a value is mutable, you can already use &x or withUnsafeMutablePointer(), and if it's immutable, you can't call withUnsafePointer() today anyway. The proposed change would just make withUnsafePointer() into the equivalent of `var x = value; callSomethingWith(&x)`. The only reason to really want a withUnsafePointer() function is if it can give you an UnsafePointer to an immutable value without copying it, but we can't do that. I'm inclined to say we should just get rid of withUnsafePointer() entirely, at least until such time as Swift has a way to pass immutable values by-ref.

-Kevin Ballard

···

On Tue, Dec 15, 2015, at 11:25 PM, Dave Abrahams wrote:

> On Dec 15, 2015, at 6:46 PM, Kevin Ballard <kevin@sb.org> wrote:
>
> On Tue, Dec 15, 2015, at 06:39 PM, Dave Abrahams wrote:
>>
>>> On Dec 15, 2015, at 6:33 PM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:
>>>
>>> On Tue, Dec 15, 2015, at 03:03 PM, Joe Groff via swift-dev wrote:
>>>>
>>>> Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.
>>>
>>> A change like that is going to break any code that relies on the inout optimization (where it uses call-by-reference instead of copy-in copy-out when possible). Yes, such code is in violation of Swift semantics today, but it does work.
>>
>> Two questions:
>>
>> 1. Don’t we want a withUnsafeMutablePointer for the mutating cases (where the inout optimization can take effect) anyway?
>
> I'm thinking here of cases like passing a context pointer to KVO. You're not actually mutating it, you just need a pointer that's the same every time you call the code.

Well, it is not possible to code a version of withUnsafePointer that makes that guarantee in Swift.

Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.

A change like that is going to break any code that relies on the inout optimization (where it uses call-by-reference instead of copy-in copy-out when possible). Yes, such code is in violation of Swift semantics today, but it does work.

Two questions:

1. Don’t we want a withUnsafeMutablePointer for the mutating cases (where the inout optimization can take effect) anyway?

I'm thinking here of cases like passing a context pointer to KVO. You're not actually mutating it, you just need a pointer that's the same every time you call the code.

Well, it is not possible to code a version of withUnsafePointer that makes that guarantee in Swift.

Yeah but we want to move in the direction of making that more reliable, not less.

I am not sure I agree with you. I would defer to John McCall on this one, but my understanding is that it's an explicit non-goal to make that guarantee.

I forget who but someone said in another thread that global variables can be reliably passed by-ref to functions that take pointers already (even though the Swift documentation does not guarantee this).

Come to think of it, what's the actual use-case for withUnsafePointer()?

I'm not sure we still have one that isn't covered by &x; that's my point.

If a value is mutable, you can already use &x or withUnsafeMutablePointer(), and if it's immutable, you can't call withUnsafePointer() today anyway. The proposed change would just make withUnsafePointer() into the equivalent of `var x = value; callSomethingWith(&x)`. The only reason to really want a withUnsafePointer() function is if it can give you an UnsafePointer to an immutable value without copying it, but we can't do that. I'm inclined to say we should just get rid of withUnsafePointer() entirely, at least until such time as Swift has a way to pass immutable values by-ref.

I'm inclined to agree. Proposal?

···

On Dec 15, 2015, at 11:51 PM, Kevin Ballard <kevin@sb.org> wrote:
On Tue, Dec 15, 2015, at 11:25 PM, Dave Abrahams wrote:

On Dec 15, 2015, at 6:46 PM, Kevin Ballard <kevin@sb.org> wrote:
On Tue, Dec 15, 2015, at 06:39 PM, Dave Abrahams wrote:

On Dec 15, 2015, at 6:33 PM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:
On Tue, Dec 15, 2015, at 03:03 PM, Joe Groff via swift-dev wrote:

>
>>
>>>
>>>>
>>>>>
>>>>>>
>>>>>> Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.
>>>>>
>>>>> A change like that is going to break any code that relies on the inout optimization (where it uses call-by-reference instead of copy-in copy-out when possible). Yes, such code is in violation of Swift semantics today, but it does work.
>>>>
>>>> Two questions:
>>>>
>>>> 1. Don’t we want a withUnsafeMutablePointer for the mutating cases (where the inout optimization can take effect) anyway?
>>>
>>> I'm thinking here of cases like passing a context pointer to KVO. You're not actually mutating it, you just need a pointer that's the same every time you call the code.
>>
>> Well, it is not possible to code a version of withUnsafePointer that makes that guarantee in Swift.
>
> Yeah but we want to move in the direction of making that more reliable, not less.

I am not sure I agree with you. I would defer to John McCall on this
one, but my understanding is that it's an explicit non-goal to make that
guarantee.

> I forget who but someone said in another thread that global variables can be reliably passed by-ref to functions that take pointers already (even though the Swift documentation does not guarantee this).
>
> Come to think of it, what's the actual use-case for withUnsafePointer()?

I'm not sure we still have one that isn't covered by &x; that's my point.

> If a value is mutable, you can already use &x or withUnsafeMutablePointer(), and if it's immutable, you can't call withUnsafePointer() today anyway. The proposed change would just make withUnsafePointer() into the equivalent of `var x = value; callSomethingWith(&x)`. The only reason to really want a withUnsafePointer() function is if it can give you an UnsafePointer to an immutable value without copying it, but we can't do that. I'm inclined to say we should just get rid of withUnsafePointer() entirely, at least until such time as Swift has a way to pass immutable values by-ref.

I'm inclined to agree. Proposal?

This line of thinking is hard for me to wrap my head around. I _get_
that in my original example I could have easily changed the code to be:

func profileName(profile: Int32) -> String? {
  var mutatingSelf = self
  let result = av_get_profile_name(&mutatingSelf, profile)
  return String.fromCString(result)
}

Now the self reference is still constant, thus no `mutating` keyword on
the func definition. From that stand point alone I suppose problem
solved.

I also get that Swift uses some sort of
copy-on-write/copy-on-modifcation for these sorts of cases therefore
there is not a significant performance overhead in the `var mutatingSelf
= self` because `av_get_profile_name` takes a `const AVCodec *` or
`UnsafePointer<AVCodec>` and thus is guaranteed to _not_ modify
mutatingSelf; thus saving the copy.

The thing that is hard for me to wrap my head around: how is any of that
obvious to the developer? I was always under the impression that one of
Swift's goals was to help a developer write clear and concise code. This
line of thinking at least partially feels like the opposite of that.

From my perspective, taking away `withUnsafePointer` we are actively

walking in the other direction of clear. Though I concede this might be
_slightly_ more concise. However, trading conciseness for clarity seems
like a bad idea.

A thought experiment that I went through when thinking about this
proposal was assume that `withUnsafeMutablePointer` or
`withUnsafePointer` had _never_ been in the Swift language. The proposed
code above still would not feel idiomatic to me. The code above
explicitly makes a mutable copy of self so that it can send it to a
function that guarantees it will not mutate it. :confused:

It feels like a hack/work-around for a limitation of the language.
Effectively this syntax says that Swift, by design, is asymmetric when
it comes to pointers. The language can _injest_ your C `const` params or
non-`const` params idiomatically but will not provide an idiomatic
method to send them back. Feels wrong.

All that having been said. I get that `const` and non-`const` are just
compiler abstractions but so are `var` and `let`. Furthermore, one or
more of my assumptions might be completely wrong and I am by no means a
language expert. Like I said at the start I don't want to make a fool of
myself. I already feel like I'm wading into neck deep water and might be
in over my head here shortly. My comments are meant to be from the
perspective of someone who knows nothing about how the Swift compiler
works; just how I'd _expect_ it to work as a consumer of the compiler.

···

On Wed, Dec 16, 2015, at 03:12 AM, Dave Abrahams via swift-dev wrote:

> On Dec 15, 2015, at 11:51 PM, Kevin Ballard <kevin@sb.org> wrote:
> On Tue, Dec 15, 2015, at 11:25 PM, Dave Abrahams wrote:
>>> On Dec 15, 2015, at 6:46 PM, Kevin Ballard <kevin@sb.org> wrote:
>>> On Tue, Dec 15, 2015, at 06:39 PM, Dave Abrahams wrote:
>>>>> On Dec 15, 2015, at 6:33 PM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:
>>>>> On Tue, Dec 15, 2015, at 03:03 PM, Joe Groff via swift-dev wrote:

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.

A change like that is going to break any code that relies on the inout optimization (where it uses call-by-reference instead of copy-in copy-out when possible). Yes, such code is in violation of Swift semantics today, but it does work.

Two questions:

1. Don’t we want a withUnsafeMutablePointer for the mutating cases (where the inout optimization can take effect) anyway?

I'm thinking here of cases like passing a context pointer to KVO. You're not actually mutating it, you just need a pointer that's the same every time you call the code.

Well, it is not possible to code a version of withUnsafePointer that makes that guarantee in Swift.

Yeah but we want to move in the direction of making that more reliable, not less.

I am not sure I agree with you. I would defer to John McCall on this one, but my understanding is that it's an explicit non-goal to make that guarantee.

I think it’s useful to be able to make this guarantee for some variables; I just don’t want it to be assumed for all variables. I’m okay with the idea that &myGlobalStoredVar will consistently yield the same pointer.

John.

···

On Dec 16, 2015, at 12:12 AM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:

On Dec 15, 2015, at 11:51 PM, Kevin Ballard <kevin@sb.org> wrote:
On Tue, Dec 15, 2015, at 11:25 PM, Dave Abrahams wrote:

On Dec 15, 2015, at 6:46 PM, Kevin Ballard <kevin@sb.org> wrote:
On Tue, Dec 15, 2015, at 06:39 PM, Dave Abrahams wrote:

On Dec 15, 2015, at 6:33 PM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:
On Tue, Dec 15, 2015, at 03:03 PM, Joe Groff via swift-dev wrote:

I forget who but someone said in another thread that global variables can be reliably passed by-ref to functions that take pointers already (even though the Swift documentation does not guarantee this).

Come to think of it, what's the actual use-case for withUnsafePointer()?

I'm not sure we still have one that isn't covered by &x; that's my point.

If a value is mutable, you can already use &x or withUnsafeMutablePointer(), and if it's immutable, you can't call withUnsafePointer() today anyway. The proposed change would just make withUnsafePointer() into the equivalent of `var x = value; callSomethingWith(&x)`. The only reason to really want a withUnsafePointer() function is if it can give you an UnsafePointer to an immutable value without copying it, but we can't do that. I'm inclined to say we should just get rid of withUnsafePointer() entirely, at least until such time as Swift has a way to pass immutable values by-ref.

I'm inclined to agree. Proposal?
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Sure, I'll write one up. I suspect that withUnsafePointer() / withUnsafeMutablePointer() are likely to be rarely used today, and most uses can probably be trivially replaced with just passing a &x ref, so this shouldn't be a big deal.

I'll also go ahead and write up one suggesting that we should allow for using &x when x is immutable when passing a parameter to a function that takes UnsafePointer.

-Kevin Ballard

···

On Wed, Dec 16, 2015, at 12:12 AM, Dave Abrahams wrote:

> Come to think of it, what's the actual use-case for withUnsafePointer()?

I'm not sure we still have one that isn't covered by &x; that's my point.

> If a value is mutable, you can already use &x or withUnsafeMutablePointer(), and if it's immutable, you can't call withUnsafePointer() today anyway. The proposed change would just make withUnsafePointer() into the equivalent of `var x = value; callSomethingWith(&x)`. The only reason to really want a withUnsafePointer() function is if it can give you an UnsafePointer to an immutable value without copying it, but we can't do that. I'm inclined to say we should just get rid of withUnsafePointer() entirely, at least until such time as Swift has a way to pass immutable values by-ref.

I'm inclined to agree. Proposal?

This was probably me; I say it a lot when explaining why other solutions don't work for KVO contexts. I can back Dave up that we're more likely to move away from guaranteeing persistent addresses for particular kinds of declarations. I could easily see it being an attribute of some kind, with all other declarations (including globals) never assumed to have persistent addresses. You just don't need them very often, and when you do need them you usually know it up front.

Jordan

···

On Dec 16, 2015, at 0:12 , Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:

Yeah but we want to move in the direction of making that more reliable, not less.

I am not sure I agree with you. I would defer to John McCall on this one, but my understanding is that it's an explicit non-goal to make that guarantee.

I forget who but someone said in another thread that global variables can be reliably passed by-ref to functions that take pointers already (even though the Swift documentation does not guarantee this).

Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.

A change like that is going to break any code that relies on the inout optimization (where it uses call-by-reference instead of copy-in copy-out when possible). Yes, such code is in violation of Swift semantics today, but it does work.

Two questions:

1. Don’t we want a withUnsafeMutablePointer for the mutating cases (where the inout optimization can take effect) anyway?

I'm thinking here of cases like passing a context pointer to KVO. You're not actually mutating it, you just need a pointer that's the same every time you call the code.

Well, it is not possible to code a version of withUnsafePointer that makes that guarantee in Swift.

Yeah but we want to move in the direction of making that more reliable, not less.

I am not sure I agree with you. I would defer to John McCall on this
one, but my understanding is that it's an explicit non-goal to make that
guarantee.

I forget who but someone said in another thread that global variables can be reliably passed by-ref to functions that take pointers already (even though the Swift documentation does not guarantee this).

Come to think of it, what's the actual use-case for withUnsafePointer()?

I'm not sure we still have one that isn't covered by &x; that's my point.

If a value is mutable, you can already use &x or withUnsafeMutablePointer(), and if it's immutable, you can't call withUnsafePointer() today anyway. The proposed change would just make withUnsafePointer() into the equivalent of `var x = value; callSomethingWith(&x)`. The only reason to really want a withUnsafePointer() function is if it can give you an UnsafePointer to an immutable value without copying it, but we can't do that. I'm inclined to say we should just get rid of withUnsafePointer() entirely, at least until such time as Swift has a way to pass immutable values by-ref.

I'm inclined to agree. Proposal?

This line of thinking is hard for me to wrap my head around. I _get_
that in my original example I could have easily changed the code to be:

func profileName(profile: Int32) -> String? {
var mutatingSelf = self
let result = av_get_profile_name(&mutatingSelf, profile)
return String.fromCString(result)
}

Now the self reference is still constant, thus no `mutating` keyword on
the func definition. From that stand point alone I suppose problem
solved.

Depending on the semantics of av_get_profile_name the above may actually be unsafe, because the compiler is allowed to destroy both self and mutatingSelf before you use result. You could make it safe by using withFixedLifetime, but that's sort of beside the point…

I was suggesting something more like this:

func profileName(profile: Int32) -> String? {
return { (pSelf: UnsafePointer)->String in String.fromCString( av_get_profile_name(pSelf, profile) }(&self) }
}

However, it appears we don't do the &x => UnsafePointer mapping for immutable x (feature request?), so that won't work. Also, it's obviously cumbersome, obtuse, and tricky to get right.

This is obviously better:

func profileName(profile: Int32) -> String? {
return withUnsafePointer(self){ String.fromCString( av_get_profile_name($0, profile) }
}

I also get that Swift uses some sort of
copy-on-write/copy-on-modifcation for these sorts of cases therefore
there is not a significant performance overhead in the `var mutatingSelf
= self` because `av_get_profile_name` takes a `const AVCodec *` or
`UnsafePointer<AVCodec>` and thus is guaranteed to _not_ modify
mutatingSelf; thus saving the copy.

No, there's no CoW here in general, but "var mutatingSelf = self" is always O(1) in Swift because copies are always O(1) in Swift (*)

(*) except for all of our existential boxes, which are eagerly copied; that should be considered a bug; we need CoW for these.

The thing that is hard for me to wrap my head around: how is any of that
obvious to the developer? I was always under the impression that one of
Swift's goals was to help a developer write clear and concise code. This
line of thinking at least partially feels like the opposite of that.
From my perspective, taking away `withUnsafePointer` we are actively
walking in the other direction of clear. Though I concede this might be
_slightly_ more concise. However, trading conciseness for clarity seems
like a bad idea.

I agree. There's a reason I introduced withUnsafePointer in the first place, and it was for cases like these.

A thought experiment that I went through when thinking about this
proposal was assume that `withUnsafeMutablePointer` or
`withUnsafePointer` had _never_ been in the Swift language. The proposed
code above still would not feel idiomatic to me. The code above
explicitly makes a mutable copy of self so that it can send it to a
function that guarantees it will not mutate it. :confused:

It feels like a hack/work-around for a limitation of the language.
Effectively this syntax says that Swift, by design, is asymmetric when
it comes to pointers. The language can _injest_ your C `const` params or
non-`const` params idiomatically but will not provide an idiomatic
method to send them back.

Not sure what you mean here.

Feels wrong.

All that having been said. I get that `const` and non-`const` are just
compiler abstractions but so are `var` and `let`. Furthermore, one or
more of my assumptions might be completely wrong and I am by no means a
language expert. Like I said at the start I don't want to make a fool of
myself. I already feel like I'm wading into neck deep water and might be
in over my head here shortly. My comments are meant to be from the
perspective of someone who knows nothing about how the Swift compiler
works; just how I'd _expect_ it to work as a consumer of the compiler.

You can relax, Ryan. You input is appreciated and I think your arguments are valid :-)

Thanks,

-Dave

···

On Dec 16, 2015, at 6:59 AM, Ryan Lovelett via swift-dev <swift-dev@swift.org> wrote:
On Wed, Dec 16, 2015, at 03:12 AM, Dave Abrahams via swift-dev wrote:

On Dec 15, 2015, at 11:51 PM, Kevin Ballard <kevin@sb.org> wrote:
On Tue, Dec 15, 2015, at 11:25 PM, Dave Abrahams wrote:

On Dec 15, 2015, at 6:46 PM, Kevin Ballard <kevin@sb.org> wrote:
On Tue, Dec 15, 2015, at 06:39 PM, Dave Abrahams wrote:

On Dec 15, 2015, at 6:33 PM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:
On Tue, Dec 15, 2015, at 03:03 PM, Joe Groff via swift-dev wrote:

Come to think of it, what's the actual use-case for withUnsafePointer()?

I'm not sure we still have one that isn't covered by &x; that's my point.

If a value is mutable, you can already use &x or withUnsafeMutablePointer(), and if it's immutable, you can't call withUnsafePointer() today anyway. The proposed change would just make withUnsafePointer() into the equivalent of `var x = value; callSomethingWith(&x)`. The only reason to really want a withUnsafePointer() function is if it can give you an UnsafePointer to an immutable value without copying it, but we can't do that. I'm inclined to say we should just get rid of withUnsafePointer() entirely, at least until such time as Swift has a way to pass immutable values by-ref.

I'm inclined to agree. Proposal?

Sure, I'll write one up. I suspect that withUnsafePointer() / withUnsafeMutablePointer() are likely to be rarely used today, and most uses can probably be trivially replaced with just passing a &x ref, so this shouldn't be a big deal.

We can't remove withUnsafe[Mutable]Pointer; as I mentioned to Dave, it's necessary to persist a pointer for more than one immediate call.

I'll also go ahead and write up one suggesting that we should allow for using &x when x is immutable when passing a parameter to a function that takes UnsafePointer.

That would be useful, though I would argue that the '&' shouldn't be necessary. We aren't C; '&x' means "this call mutates x", not "I'm taking a pointer to x".

-Joe

···

On Dec 16, 2015, at 11:24 AM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:
On Wed, Dec 16, 2015, at 12:12 AM, Dave Abrahams wrote:

I agree we can make that guarantee for globals, and maybe local stored properties as well. It's tricky to do that for potentially computed properties, which includes any class properties or properties outside your resilience domain, so though we can make a best effort not to wantonly change pointer identity across calls that take pointers to the same property, I don't think we can fully eliminate the need for an explicit withUnsafePointer form.

-Joe

···

On Dec 16, 2015, at 10:34 AM, John McCall via swift-dev <swift-dev@swift.org> wrote:

On Dec 16, 2015, at 12:12 AM, Dave Abrahams via swift-dev <swift-dev@swift.org> wrote:

On Dec 15, 2015, at 11:51 PM, Kevin Ballard <kevin@sb.org> wrote:
On Tue, Dec 15, 2015, at 11:25 PM, Dave Abrahams wrote:

On Dec 15, 2015, at 6:46 PM, Kevin Ballard <kevin@sb.org> wrote:

On Tue, Dec 15, 2015, at 06:39 PM, Dave Abrahams wrote:

On Dec 15, 2015, at 6:33 PM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:

On Tue, Dec 15, 2015, at 03:03 PM, Joe Groff via swift-dev wrote:

Yeah, it seems to me like a reasonable refinement for 'withUnsafePointer' to take an immutable parameter. Since this is a stdlib API change, you should suggest that on swift-evolution.

A change like that is going to break any code that relies on the inout optimization (where it uses call-by-reference instead of copy-in copy-out when possible). Yes, such code is in violation of Swift semantics today, but it does work.

Two questions:

1. Don’t we want a withUnsafeMutablePointer for the mutating cases (where the inout optimization can take effect) anyway?

I'm thinking here of cases like passing a context pointer to KVO. You're not actually mutating it, you just need a pointer that's the same every time you call the code.

Well, it is not possible to code a version of withUnsafePointer that makes that guarantee in Swift.

Yeah but we want to move in the direction of making that more reliable, not less.

I am not sure I agree with you. I would defer to John McCall on this one, but my understanding is that it's an explicit non-goal to make that guarantee.

I think it’s useful to be able to make this guarantee for some variables; I just don’t want it to be assumed for all variables. I’m okay with the idea that &myGlobalStoredVar will consistently yield the same pointer.

>
>>
>>> Come to think of it, what's the actual use-case for withUnsafePointer()?
>>
>> I'm not sure we still have one that isn't covered by &x; that's my point.
>>
>>> If a value is mutable, you can already use &x or withUnsafeMutablePointer(), and if it's immutable, you can't call withUnsafePointer() today anyway. The proposed change would just make withUnsafePointer() into the equivalent of `var x = value; callSomethingWith(&x)`. The only reason to really want a withUnsafePointer() function is if it can give you an UnsafePointer to an immutable value without copying it, but we can't do that. I'm inclined to say we should just get rid of withUnsafePointer() entirely, at least until such time as Swift has a way to pass immutable values by-ref.
>>
>> I'm inclined to agree. Proposal?
>
> Sure, I'll write one up. I suspect that withUnsafePointer() / withUnsafeMutablePointer() are likely to be rarely used today, and most uses can probably be trivially replaced with just passing a &x ref, so this shouldn't be a big deal.

We can't remove withUnsafe[Mutable]Pointer; as I mentioned to Dave, it's necessary to persist a pointer for more than one immediate call.

Rare cases like that can be covered by either declaring a nested function taking the pointer and calling it, calling an anonymous local closure of the right type, or even using withExtendedLifetime(&x) { (ptr: UnsafePointer<T>) in ... }.

> I'll also go ahead and write up one suggesting that we should allow for using &x when x is immutable when passing a parameter to a function that takes UnsafePointer.

That would be useful, though I would argue that the '&' shouldn't be necessary. We aren't C; '&x' means "this call mutates x", not "I'm taking a pointer to x".

Offhand, it feels surprising to allow `foo(x)` where foo takes an UnsafePointer.

Proposals submitted as

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/002861.html
https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151214/002863.html

-Kevin Ballard

···

On Wed, Dec 16, 2015, at 11:42 AM, Joe Groff wrote:

> On Dec 16, 2015, at 11:24 AM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:
> On Wed, Dec 16, 2015, at 12:12 AM, Dave Abrahams wrote:

These are all just withUnsafePointer with different spelling.

-Joe

···

On Dec 16, 2015, at 11:47 AM, Kevin Ballard <kevin@sb.org> wrote:

On Wed, Dec 16, 2015, at 11:42 AM, Joe Groff wrote:

On Dec 16, 2015, at 11:24 AM, Kevin Ballard via swift-dev <swift-dev@swift.org> wrote:

On Wed, Dec 16, 2015, at 12:12 AM, Dave Abrahams wrote:

Come to think of it, what's the actual use-case for withUnsafePointer()?

I'm not sure we still have one that isn't covered by &x; that's my point.

If a value is mutable, you can already use &x or withUnsafeMutablePointer(), and if it's immutable, you can't call withUnsafePointer() today anyway. The proposed change would just make withUnsafePointer() into the equivalent of `var x = value; callSomethingWith(&x)`. The only reason to really want a withUnsafePointer() function is if it can give you an UnsafePointer to an immutable value without copying it, but we can't do that. I'm inclined to say we should just get rid of withUnsafePointer() entirely, at least until such time as Swift has a way to pass immutable values by-ref.

I'm inclined to agree. Proposal?

Sure, I'll write one up. I suspect that withUnsafePointer() / withUnsafeMutablePointer() are likely to be rarely used today, and most uses can probably be trivially replaced with just passing a &x ref, so this shouldn't be a big deal.

We can't remove withUnsafe[Mutable]Pointer; as I mentioned to Dave, it's necessary to persist a pointer for more than one immediate call.

Rare cases like that can be covered by either declaring a nested function taking the pointer and calling it, calling an anonymous local closure of the right type, or even using withExtendedLifetime(&x) { (ptr: UnsafePointer<T>) in ... }.

That's true, but they all have the benefit of not requiring extra stdlib
functions. swiftdoc.org lists 36 global functions, a full 4 of which are
withUnsafePointer + variants. The existence of the functions also
implies that they're necessary to work with pointers (especially the
documentation). My suspicion is that nearly all current uses of these
functions can be replaced with &x refs with no change in functionality.
Even the stdlib only has a handful of uses of withUnsafeMutablePointer
(and none at all for withUnsafePointer), and looking through them
quickly, it looks like only the ones in public/core/Runtime.swift.gyb
require a workaround.

-Kevin

···

On Wed, Dec 16, 2015, at 11:49 AM, Joe Groff wrote:

On Dec 16, 2015, at 11:47 AM, Kevin Ballard <kevin@sb.org> wrote:

On Wed, Dec 16, 2015, at 11:42 AM, Joe Groff wrote:

On Dec 16, 2015, at 11:24 AM, Kevin Ballard via swift-dev <swift- >>>> dev@swift.org> wrote:

On Wed, Dec 16, 2015, at 12:12 AM, Dave Abrahams wrote:

Come to think of it, what's the actual use-case for
withUnsafePointer()?

I'm not sure we still have one that isn't covered by &x; that's my
point.

If a value is mutable, you can already use &x or
withUnsafeMutablePointer(), and if it's immutable, you can't call
withUnsafePointer() today anyway. The proposed change would just
make withUnsafePointer() into the equivalent of `var x = value;
callSomethingWith(&x)`. The only reason to really want a
withUnsafePointer() function is if it can give you an
UnsafePointer to an immutable value without copying it, but we
can't do that. I'm inclined to say we should just get rid of
withUnsafePointer() entirely, at least until such time as Swift
has a way to pass immutable values by-ref.

I'm inclined to agree. Proposal?

Sure, I'll write one up. I suspect that withUnsafePointer() /
withUnsafeMutablePointer() are likely to be rarely used today, and
most uses can probably be trivially replaced with just passing a &x
ref, so this shouldn't be a big deal.

We can't remove withUnsafe[Mutable]Pointer; as I mentioned to
Dave, it's necessary to persist a pointer for more than one
immediate call.

Rare cases like that can be covered by either declaring a nested
function taking the pointer and calling it, calling an anonymous
local closure of the right type, or even using
withExtendedLifetime(&x) { (ptr: UnsafePointer<T>) in ... }.

These are all just withUnsafePointer with different spelling.