[Review] SE-0055 Make unsafe pointer nullability explicit using Optional

Chris_Lattner · March 24, 2016, 6:00pm

Hello Swift community,

The review of "Make unsafe pointer nullability explicit using Optional" begins now and runs through March 29th. The proposal is available here:

swift-evolution/0055-optional-unsafe-pointers.md at master · apple/swift-evolution · GitHub

Reviews are an important part of the Swift evolution process. All reviews should be sent to the swift-evolution mailing list at:
https://lists.swift.org/mailman/listinfo/swift-evolution
or, if you would like to keep your feedback private, directly to the review manager.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  * What is your evaluation of the proposal?
  * Is the problem being addressed significant enough to warrant a change to Swift?
  * Does this proposal fit well with the feel and direction of Swift?
  * If you have you used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

swift-evolution/process.md at master · apple/swift-evolution · GitHub

Thank you,

-Chris Lattner
Review Manager

Brent_Royal-Gordon · March 24, 2016, 10:56pm

* What is your evaluation of the proposal?

I'm in favor. The ability of UnsafePointer to represent nil pointers has always been an odd duck, introducing an unmarked (and undocumented!) implicit trap in `memory` (now `pointee`), preventing the use of Optional constructs, and making Swift pointer types less expressive than C, which is just *wrong*. Making nullability explicit is a good move.

* Is the problem being addressed significant enough to warrant a change to Swift?

Yes.

* Does this proposal fit well with the feel and direction of Swift?

Yes.

* If you have you used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

The nullability specifiers introduced recently in clang have markedly removed my Objective-C code, both by better documenting system APIs and by forcing me to improve my own calls.

* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Read the proposal, participated in discussion.

···

--
Brent Royal-Gordon
Architechies

anandabits · March 25, 2016, 1:32am

* What is your evaluation of the proposal?

+1. This is a very nice improvement for code that needs to work directly with pointers.

* Is the problem being addressed significant enough to warrant a change to Swift?

Yes.

* Does this proposal fit well with the feel and direction of Swift?

Very much so.

* If you have you used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

It aligns well with languages that improve safety by explicitly modeling null.

* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Followed the discussion and read the proposal. I was very pleased to see Jordan propose this.

···

More information about the Swift evolution process is available at

https://github.com/apple/swift-evolution/blob/master/process.md

Thank you,

-Chris Lattner
Review Manager

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Richard_Ross · March 24, 2016, 9:01pm

* What is your evaluation of the proposal?

Currently, I'm -0.8 on this, for a few reasons.

Firstly, UnsafePointers are explicitly 'Unsafe' already. Making them more 'safe' could encourage more widespread usage of them, when in reality they're not necessary for 90% of cases.

Secondly, we will run into the IOU situation for C (and hopefully eventually C++) APIs, that now need to be audited and updated to reflect nullability status, which I'd wager that most existing headers *won't* be audited, especially on linux.

Thirdly, as mentioned in the prior discussion it's certainly possible on some platforms to remap the memory page at address 0x0 and make it usable to userland code. Even if we don't currently support any such platforms, we shouldn't lock ourselves into a situation where we need to be able to do this.

Finally, having nullable UnsafePointers currently is the only way from swift code to convert an UnsafePointer to an Int of its raw address, short of using another level of indirection:

let rawAddress: Int = UnsafePointer<UInt8>(nil).distanceTo(myPointer)

Unless the other proposal that's in the works to get the raw address of an UnsafePointer goes through (which I'm a supporter of), this would remove a very important functionality from the swift language, especially when dealing with lower level APIs that can take pointers as integers.

* Is the problem being addressed significant enough to warrant a change to Swift?

I think the goal of being able to write safer code is a noble one, and one we should strive for overall in the language. However, I don't believe that this is enough of an approach, nor is it inherently 'safer' (as nothing stops you from creating another invalid pointer that crashes rather than just NULL). Unless we plan on having some way to confirm that a pointer is, in fact, valid, I think we should leave UnsafePointer in its 'unsafe' state.

* Does this proposal fit well with the feel and direction of Swift?

I actually think it does. This will provide a swift-ier interface with raw pointers (and causing less confusion amongst beginners (who probably shouldn't be using UnsafePointer, but still) and improving API contracts). Again, however, UnsafePointer, while powerful, is rarely the proper tool for the job, and I think should be left in its current state - powerful, but difficult to use, to prevent a foot-gun scenario like we see in C++ and references (ask any C++ beginner about the lifetime of a reference...)

* If you have you used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

I'm actually unsure of any other languages that have explicit nullability on a raw pointer type. (Maybe rust has it? I'm not very familiar with rust).

* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

I've been following the proposal since day 1, and have put a lot of thought into it (one of my current projects uses UnsafePointer quite a bit). I'd like to think that I did a rather in-depth review of the subject.

If you've made it this far, thanks for reading my thesis on UnsafePointer :)

···

--
Richard

On Mar 24, 2016, at 11:00 AM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

Hello Swift community,

The review of "Make unsafe pointer nullability explicit using Optional" begins now and runs through March 29th. The proposal is available here:

  https://github.com/apple/swift-evolution/blob/master/proposals/0055-optional-unsafe-pointers.md

Reviews are an important part of the Swift evolution process. All reviews should be sent to the swift-evolution mailing list at:
  https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.swift.org_mailman_listinfo_swift-2Devolution&d=CwIGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=Ezje1IF3xGXfUMfsj4fBc7oM7BcJys1dhQ6psfXzLMU&m=_MgWpo4MYLScsGzB9CqrI-eCyzjjkWNyv8hjBeDANw0&s=HZtE3Eh63CLwxWA4MExIZyp1Dn7CsH2Te9eRrsFZNfE&e=
or, if you would like to keep your feedback private, directly to the review manager.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  * What is your evaluation of the proposal?
  * Is the problem being addressed significant enough to warrant a change to Swift?
  * Does this proposal fit well with the feel and direction of Swift?
  * If you have you used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

  https://github.com/apple/swift-evolution/blob/master/process.md

Thank you,

-Chris Lattner
Review Manager

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.swift.org_mailman_listinfo_swift-2Devolution&d=CwIGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=Ezje1IF3xGXfUMfsj4fBc7oM7BcJys1dhQ6psfXzLMU&m=_MgWpo4MYLScsGzB9CqrI-eCyzjjkWNyv8hjBeDANw0&s=HZtE3Eh63CLwxWA4MExIZyp1Dn7CsH2Te9eRrsFZNfE&e=

hlovatt · March 24, 2016, 9:20pm

* What is your evaluation of the proposal?

Good idea, much more logical

* Is the problem being addressed significant enough to warrant a
change to Swift?

Yes, it is an anomaly

* Does this proposal fit well with the feel and direction of Swift?

Yes, optional a are used to express nullability

* If you have you used other languages or libraries with a similar
feature, how do you feel that this proposal compares to those?

No

* How much effort did you put into your review? A glance, a quick
reading, or an in-depth study?

Followed discussions on swift-evolution

···

--
-- Howard.

Brent_Royal-Gordon · March 24, 2016, 10:50pm

Thirdly, as mentioned in the prior discussion it's certainly possible on some platforms to remap the memory page at address 0x0 and make it usable to userland code. Even if we don't currently support any such platforms, we shouldn't lock ourselves into a situation where we need to be able to do this.

I don't think this is mentioned in the proposal itself, but it came up in the discussion.

The C standard requires that there be a "null" pointer address which can be stored into any pointer but which can never actually be valid. It does *not* require that the null pointer address be 0x0. Most platforms do use 0x0, and clang doesn't support a non-0x0 null pointer, but this is not required by the C standard.

I believe Swift should mimic this behavior. On platforms where 0x0 is a valid address, Swift should not use 0x0 as the null pointer, but rather use some other address which isn't valid (perhaps ~0x0). Pointer types should treat this address as an unused value, which the enum machinery will then exploit to represent Optional.none.

For now, this design should probably just be documented somewhere, perhaps in a porting guide; actually implementing it is not really a priority.

Finally, having nullable UnsafePointers currently is the only way from swift code to convert an UnsafePointer to an Int of its raw address, short of using another level of indirection:

let rawAddress: Int = UnsafePointer<UInt8>(nil).distanceTo(myPointer)

Given what I discussed about `nil` not necessarily being 0x0, this construct is not necessarily valid anyway.

Besides, we *do* have a construct which converts between pointers and integers without any semantic confusion about `nil`—and it even expresses what it's doing more clearly than your `distanceTo` trick:

unsafeBitCast(pointer, Int.self)

I'm actually unsure of any other languages that have explicit nullability on a raw pointer type. (Maybe rust has it? I'm not very familiar with rust).

Apple variants of C, C++, and Objective-C now do. Actually, I believe they had some nullability control even before the recent keywords through __attribute__.

···

--
Brent Royal-Gordon
Architechies

John_McCall · March 25, 2016, 2:28am

It is already a known constraint that the representation of an imported type (ignoring bridging conversions) has to match up with the representation of the original C type. If we supported a target with a non-zero null value, we would have to ensure that Optionals of the appropriate pointer type were represented that way.

I'd actually be much more concerned about the pervasive assumptions about pointer representation in Clang and LLVM than I would be with Swift.

Now, it would probably be very annoying to support a platform where different pointer types had different null values, but the last of those died awhile ago, I believe.

All that said, I have negligible interest in putting any effort into supporting the ability to write to address zero in Swift.

John.

···

On Mar 24, 2016, at 3:50 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

Thirdly, as mentioned in the prior discussion it's certainly possible on some platforms to remap the memory page at address 0x0 and make it usable to userland code. Even if we don't currently support any such platforms, we shouldn't lock ourselves into a situation where we need to be able to do this.

I don't think this is mentioned in the proposal itself, but it came up in the discussion.

The C standard requires that there be a "null" pointer address which can be stored into any pointer but which can never actually be valid. It does *not* require that the null pointer address be 0x0. Most platforms do use 0x0, and clang doesn't support a non-0x0 null pointer, but this is not required by the C standard.

I believe Swift should mimic this behavior. On platforms where 0x0 is a valid address, Swift should not use 0x0 as the null pointer, but rather use some other address which isn't valid (perhaps ~0x0). Pointer types should treat this address as an unused value, which the enum machinery will then exploit to represent Optional.none.

Chris_Lattner · March 25, 2016, 4:43am

The last significant one IIRC was Itanium, in the case of C++ member pointers (themselves an odd beast), which had a different representation than data pointers (-1 is null).

-Chris

···

On Mar 24, 2016, at 7:28 PM, John McCall via swift-evolution <swift-evolution@swift.org> wrote:

I'd actually be much more concerned about the pervasive assumptions about pointer representation in Clang and LLVM than I would be with Swift.

Now, it would probably be very annoying to support a platform where different pointer types had different null values, but the last of those died awhile ago, I believe.

Dany_St-Amant · March 25, 2016, 2:02pm

Thirdly, as mentioned in the prior discussion it's certainly possible on some platforms to remap the memory page at address 0x0 and make it usable to userland code. Even if we don't currently support any such platforms, we shouldn't lock ourselves into a situation where we need to be able to do this.

I don't think this is mentioned in the proposal itself, but it came up in the discussion.

The C standard requires that there be a "null" pointer address which can be stored into any pointer but which can never actually be valid. It does *not* require that the null pointer address be 0x0. Most platforms do use 0x0, and clang doesn't support a non-0x0 null pointer, but this is not required by the C standard.

I believe Swift should mimic this behavior. On platforms where 0x0 is a valid address, Swift should not use 0x0 as the null pointer, but rather use some other address which isn't valid (perhaps ~0x0). Pointer types should treat this address as an unused value, which the enum machinery will then exploit to represent Optional.none.

In the low level world, there is no such thing as an invalid address; both 0x0 and ~0x0 are perfectly valid byte pointers. So using something else than 0x0 for Swift invalid pointer just shuffle the problem around.

Dany

···

Le 24 mars 2016 à 18:50, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> a écrit :

For now, this design should probably just be documented somewhere, perhaps in a porting guide; actually implementing it is not really a priority.

Finally, having nullable UnsafePointers currently is the only way from swift code to convert an UnsafePointer to an Int of its raw address, short of using another level of indirection:

let rawAddress: Int = UnsafePointer<UInt8>(nil).distanceTo(myPointer)

Given what I discussed about `nil` not necessarily being 0x0, this construct is not necessarily valid anyway.

Besides, we *do* have a construct which converts between pointers and integers without any semantic confusion about `nil`—and it even expresses what it's doing more clearly than your `distanceTo` trick:

unsafeBitCast(pointer, Int.self)

I'm actually unsure of any other languages that have explicit nullability on a raw pointer type. (Maybe rust has it? I'm not very familiar with rust).

Apple variants of C, C++, and Objective-C now do. Actually, I believe they had some nullability control even before the recent keywords through __attribute__.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Russ_Bishop1 · March 25, 2016, 11:05pm

You said it better than I could.

+1 on the proposal.

Russ

···

On Mar 24, 2016, at 3:56 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

   * What is your evaluation of the proposal?

I'm in favor. The ability of UnsafePointer to represent nil pointers has always been an odd duck, introducing an unmarked (and undocumented!) implicit trap in `memory` (now `pointee`), preventing the use of Optional constructs, and making Swift pointer types less expressive than C, which is just *wrong*. Making nullability explicit is a good move.

   * Is the problem being addressed significant enough to warrant a change to Swift?

Yes.

   * Does this proposal fit well with the feel and direction of Swift?

Yes.

   * If you have you used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

The nullability specifiers introduced recently in clang have markedly removed my Objective-C code, both by better documenting system APIs and by forcing me to improve my own calls.

   * How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Read the proposal, participated in discussion.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

John_McCall · March 25, 2016, 5:44am

That's not really target-specific; it's equally true for everybody using that C++ ABI, including us. But member pointers are a fundamentally different concept from C pointers, despite the name.

John.

···

On Mar 24, 2016, at 9:43 PM, Chris Lattner <clattner@apple.com> wrote:
On Mar 24, 2016, at 7:28 PM, John McCall via swift-evolution <swift-evolution@swift.org> wrote:

I'd actually be much more concerned about the pervasive assumptions about pointer representation in Clang and LLVM than I would be with Swift.

Now, it would probably be very annoying to support a platform where different pointer types had different null values, but the last of those died awhile ago, I believe.

The last significant one IIRC was Itanium, in the case of C++ member pointers (themselves an odd beast), which had a different representation than data pointers (-1 is null).

Brent_Royal-Gordon · March 25, 2016, 11:00pm

In the low level world, there is no such thing as an invalid address; both 0x0 and ~0x0 are perfectly valid byte pointers. So using something else than 0x0 for Swift invalid pointer just shuffle the problem around.

Let me state it this way: You cannot write a fully-conforming C compiler for a platform which does not have some way to represent an invalid pointer. However C does it, Swift can do the same thing.

···

--
Brent Royal-Gordon
Architechies

jeremyp · March 30, 2016, 10:11am

In the low level world, there is no such thing as an invalid address; both 0x0 and ~0x0 are perfectly valid byte pointers. So using something else than 0x0 for Swift invalid pointer just shuffle the problem around.

Let me state it this way: You cannot write a fully-conforming C compiler for a platform which does not have some way to represent an invalid pointer.

This is not true.

In both C99 and C11, the result of dereferencing an invalid pointer (of which the null pointer is one example) is undefined behaviour. This means it is perfectly fine for the null pointer to be represented by a bit pattern that is also a valid address and for the compiler not to bother generating a check that the pointer is not null.

However C does it, Swift can do the same thing.

C does it by sweeping the problem under the carpet of undefined behaviour.

···

On 25 Mar 2016, at 23:00, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

jrose · March 30, 2016, 6:00pm

Clever! I worked through this for the benefit of the list:

C11 6.3.2.3p3: If a null pointer constant is converted to a pointer type, the resulting pointer, called a null pointer, is guaranteed to compare unequal to a pointer to any object or function.

C11 6.3.2.3p4: Conversion of a null pointer to another pointer type yields a null pointer of that type. Any two null pointers shall compare equal.

So there must be at least one bit pattern recognizable as a null pointer…

C11 6.5.3.2p1: The operand of the unary & operator shall be either a function designator, the result of a or unary * operator, or an lvalue that designates an object that is not a bit-field and is not declared with the 'register' storage-class specifier.

C11 6.5.3.2p3: […] Otherwise [when not using or *], the result is a pointer to the object or function designated by its operand.

…and it must not have the same representation as a bit pattern of an object or function declared in C…

C11 6.5.3.2p4: If an invalid value has been assigned to the pointer, the behavior of the unary * operator is undefined.

C11 6.3.2.1p1: An lvalue is an expression (with an object type other than 'void') that potentially designates an object; if an lvalue does not designate an object when it is evaluated, the behavior is undefined.

…but as you say there's nothing stopping a particular compiler from giving the dereference defined behavior.

However, what this can't do is remove its use as a sentinel value. For example:

C11 7.24.5.1p3: The 'memchr' function returns a pointer to the located character, or a null pointer if the character does not occur in the object.

So ultimately I see Swift as being just as compatible as C in this case. Both languages designate a certain bit pattern for use as a placeholder. Swift is a little more aggressive in keeping you from loading from it, but on such a platform the standard library could certainly expose a "safely load from this optional pointer" builtin, which could then perhaps be exposed as a member on "Optional where Wrapped: _Pointer". (I'm not sure if that's the best way to do it, but it's an idea.)

As Chris has said previously, both Clang and LLVM themselves have a lot of assumptions about what can and can't be done with a null pointer, so porting to any such platform would include a fair amount of work simply extricating that logic and placing it behind a flag.

Best,
Jordan

···

On Mar 30, 2016, at 3:11 , Jeremy Pereira via swift-evolution <swift-evolution@swift.org> wrote:

On 25 Mar 2016, at 23:00, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

In the low level world, there is no such thing as an invalid address; both 0x0 and ~0x0 are perfectly valid byte pointers. So using something else than 0x0 for Swift invalid pointer just shuffle the problem around.

Let me state it this way: You cannot write a fully-conforming C compiler for a platform which does not have some way to represent an invalid pointer.

This is not true.

In both C99 and C11, the result of dereferencing an invalid pointer (of which the null pointer is one example) is undefined behaviour. This means it is perfectly fine for the null pointer to be represented by a bit pattern that is also a valid address and for the compiler not to bother generating a check that the pointer is not null.

John_McCall · March 30, 2016, 9:00pm

In the low level world, there is no such thing as an invalid address; both 0x0 and ~0x0 are perfectly valid byte pointers. So using something else than 0x0 for Swift invalid pointer just shuffle the problem around.

Let me state it this way: You cannot write a fully-conforming C compiler for a platform which does not have some way to represent an invalid pointer.

This is not true.

In both C99 and C11, the result of dereferencing an invalid pointer (of which the null pointer is one example) is undefined behaviour. This means it is perfectly fine for the null pointer to be represented by a bit pattern that is also a valid address and for the compiler not to bother generating a check that the pointer is not null.

This is not correct. Two pointers are not permitted to compare equal when one operand is a null pointer and the other is a pointer to an object.

John.

···

On Mar 30, 2016, at 3:11 AM, Jeremy Pereira via swift-evolution <swift-evolution@swift.org> wrote:

On 25 Mar 2016, at 23:00, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

However C does it, Swift can do the same thing.

C does it by sweeping the problem under the carpet of undefined behaviour.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

dabrahams · March 31, 2016, 6:19pm

In the low level world, there is no such thing as an invalid
address; both 0x0 and ~0x0 are perfectly valid byte pointers. So
using something else than 0x0 for Swift invalid pointer just
shuffle the problem around.

Let me state it this way: You cannot write a fully-conforming C
compiler for a platform which does not have some way to represent an
invalid pointer.

This is not true.

In both C99 and C11, the result of dereferencing an invalid pointer
(of which the null pointer is one example)

The null pointer is valid, it's just not dereferenceable. Invalid
pointers can't even be compared with one another.

···

on Wed Mar 30 2016, Jeremy Pereira <swift-evolution@swift.org> wrote:

On 25 Mar 2016, at 23:00, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

is undefined behaviour. This means it is perfectly fine for the null
pointer to be represented by a bit pattern that is also a valid
address and for the compiler not to bother generating a check that the
pointer is not null.

However C does it, Swift can do the same thing.

C does it by sweeping the problem under the carpet of undefined behaviour.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
Dave