Making pointer nullability explicit (using Optional)


(Jordan Rose) #1

Hey, everyone. If you're like me, you're sick of the fact that 'UnsafePointer<Int>' doesn't tell you whether or not the pointer can be nil. Why do we need to suffer this indignity when reference types—including function pointers!—can distinguish "present" from "absent" with the standard type 'Optional'? Well, good news: here's a proposal to make pointer nullability explicit. 'UnsafePointer<Int>?' can be null (nil), while 'UnsafePointer<Int>' cannot. Read on for details!

https://github.com/jrose-apple/swift-evolution/blob/optional-pointers/proposals/nnnn-optional-pointers.md

Bonus good news: I've implemented this locally and updated nearly all the tests already. Assuming this is accepting, the actual changes will go through review as a PR on GitHub, although it's mostly going to be one big mega-patch because the core change has a huge ripple effect.

Jordan

···

---

Make pointer nullability explicit using Optional

Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
Author(s): Jordan Rose <https://github.com/jrose-apple>
Status: Awaiting review
Review manager: TBD
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#introduction>Introduction

In Objective-C, pointers (whether to objects or to a non-object type) can be marked as nullable or nonnull, depending on whether the pointer value can ever be null. In Swift, however, there is no such way to make this distinction for pointers to non-object types: an UnsafePointer<Int> might be null, or it might never be.

We already have a way to describe this: Optionals. This proposal makes UnsafePointer<Int> represent a non-nullable pointer, and UnsafePointer<Int>? a nullable pointer. This also allows us to preserve information about pointer nullability available in header files for imported C and Objective-C APIs.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#motivation>Motivation

Today, UnsafePointer and friends suffer from a problem inherited from C: every pointer value could potentially be null, and code that works with pointers may or may not expect this. Failing to take the null pointer case into account can lead to assertion failures or crashes. For example, pretty much every operation on UnsafePointer itself requires a valid pointer (reading, writing, and initializing the pointee or performing arithmetic operations).

Fortunately, when a type has a single invalid value for which no operations are valid, Swift already has a solution: Optionals. Applying this to pointer types makes things very clear: if the type is non-optional, the pointer will never be null, and if it isoptional, the developer must take the "null pointer" case into account. This clarity has already been appreciated in Apple's Objective-C headers, which include nullability annotations for all pointer types (not just object pointers).

This change also allows developers working with pointers to take advantage of the many syntactic conveniences already built around optionals. For example, the standard library currently has a helper method on UnsafeMutablePointer called _setIfNonNil; with "optional pointers" this can be written simply and clearly:

ptr?.pointee = newValue
Finally, this change also reduces the number of types that conform to NilLiteralConvertible, a source of confusion for newcomers who (reasonably) associate nil directly with optionals. Currently the standard library includes the following NilLiteralConvertible types:

Optional
ImplicitlyUnwrappedOptional (subject of a separate proposal by Chris Willmore)
_OptionalNilComparisonType (used for optionalValue == nil)
UnsafePointer
UnsafeMutablePointer
AutoreleasingUnsafeMutablePointer
OpaquePointer
plus these Objective-C-specific types:

Selector
NSZone (only used to pass nil in Swift)
All of the italicized types would drop their conformance to NilLiteralConvertible; the "null pointer" would be represented by a nil optional of a particular type.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#proposed-solution>Proposed solution

Have the compiler assume that all values with pointer type (the italicized types listed above) are non-null. This allows the representation of Optional.none for a pointer type to be a null pointer value.

Drop NilLiteralConvertible conformance for all pointer types.

Teach the Clang importer to treat _Nullable pointers as Optional (and _Null_unspecified pointers as ImplicitlyUnwrappedOptional).

Deal with the fallout, i.e. adjust the compiler and the standard library to handle this new behavior.

Test migration and improve the migrator as necessary.

This proposal does not include the removal of the NilLiteralConvertible protocol altogether; besides still having two distinct optional types, we've seen people wanting to use nil for their own types (e.g. JSON values). (Changing this in the future is not out of the question; it's just out of scope for this proposal.)

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#detailed-design>Detailed design

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#api-changes>API Changes

Conformance to NilLiteralConvertible is removed from all types except Optional, ImplicitlyUnwrappedOptional, and _OptionalNilComparisonType, along with the implementation of init(nilLiteral:).

init(bitPattern: Int) and init(bitPattern: UInt) on all pointer types become failable; if the bit pattern represents a null pointer, nil is returned.

Process.unsafeArgv is a pointer to a null-terminated C array of C strings, so its type changes from UnsafeMutablePointer<UnsafeMutablePointer<Int8>> to UnsafeMutablePointer<UnsafeMutablePointer<Int8>?>, i.e. the inner pointer type becomes optional. It is then an error to access Process.unsafeArgv before entering main. (Previously you would get a null pointer value.)

NSErrorPointer becomes optional:

-public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>
+public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>?
A number of methods on String that came from NSString now have optional parameters:
   public func completePathIntoString(
- outputName: UnsafeMutablePointer<String> = nil,
+ outputName: UnsafeMutablePointer<String>? = nil,
     caseSensitive: Bool,
- matchesIntoArray: UnsafeMutablePointer<[String]> = nil,
+ matchesIntoArray: UnsafeMutablePointer<[String]>? = nil,
     filterTypes: [String]? = nil
   ) -> Int {
   public init(
     contentsOfFile path: String,
- usedEncoding: UnsafeMutablePointer<NSStringEncoding> = nil
+ usedEncoding: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {

   public init(
     contentsOfURL url: NSURL,
- usedEncoding enc: UnsafeMutablePointer<NSStringEncoding> = nil
+ usedEncoding enc: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {
   public func linguisticTags(
     in range: Range<Index>,
     scheme tagScheme: String,
     options opts: NSLinguisticTaggerOptions = [],
     orthography: NSOrthography? = nil,
- tokenRanges: UnsafeMutablePointer<[Range<Index>]> = nil
+ tokenRanges: UnsafeMutablePointer<[Range<Index>]>? = nil
   ) -> [String] {
NSZone's no-argument initializer is gone. (It probably should have been removed already as part of the Swift 3 naming cleanup.)

A small regression: optional pointers can no longer be passed using withVaList because it would require a conditional conformance to the CVarArg protocol. For now, using unsafeBitCast to reinterpret the optional pointer as an Int is the best alternative; Int has the same C variadic calling conventions as a pointer on all supported platforms.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#conversion-between-pointers>Conversion between pointers

Currently each pointer type has initializers of this form:

init<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>)
This simply makes a pointer with a different type but the same address as otherPointer. However, in making pointer nullability explicit, this now only converts non-nil pointers to non-nil pointers. In my experiments, this has led to this idiom becoming very common:

// Before:
let untypedPointer = UnsafePointer<Void>(ptr)

// After:
let untypedPointer = ptr.map(UnsafePointer<Void>.init)

// Usually the pointee type is actually inferred:
foo(ptr.map(UnsafePointer.init))
I consider this a bit more difficult to understand than the original code, at least at a glance. We should therefore add new initializers of the following form:

init?<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>?) {
  guard let nonnullPointer = otherPointer else {
    return nil
  }
  self.init(nonnullPointer)
}
The body is for explanation purposes only; we'll make sure the actual implementation does not require an extra comparison.

(This would need to be an overload rather than replacing the previous initializer because the "non-null-ness" should be preserved through the type conversion.)

The alternative is to leave this initializer out, and require the nil case to be explicitly handled or mapped away.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#open-issue-unsafebufferpointer>Open Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region with no ownership or lifetime semantics; it is logically a bare typed pointer (its baseAddress) and a length (count). For a buffer with 0 elements, however, there's no need to provide the address of allocated memory, since it can't be read from. Previously this case would be represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to access the base address: they need to consider the nil case explicitly, where previously they wouldn't have had to. There are several possibilities here, each with their own possible implementations:

Like UnsafePointer, UnsafeBufferPointer should always have a valid base address, even when the count is 0. An UnsafeBufferPointer with a potentially-nil base address should be optional.

UnsafeBufferPointer's initializer accepts an optional pointer and becomes failable, returning nil if the input pointer is nil.

UnsafeBufferPointer's initializer accepts an optional pointer and synthesizes a non-null aligned pointer value if given nil as a base address.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients such as withUnsafeBufferPointermust synthesize a non-null aligned pointer value if they do not have a valid pointer to provide.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients using withUnsafeBufferPointermust handle a nil buffer.

UnsafeBufferPointer should allow nil base addresses, i.e. the baseAddress property will be optional. Clients will need to handle this case explicitly.

UnsafeBufferPointer's initializer accepts an optional pointer, but no other changes are made.

UnsafeBufferPointer's initializer accepts an optional pointer. Additionally, any buffers initialized with a count of 0 will be canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer and length probably shouldn't require the pointer to be non-null, but if they do then perhaps there's a reason for it. It's also the least work.

Chris (Lattner) is leaning towards option (1ii), which treats UnsafeBufferPointer similar to UnsafePointer while not penalizing the common case of withUnsafeBufferPointer.
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#impact-on-existing-code>Impact on existing code

Any code that uses a pointer type (including Selector or NSZone) may be affected by this change. For the most part our existing logic to handle last year's nullability audit should cover this, but the implementer should test migration of several projects to see what issues might arise.

Anecdotally, in migrating the standard library to use this new logic I've been quite happy with nullability being made explicit. There are many places where a pointer really can't be nil.
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#alternatives-considered>Alternatives considered

The primary alternative here would be to leave everything as it is today, with UnsafePointer and friends including the null pointer as one of their normal values. This has obviously worked just fine for nearly two years of Swift, but it is leaving information on the table that can help avoid bugs, and is strange in a language that makes fluent use of Optional. As a fairly major source-breaking change, it is also something that we probably should do sooner rather than later in the language's evolution.


(Russ Bishop) #2

I’m very much +1 on this idea.

I consider this a bit more difficult to understand than the original code, at least at a glance. We should therefore add new initializers of the following form:

init?<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>?) {
  guard let nonnullPointer = otherPointer else {
    return nil
  }
  self.init(nonnullPointer)
}
The body is for explanation purposes only; we'll make sure the actual implementation does not require an extra comparison.

(This would need to be an overload rather than replacing the previous initializer because the "non-null-ness" should be preserved through the type conversion.)

The alternative is to leave this initializer out, and require the nil case to be explicitly handled or mapped away.

This should definitely be included. Everyone is used to “casting” by calling the appropriate initializers, and UnsafePointer<SpecificType>(someUnsafeVoidPointer) maps directly to what a C/Objective-C programmer would expect.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#open-issue-unsafebufferpointer>Open Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region with no ownership or lifetime semantics; it is logically a bare typed pointer (its baseAddress) and a length (count). For a buffer with 0 elements, however, there's no need to provide the address of allocated memory, since it can't be read from. Previously this case would be represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to access the base address: they need to consider the nil case explicitly, where previously they wouldn't have had to. There are several possibilities here, each with their own possible implementations:

Like UnsafePointer, UnsafeBufferPointer should always have a valid base address, even when the count is 0. An UnsafeBufferPointer with a potentially-nil base address should be optional.

UnsafeBufferPointer's initializer accepts an optional pointer and becomes failable, returning nil if the input pointer is nil.

UnsafeBufferPointer's initializer accepts an optional pointer and synthesizes a non-null aligned pointer value if given nil as a base address.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients such as withUnsafeBufferPointermust synthesize a non-null aligned pointer value if they do not have a valid pointer to provide.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients using withUnsafeBufferPointermust handle a nil buffer.

UnsafeBufferPointer should allow nil base addresses, i.e. the baseAddress property will be optional. Clients will need to handle this case explicitly.

UnsafeBufferPointer's initializer accepts an optional pointer, but no other changes are made.

UnsafeBufferPointer's initializer accepts an optional pointer. Additionally, any buffers initialized with a count of 0 will be canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer and length probably shouldn't require the pointer to be non-null, but if they do then perhaps there's a reason for it. It's also the least work.

Chris (Lattner) is leaning towards option (1ii), which treats UnsafeBufferPointer similar to UnsafePointer while not penalizing the common case of withUnsafeBufferPointer.

What’s the use of an UnsafeBufferPointer with zero count? Semantically that is making a claim that it can’t back up (“I have allocated memory at location X” which isn’t compatible with the idea of “zero count/size").

Without knowing more context I’d strongly favor (1i). If an array is empty the natural expectation for withUnsafeBufferPointer is you get UnsafeBufferPointer<Element>? = nil, which follows the behavior of the rest of the language and things like guard let make it trivial to handle properly. If someone really has a problem with it they can add ifUnsafeBufferPointer() that returns a non-optional pointer and skips executing the closure if the Array is empty (which is the behavior of your standard for loop).

Russ

···

On Mar 17, 2016, at 6:59 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:


(Félix Cloutier) #3

Fortunately, when a type has a single invalid value for which no operations are valid, Swift already has a solution: Optionals.

To me, this makes it sound like dereferencing an unsafe pointer is unsafe only if the pointer is nil. (Nil does have the merit that it's generally one of the only addresses known to be invalid, though.)

One thing that I would worry about, though, is the case where dereferencing address 0 is legal. My understanding is that Swift is aimed to be a systems language, and this case is relatively frequent on embedded devices.

Félix

···

Le 17 mars 2016 à 21:59:49, Jordan Rose via swift-evolution <swift-evolution@swift.org> a écrit :

Hey, everyone. If you're like me, you're sick of the fact that 'UnsafePointer<Int>' doesn't tell you whether or not the pointer can be nil. Why do we need to suffer this indignity when reference types—including function pointers!—can distinguish "present" from "absent" with the standard type 'Optional'? Well, good news: here's a proposal to make pointer nullability explicit. 'UnsafePointer<Int>?' can be null (nil), while 'UnsafePointer<Int>' cannot. Read on for details!

https://github.com/jrose-apple/swift-evolution/blob/optional-pointers/proposals/nnnn-optional-pointers.md

Bonus good news: I've implemented this locally and updated nearly all the tests already. Assuming this is accepting, the actual changes will go through review as a PR on GitHub, although it's mostly going to be one big mega-patch because the core change has a huge ripple effect.

Jordan

---

Make pointer nullability explicit using Optional

Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
Author(s): Jordan Rose <https://github.com/jrose-apple>
Status: Awaiting review
Review manager: TBD
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#introduction>Introduction

In Objective-C, pointers (whether to objects or to a non-object type) can be marked as nullable or nonnull, depending on whether the pointer value can ever be null. In Swift, however, there is no such way to make this distinction for pointers to non-object types: an UnsafePointer<Int> might be null, or it might never be.

We already have a way to describe this: Optionals. This proposal makes UnsafePointer<Int> represent a non-nullable pointer, and UnsafePointer<Int>? a nullable pointer. This also allows us to preserve information about pointer nullability available in header files for imported C and Objective-C APIs.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#motivation>Motivation

Today, UnsafePointer and friends suffer from a problem inherited from C: every pointer value could potentially be null, and code that works with pointers may or may not expect this. Failing to take the null pointer case into account can lead to assertion failures or crashes. For example, pretty much every operation on UnsafePointer itself requires a valid pointer (reading, writing, and initializing the pointee or performing arithmetic operations).

Fortunately, when a type has a single invalid value for which no operations are valid, Swift already has a solution: Optionals. Applying this to pointer types makes things very clear: if the type is non-optional, the pointer will never be null, and if it isoptional, the developer must take the "null pointer" case into account. This clarity has already been appreciated in Apple's Objective-C headers, which include nullability annotations for all pointer types (not just object pointers).

This change also allows developers working with pointers to take advantage of the many syntactic conveniences already built around optionals. For example, the standard library currently has a helper method on UnsafeMutablePointer called _setIfNonNil; with "optional pointers" this can be written simply and clearly:

ptr?.pointee = newValue
Finally, this change also reduces the number of types that conform to NilLiteralConvertible, a source of confusion for newcomers who (reasonably) associate nil directly with optionals. Currently the standard library includes the following NilLiteralConvertible types:

Optional
ImplicitlyUnwrappedOptional (subject of a separate proposal by Chris Willmore)
_OptionalNilComparisonType (used for optionalValue == nil)
UnsafePointer
UnsafeMutablePointer
AutoreleasingUnsafeMutablePointer
OpaquePointer
plus these Objective-C-specific types:

Selector
NSZone (only used to pass nil in Swift)
All of the italicized types would drop their conformance to NilLiteralConvertible; the "null pointer" would be represented by a nil optional of a particular type.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#proposed-solution>Proposed solution

Have the compiler assume that all values with pointer type (the italicized types listed above) are non-null. This allows the representation of Optional.none for a pointer type to be a null pointer value.

Drop NilLiteralConvertible conformance for all pointer types.

Teach the Clang importer to treat _Nullable pointers as Optional (and _Null_unspecified pointers as ImplicitlyUnwrappedOptional).

Deal with the fallout, i.e. adjust the compiler and the standard library to handle this new behavior.

Test migration and improve the migrator as necessary.

This proposal does not include the removal of the NilLiteralConvertible protocol altogether; besides still having two distinct optional types, we've seen people wanting to use nil for their own types (e.g. JSON values). (Changing this in the future is not out of the question; it's just out of scope for this proposal.)

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#detailed-design>Detailed design

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#api-changes>API Changes

Conformance to NilLiteralConvertible is removed from all types except Optional, ImplicitlyUnwrappedOptional, and _OptionalNilComparisonType, along with the implementation of init(nilLiteral:).

init(bitPattern: Int) and init(bitPattern: UInt) on all pointer types become failable; if the bit pattern represents a null pointer, nil is returned.

Process.unsafeArgv is a pointer to a null-terminated C array of C strings, so its type changes from UnsafeMutablePointer<UnsafeMutablePointer<Int8>> to UnsafeMutablePointer<UnsafeMutablePointer<Int8>?>, i.e. the inner pointer type becomes optional. It is then an error to access Process.unsafeArgv before entering main. (Previously you would get a null pointer value.)

NSErrorPointer becomes optional:

-public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>
+public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>?
A number of methods on String that came from NSString now have optional parameters:
   public func completePathIntoString(
- outputName: UnsafeMutablePointer<String> = nil,
+ outputName: UnsafeMutablePointer<String>? = nil,
     caseSensitive: Bool,
- matchesIntoArray: UnsafeMutablePointer<[String]> = nil,
+ matchesIntoArray: UnsafeMutablePointer<[String]>? = nil,
     filterTypes: [String]? = nil
   ) -> Int {
   public init(
     contentsOfFile path: String,
- usedEncoding: UnsafeMutablePointer<NSStringEncoding> = nil
+ usedEncoding: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {

   public init(
     contentsOfURL url: NSURL,
- usedEncoding enc: UnsafeMutablePointer<NSStringEncoding> = nil
+ usedEncoding enc: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {
   public func linguisticTags(
     in range: Range<Index>,
     scheme tagScheme: String,
     options opts: NSLinguisticTaggerOptions = [],
     orthography: NSOrthography? = nil,
- tokenRanges: UnsafeMutablePointer<[Range<Index>]> = nil
+ tokenRanges: UnsafeMutablePointer<[Range<Index>]>? = nil
   ) -> [String] {
NSZone's no-argument initializer is gone. (It probably should have been removed already as part of the Swift 3 naming cleanup.)

A small regression: optional pointers can no longer be passed using withVaList because it would require a conditional conformance to the CVarArg protocol. For now, using unsafeBitCast to reinterpret the optional pointer as an Int is the best alternative; Int has the same C variadic calling conventions as a pointer on all supported platforms.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#conversion-between-pointers>Conversion between pointers

Currently each pointer type has initializers of this form:

init<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>)
This simply makes a pointer with a different type but the same address as otherPointer. However, in making pointer nullability explicit, this now only converts non-nil pointers to non-nil pointers. In my experiments, this has led to this idiom becoming very common:

// Before:
let untypedPointer = UnsafePointer<Void>(ptr)

// After:
let untypedPointer = ptr.map(UnsafePointer<Void>.init)

// Usually the pointee type is actually inferred:
foo(ptr.map(UnsafePointer.init))
I consider this a bit more difficult to understand than the original code, at least at a glance. We should therefore add new initializers of the following form:

init?<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>?) {
  guard let nonnullPointer = otherPointer else {
    return nil
  }
  self.init(nonnullPointer)
}
The body is for explanation purposes only; we'll make sure the actual implementation does not require an extra comparison.

(This would need to be an overload rather than replacing the previous initializer because the "non-null-ness" should be preserved through the type conversion.)

The alternative is to leave this initializer out, and require the nil case to be explicitly handled or mapped away.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#open-issue-unsafebufferpointer>Open Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region with no ownership or lifetime semantics; it is logically a bare typed pointer (its baseAddress) and a length (count). For a buffer with 0 elements, however, there's no need to provide the address of allocated memory, since it can't be read from. Previously this case would be represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to access the base address: they need to consider the nil case explicitly, where previously they wouldn't have had to. There are several possibilities here, each with their own possible implementations:

Like UnsafePointer, UnsafeBufferPointer should always have a valid base address, even when the count is 0. An UnsafeBufferPointer with a potentially-nil base address should be optional.

UnsafeBufferPointer's initializer accepts an optional pointer and becomes failable, returning nil if the input pointer is nil.

UnsafeBufferPointer's initializer accepts an optional pointer and synthesizes a non-null aligned pointer value if given nil as a base address.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients such as withUnsafeBufferPointermust synthesize a non-null aligned pointer value if they do not have a valid pointer to provide.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients using withUnsafeBufferPointermust handle a nil buffer.

UnsafeBufferPointer should allow nil base addresses, i.e. the baseAddress property will be optional. Clients will need to handle this case explicitly.

UnsafeBufferPointer's initializer accepts an optional pointer, but no other changes are made.

UnsafeBufferPointer's initializer accepts an optional pointer. Additionally, any buffers initialized with a count of 0 will be canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer and length probably shouldn't require the pointer to be non-null, but if they do then perhaps there's a reason for it. It's also the least work.

Chris (Lattner) is leaning towards option (1ii), which treats UnsafeBufferPointer similar to UnsafePointer while not penalizing the common case of withUnsafeBufferPointer.
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#impact-on-existing-code>Impact on existing code

Any code that uses a pointer type (including Selector or NSZone) may be affected by this change. For the most part our existing logic to handle last year's nullability audit should cover this, but the implementer should test migration of several projects to see what issues might arise.

Anecdotally, in migrating the standard library to use this new logic I've been quite happy with nullability being made explicit. There are many places where a pointer really can't be nil.
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#alternatives-considered>Alternatives considered

The primary alternative here would be to leave everything as it is today, with UnsafePointer and friends including the null pointer as one of their normal values. This has obviously worked just fine for nearly two years of Swift, but it is leaving information on the table that can help avoid bugs, and is strange in a language that makes fluent use of Optional. As a fairly major source-breaking change, it is also something that we probably should do sooner rather than later in the language's evolution.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Rob Mayoff) #4

Option 2i is the only one that neither discards nor makes up information,
and it doesn't seem like 2i would impose a significant burden on users.​


(Michel Fortin) #5

`baseAddress` not returning the same value as was passed in the constructor seems bizarre to me. Surprising semantics is probably not what we should have in code that deals with unsafe pointers. That's why I feel like (2i) is the best solution.

My interpretation is that what Chris (Lattner) likes with (1ii) is a value that traps when `nil` but does not require explicit checking for `nil`. To me that sounds like an implicitly-unwrapped optional. Maybe we could have (2i) but with an IUO for `baseAddress`. That'd be much cleaner than silently replacing `nil` with an arbitrary non-nil underefrencable address.

···

Le 17 mars 2016 à 21:59, Jordan Rose via swift-evolution <swift-evolution@swift.org> a écrit :

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#open-issue-unsafebufferpointer>Open Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region with no ownership or lifetime semantics; it is logically a bare typed pointer (its baseAddress) and a length (count). For a buffer with 0 elements, however, there's no need to provide the address of allocated memory, since it can't be read from. Previously this case would be represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to access the base address: they need to consider the nil case explicitly, where previously they wouldn't have had to. There are several possibilities here, each with their own possible implementations:

Like UnsafePointer, UnsafeBufferPointer should always have a valid base address, even when the count is 0. An UnsafeBufferPointer with a potentially-nil base address should be optional.

UnsafeBufferPointer's initializer accepts an optional pointer and becomes failable, returning nil if the input pointer is nil.

UnsafeBufferPointer's initializer accepts an optional pointer and synthesizes a non-null aligned pointer value if given nil as a base address.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients such as withUnsafeBufferPointermust synthesize a non-null aligned pointer value if they do not have a valid pointer to provide.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients using withUnsafeBufferPointermust handle a nil buffer.

UnsafeBufferPointer should allow nil base addresses, i.e. the baseAddress property will be optional. Clients will need to handle this case explicitly.

UnsafeBufferPointer's initializer accepts an optional pointer, but no other changes are made.

UnsafeBufferPointer's initializer accepts an optional pointer. Additionally, any buffers initialized with a count of 0 will be canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer and length probably shouldn't require the pointer to be non-null, but if they do then perhaps there's a reason for it. It's also the least work.

Chris (Lattner) is leaning towards option (1ii), which treats UnsafeBufferPointer similar to UnsafePointer while not penalizing the common case of withUnsafeBufferPointer.

--
Michel Fortin
https://michelf.ca


(David Waite) #6

I’m +1 on this idea, but I think we should be more careful in the introduction. I don’t think at the swift level the concept of a null pointer should exist at all.

It is the platform and/or languages Swift integrates with which define a zero pointer as indicating ‘not pointing to anything’. It is the bridging between the platform or languages which should understand that a parameter or resulting value may optionally have on pointer, and how that pointer is represented by the underlying swift runtime.

I imagine that using RandomAccessIndexType methods to navigate to the zero value will not result in a precondition failure?

-DW

···

On Mar 17, 2016, at 7:59 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:

Hey, everyone. If you're like me, you're sick of the fact that 'UnsafePointer<Int>' doesn't tell you whether or not the pointer can be nil. Why do we need to suffer this indignity when reference types—including function pointers!—can distinguish "present" from "absent" with the standard type 'Optional'? Well, good news: here's a proposal to make pointer nullability explicit. 'UnsafePointer<Int>?' can be null (nil), while 'UnsafePointer<Int>' cannot. Read on for details!

https://github.com/jrose-apple/swift-evolution/blob/optional-pointers/proposals/nnnn-optional-pointers.md

Bonus good news: I've implemented this locally and updated nearly all the tests already. Assuming this is accepting, the actual changes will go through review as a PR on GitHub, although it's mostly going to be one big mega-patch because the core change has a huge ripple effect.

Jordan

---

Make pointer nullability explicit using Optional

Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
Author(s): Jordan Rose <https://github.com/jrose-apple>
Status: Awaiting review
Review manager: TBD
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#introduction>Introduction

In Objective-C, pointers (whether to objects or to a non-object type) can be marked as nullable or nonnull, depending on whether the pointer value can ever be null. In Swift, however, there is no such way to make this distinction for pointers to non-object types: an UnsafePointer<Int> might be null, or it might never be.

We already have a way to describe this: Optionals. This proposal makes UnsafePointer<Int> represent a non-nullable pointer, and UnsafePointer<Int>? a nullable pointer. This also allows us to preserve information about pointer nullability available in header files for imported C and Objective-C APIs.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#motivation>Motivation

Today, UnsafePointer and friends suffer from a problem inherited from C: every pointer value could potentially be null, and code that works with pointers may or may not expect this. Failing to take the null pointer case into account can lead to assertion failures or crashes. For example, pretty much every operation on UnsafePointer itself requires a valid pointer (reading, writing, and initializing the pointee or performing arithmetic operations).

Fortunately, when a type has a single invalid value for which no operations are valid, Swift already has a solution: Optionals. Applying this to pointer types makes things very clear: if the type is non-optional, the pointer will never be null, and if it isoptional, the developer must take the "null pointer" case into account. This clarity has already been appreciated in Apple's Objective-C headers, which include nullability annotations for all pointer types (not just object pointers).

This change also allows developers working with pointers to take advantage of the many syntactic conveniences already built around optionals. For example, the standard library currently has a helper method on UnsafeMutablePointer called _setIfNonNil; with "optional pointers" this can be written simply and clearly:

ptr?.pointee = newValue
Finally, this change also reduces the number of types that conform to NilLiteralConvertible, a source of confusion for newcomers who (reasonably) associate nil directly with optionals. Currently the standard library includes the following NilLiteralConvertible types:

Optional
ImplicitlyUnwrappedOptional (subject of a separate proposal by Chris Willmore)
_OptionalNilComparisonType (used for optionalValue == nil)
UnsafePointer
UnsafeMutablePointer
AutoreleasingUnsafeMutablePointer
OpaquePointer
plus these Objective-C-specific types:

Selector
NSZone (only used to pass nil in Swift)
All of the italicized types would drop their conformance to NilLiteralConvertible; the "null pointer" would be represented by a nil optional of a particular type.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#proposed-solution>Proposed solution

Have the compiler assume that all values with pointer type (the italicized types listed above) are non-null. This allows the representation of Optional.none for a pointer type to be a null pointer value.

Drop NilLiteralConvertible conformance for all pointer types.

Teach the Clang importer to treat _Nullable pointers as Optional (and _Null_unspecified pointers as ImplicitlyUnwrappedOptional).

Deal with the fallout, i.e. adjust the compiler and the standard library to handle this new behavior.

Test migration and improve the migrator as necessary.

This proposal does not include the removal of the NilLiteralConvertible protocol altogether; besides still having two distinct optional types, we've seen people wanting to use nil for their own types (e.g. JSON values). (Changing this in the future is not out of the question; it's just out of scope for this proposal.)

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#detailed-design>Detailed design

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#api-changes>API Changes

Conformance to NilLiteralConvertible is removed from all types except Optional, ImplicitlyUnwrappedOptional, and _OptionalNilComparisonType, along with the implementation of init(nilLiteral:).

init(bitPattern: Int) and init(bitPattern: UInt) on all pointer types become failable; if the bit pattern represents a null pointer, nil is returned.

Process.unsafeArgv is a pointer to a null-terminated C array of C strings, so its type changes from UnsafeMutablePointer<UnsafeMutablePointer<Int8>> to UnsafeMutablePointer<UnsafeMutablePointer<Int8>?>, i.e. the inner pointer type becomes optional. It is then an error to access Process.unsafeArgv before entering main. (Previously you would get a null pointer value.)

NSErrorPointer becomes optional:

-public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>
+public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>?
A number of methods on String that came from NSString now have optional parameters:
   public func completePathIntoString(
- outputName: UnsafeMutablePointer<String> = nil,
+ outputName: UnsafeMutablePointer<String>? = nil,
     caseSensitive: Bool,
- matchesIntoArray: UnsafeMutablePointer<[String]> = nil,
+ matchesIntoArray: UnsafeMutablePointer<[String]>? = nil,
     filterTypes: [String]? = nil
   ) -> Int {
   public init(
     contentsOfFile path: String,
- usedEncoding: UnsafeMutablePointer<NSStringEncoding> = nil
+ usedEncoding: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {

   public init(
     contentsOfURL url: NSURL,
- usedEncoding enc: UnsafeMutablePointer<NSStringEncoding> = nil
+ usedEncoding enc: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {
   public func linguisticTags(
     in range: Range<Index>,
     scheme tagScheme: String,
     options opts: NSLinguisticTaggerOptions = [],
     orthography: NSOrthography? = nil,
- tokenRanges: UnsafeMutablePointer<[Range<Index>]> = nil
+ tokenRanges: UnsafeMutablePointer<[Range<Index>]>? = nil
   ) -> [String] {
NSZone's no-argument initializer is gone. (It probably should have been removed already as part of the Swift 3 naming cleanup.)

A small regression: optional pointers can no longer be passed using withVaList because it would require a conditional conformance to the CVarArg protocol. For now, using unsafeBitCast to reinterpret the optional pointer as an Int is the best alternative; Int has the same C variadic calling conventions as a pointer on all supported platforms.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#conversion-between-pointers>Conversion between pointers

Currently each pointer type has initializers of this form:

init<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>)
This simply makes a pointer with a different type but the same address as otherPointer. However, in making pointer nullability explicit, this now only converts non-nil pointers to non-nil pointers. In my experiments, this has led to this idiom becoming very common:

// Before:
let untypedPointer = UnsafePointer<Void>(ptr)

// After:
let untypedPointer = ptr.map(UnsafePointer<Void>.init)

// Usually the pointee type is actually inferred:
foo(ptr.map(UnsafePointer.init))
I consider this a bit more difficult to understand than the original code, at least at a glance. We should therefore add new initializers of the following form:

init?<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>?) {
  guard let nonnullPointer = otherPointer else {
    return nil
  }
  self.init(nonnullPointer)
}
The body is for explanation purposes only; we'll make sure the actual implementation does not require an extra comparison.

(This would need to be an overload rather than replacing the previous initializer because the "non-null-ness" should be preserved through the type conversion.)

The alternative is to leave this initializer out, and require the nil case to be explicitly handled or mapped away.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#open-issue-unsafebufferpointer>Open Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region with no ownership or lifetime semantics; it is logically a bare typed pointer (its baseAddress) and a length (count). For a buffer with 0 elements, however, there's no need to provide the address of allocated memory, since it can't be read from. Previously this case would be represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to access the base address: they need to consider the nil case explicitly, where previously they wouldn't have had to. There are several possibilities here, each with their own possible implementations:

Like UnsafePointer, UnsafeBufferPointer should always have a valid base address, even when the count is 0. An UnsafeBufferPointer with a potentially-nil base address should be optional.

UnsafeBufferPointer's initializer accepts an optional pointer and becomes failable, returning nil if the input pointer is nil.

UnsafeBufferPointer's initializer accepts an optional pointer and synthesizes a non-null aligned pointer value if given nil as a base address.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients such as withUnsafeBufferPointermust synthesize a non-null aligned pointer value if they do not have a valid pointer to provide.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients using withUnsafeBufferPointermust handle a nil buffer.

UnsafeBufferPointer should allow nil base addresses, i.e. the baseAddress property will be optional. Clients will need to handle this case explicitly.

UnsafeBufferPointer's initializer accepts an optional pointer, but no other changes are made.

UnsafeBufferPointer's initializer accepts an optional pointer. Additionally, any buffers initialized with a count of 0 will be canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer and length probably shouldn't require the pointer to be non-null, but if they do then perhaps there's a reason for it. It's also the least work.

Chris (Lattner) is leaning towards option (1ii), which treats UnsafeBufferPointer similar to UnsafePointer while not penalizing the common case of withUnsafeBufferPointer.
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#impact-on-existing-code>Impact on existing code

Any code that uses a pointer type (including Selector or NSZone) may be affected by this change. For the most part our existing logic to handle last year's nullability audit should cover this, but the implementer should test migration of several projects to see what issues might arise.

Anecdotally, in migrating the standard library to use this new logic I've been quite happy with nullability being made explicit. There are many places where a pointer really can't be nil.
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#alternatives-considered>Alternatives considered

The primary alternative here would be to leave everything as it is today, with UnsafePointer and friends including the null pointer as one of their normal values. This has obviously worked just fine for nearly two years of Swift, but it is leaving information on the table that can help avoid bugs, and is strange in a language that makes fluent use of Optional. As a fairly major source-breaking change, it is also something that we probably should do sooner rather than later in the language's evolution.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jordan Rose) #7

I've recorded everyone's feedback so far (mostly on the UnsafeBufferPointer issue) and opened a pull request: https://github.com/apple/swift-evolution/pull/219. Thanks to everyone who's contributed to the draft!

Jordan

···

On Mar 17, 2016, at 18:59 , Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:

Hey, everyone. If you're like me, you're sick of the fact that 'UnsafePointer<Int>' doesn't tell you whether or not the pointer can be nil. Why do we need to suffer this indignity when reference types—including function pointers!—can distinguish "present" from "absent" with the standard type 'Optional'? Well, good news: here's a proposal to make pointer nullability explicit. 'UnsafePointer<Int>?' can be null (nil), while 'UnsafePointer<Int>' cannot. Read on for details!

https://github.com/jrose-apple/swift-evolution/blob/optional-pointers/proposals/nnnn-optional-pointers.md

Bonus good news: I've implemented this locally and updated nearly all the tests already. Assuming this is accepting, the actual changes will go through review as a PR on GitHub, although it's mostly going to be one big mega-patch because the core change has a huge ripple effect.

Jordan

---

Make pointer nullability explicit using Optional

Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
Author(s): Jordan Rose <https://github.com/jrose-apple>
Status: Awaiting review
Review manager: TBD
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#introduction>Introduction

In Objective-C, pointers (whether to objects or to a non-object type) can be marked as nullable or nonnull, depending on whether the pointer value can ever be null. In Swift, however, there is no such way to make this distinction for pointers to non-object types: an UnsafePointer<Int> might be null, or it might never be.

We already have a way to describe this: Optionals. This proposal makes UnsafePointer<Int> represent a non-nullable pointer, and UnsafePointer<Int>? a nullable pointer. This also allows us to preserve information about pointer nullability available in header files for imported C and Objective-C APIs.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#motivation>Motivation

Today, UnsafePointer and friends suffer from a problem inherited from C: every pointer value could potentially be null, and code that works with pointers may or may not expect this. Failing to take the null pointer case into account can lead to assertion failures or crashes. For example, pretty much every operation on UnsafePointer itself requires a valid pointer (reading, writing, and initializing the pointee or performing arithmetic operations).

Fortunately, when a type has a single invalid value for which no operations are valid, Swift already has a solution: Optionals. Applying this to pointer types makes things very clear: if the type is non-optional, the pointer will never be null, and if it isoptional, the developer must take the "null pointer" case into account. This clarity has already been appreciated in Apple's Objective-C headers, which include nullability annotations for all pointer types (not just object pointers).

This change also allows developers working with pointers to take advantage of the many syntactic conveniences already built around optionals. For example, the standard library currently has a helper method on UnsafeMutablePointer called _setIfNonNil; with "optional pointers" this can be written simply and clearly:

ptr?.pointee = newValue
Finally, this change also reduces the number of types that conform to NilLiteralConvertible, a source of confusion for newcomers who (reasonably) associate nil directly with optionals. Currently the standard library includes the following NilLiteralConvertible types:

Optional
ImplicitlyUnwrappedOptional (subject of a separate proposal by Chris Willmore)
_OptionalNilComparisonType (used for optionalValue == nil)
UnsafePointer
UnsafeMutablePointer
AutoreleasingUnsafeMutablePointer
OpaquePointer
plus these Objective-C-specific types:

Selector
NSZone (only used to pass nil in Swift)
All of the italicized types would drop their conformance to NilLiteralConvertible; the "null pointer" would be represented by a nil optional of a particular type.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#proposed-solution>Proposed solution

Have the compiler assume that all values with pointer type (the italicized types listed above) are non-null. This allows the representation of Optional.none for a pointer type to be a null pointer value.

Drop NilLiteralConvertible conformance for all pointer types.

Teach the Clang importer to treat _Nullable pointers as Optional (and _Null_unspecified pointers as ImplicitlyUnwrappedOptional).

Deal with the fallout, i.e. adjust the compiler and the standard library to handle this new behavior.

Test migration and improve the migrator as necessary.

This proposal does not include the removal of the NilLiteralConvertible protocol altogether; besides still having two distinct optional types, we've seen people wanting to use nil for their own types (e.g. JSON values). (Changing this in the future is not out of the question; it's just out of scope for this proposal.)

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#detailed-design>Detailed design

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#api-changes>API Changes

Conformance to NilLiteralConvertible is removed from all types except Optional, ImplicitlyUnwrappedOptional, and _OptionalNilComparisonType, along with the implementation of init(nilLiteral:).

init(bitPattern: Int) and init(bitPattern: UInt) on all pointer types become failable; if the bit pattern represents a null pointer, nil is returned.

Process.unsafeArgv is a pointer to a null-terminated C array of C strings, so its type changes from UnsafeMutablePointer<UnsafeMutablePointer<Int8>> to UnsafeMutablePointer<UnsafeMutablePointer<Int8>?>, i.e. the inner pointer type becomes optional. It is then an error to access Process.unsafeArgv before entering main. (Previously you would get a null pointer value.)

NSErrorPointer becomes optional:

-public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>
+public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>?
A number of methods on String that came from NSString now have optional parameters:
   public func completePathIntoString(
- outputName: UnsafeMutablePointer<String> = nil,
+ outputName: UnsafeMutablePointer<String>? = nil,
     caseSensitive: Bool,
- matchesIntoArray: UnsafeMutablePointer<[String]> = nil,
+ matchesIntoArray: UnsafeMutablePointer<[String]>? = nil,
     filterTypes: [String]? = nil
   ) -> Int {
   public init(
     contentsOfFile path: String,
- usedEncoding: UnsafeMutablePointer<NSStringEncoding> = nil
+ usedEncoding: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {

   public init(
     contentsOfURL url: NSURL,
- usedEncoding enc: UnsafeMutablePointer<NSStringEncoding> = nil
+ usedEncoding enc: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {
   public func linguisticTags(
     in range: Range<Index>,
     scheme tagScheme: String,
     options opts: NSLinguisticTaggerOptions = [],
     orthography: NSOrthography? = nil,
- tokenRanges: UnsafeMutablePointer<[Range<Index>]> = nil
+ tokenRanges: UnsafeMutablePointer<[Range<Index>]>? = nil
   ) -> [String] {
NSZone's no-argument initializer is gone. (It probably should have been removed already as part of the Swift 3 naming cleanup.)

A small regression: optional pointers can no longer be passed using withVaList because it would require a conditional conformance to the CVarArg protocol. For now, using unsafeBitCast to reinterpret the optional pointer as an Int is the best alternative; Int has the same C variadic calling conventions as a pointer on all supported platforms.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#conversion-between-pointers>Conversion between pointers

Currently each pointer type has initializers of this form:

init<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>)
This simply makes a pointer with a different type but the same address as otherPointer. However, in making pointer nullability explicit, this now only converts non-nil pointers to non-nil pointers. In my experiments, this has led to this idiom becoming very common:

// Before:
let untypedPointer = UnsafePointer<Void>(ptr)

// After:
let untypedPointer = ptr.map(UnsafePointer<Void>.init)

// Usually the pointee type is actually inferred:
foo(ptr.map(UnsafePointer.init))
I consider this a bit more difficult to understand than the original code, at least at a glance. We should therefore add new initializers of the following form:

init?<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>?) {
  guard let nonnullPointer = otherPointer else {
    return nil
  }
  self.init(nonnullPointer)
}
The body is for explanation purposes only; we'll make sure the actual implementation does not require an extra comparison.

(This would need to be an overload rather than replacing the previous initializer because the "non-null-ness" should be preserved through the type conversion.)

The alternative is to leave this initializer out, and require the nil case to be explicitly handled or mapped away.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#open-issue-unsafebufferpointer>Open Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region with no ownership or lifetime semantics; it is logically a bare typed pointer (its baseAddress) and a length (count). For a buffer with 0 elements, however, there's no need to provide the address of allocated memory, since it can't be read from. Previously this case would be represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to access the base address: they need to consider the nil case explicitly, where previously they wouldn't have had to. There are several possibilities here, each with their own possible implementations:

Like UnsafePointer, UnsafeBufferPointer should always have a valid base address, even when the count is 0. An UnsafeBufferPointer with a potentially-nil base address should be optional.

UnsafeBufferPointer's initializer accepts an optional pointer and becomes failable, returning nil if the input pointer is nil.

UnsafeBufferPointer's initializer accepts an optional pointer and synthesizes a non-null aligned pointer value if given nil as a base address.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients such as withUnsafeBufferPointermust synthesize a non-null aligned pointer value if they do not have a valid pointer to provide.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients using withUnsafeBufferPointermust handle a nil buffer.

UnsafeBufferPointer should allow nil base addresses, i.e. the baseAddress property will be optional. Clients will need to handle this case explicitly.

UnsafeBufferPointer's initializer accepts an optional pointer, but no other changes are made.

UnsafeBufferPointer's initializer accepts an optional pointer. Additionally, any buffers initialized with a count of 0 will be canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer and length probably shouldn't require the pointer to be non-null, but if they do then perhaps there's a reason for it. It's also the least work.

Chris (Lattner) is leaning towards option (1ii), which treats UnsafeBufferPointer similar to UnsafePointer while not penalizing the common case of withUnsafeBufferPointer.
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#impact-on-existing-code>Impact on existing code

Any code that uses a pointer type (including Selector or NSZone) may be affected by this change. For the most part our existing logic to handle last year's nullability audit should cover this, but the implementer should test migration of several projects to see what issues might arise.

Anecdotally, in migrating the standard library to use this new logic I've been quite happy with nullability being made explicit. There are many places where a pointer really can't be nil.
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#alternatives-considered>Alternatives considered

The primary alternative here would be to leave everything as it is today, with UnsafePointer and friends including the null pointer as one of their normal values. This has obviously worked just fine for nearly two years of Swift, but it is leaving information on the table that can help avoid bugs, and is strange in a language that makes fluent use of Optional. As a fairly major source-breaking change, it is also something that we probably should do sooner rather than later in the language's evolution.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Howard Lovatt) #8

+1

···

On Friday, 18 March 2016, Jordan Rose via swift-evolution < swift-evolution@swift.org> wrote:

Hey, everyone. If you're like me, you're sick of the fact that
'UnsafePointer<Int>' doesn't tell you whether or not the pointer can be
nil. Why do we need to suffer this indignity when reference types—including
function pointers!—can distinguish "present" from "absent" with the
standard type 'Optional'? Well, good news: here's a proposal to make
pointer nullability explicit. 'UnsafePointer<Int>?' can be null (nil),
while 'UnsafePointer<Int>' cannot. Read on for details!

https://github.com/jrose-apple/swift-evolution/blob/optional-pointers/proposals/nnnn-optional-pointers.md

Bonus good news: I've implemented this locally and updated nearly all the
tests already. Assuming this is accepting, the actual changes will go
through review as a PR on GitHub, although it's mostly going to be one big
mega-patch because the core change has a huge ripple effect.

Jordan

---

Make pointer nullability explicit using Optional

   - Proposal: SE-NNNN
   <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
   - Author(s): Jordan Rose <https://github.com/jrose-apple>
   - Status: *Awaiting review*
   - Review manager: TBD

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#introduction>
Introduction

In Objective-C, pointers (whether to objects or to a non-object type) can
be marked as nullable or nonnull, depending on whether the pointer value
can ever be null. In Swift, however, there is no such way to make this
distinction for pointers to non-object types: an UnsafePointer<Int> might
be null, or it might never be.

We already have a way to describe this: Optionals. This proposal makes
UnsafePointer<Int> represent a non-nullable pointer, and
UnsafePointer<Int>? a nullable pointer. This also allows us to preserve
information about pointer nullability available in header files for
imported C and Objective-C APIs.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#motivation>
Motivation

Today, UnsafePointer and friends suffer from a problem inherited from C:
every pointer value could potentially be null, and code that works with
pointers may or may not expect this. Failing to take the null pointer case
into account can lead to assertion failures or crashes. For example, pretty
much every operation on UnsafePointer itself requires a valid pointer
(reading, writing, and initializing the pointee or performing arithmetic
operations).

Fortunately, when a type has a single invalid value for which no
operations are valid, Swift already has a solution: Optionals. Applying
this to pointer types makes things very clear: if the type is non-optional,
the pointer will never be null, and if it *is*optional, the developer
must take the "null pointer" case into account. This clarity has already
been appreciated in Apple's Objective-C headers, which include nullability
annotations for all pointer types (not just object pointers).

This change also allows developers working with pointers to take advantage
of the many syntactic conveniences already built around optionals. For
example, the standard library currently has a helper method on
UnsafeMutablePointer called _setIfNonNil; with "optional pointers" this
can be written simply and clearly:

ptr?.pointee = newValue

Finally, this change also reduces the number of types that conform to
NilLiteralConvertible, a source of confusion for newcomers who (reasonably)
associate nil directly with optionals. Currently the standard library
includes the following NilLiteralConvertible types:

   - Optional
   - ImplicitlyUnwrappedOptional (subject of a separate proposal by Chris
   Willmore)
   - _OptionalNilComparisonType (used for optionalValue == nil)
   - *UnsafePointer*
   - *UnsafeMutablePointer*
   - *AutoreleasingUnsafeMutablePointer*
   - *OpaquePointer*

plus these Objective-C-specific types:

   - *Selector*
   - *NSZone* (only used to pass nil in Swift)

All of the italicized types would drop their conformance to
NilLiteralConvertible; the "null pointer" would be represented by a nil
optional of a particular type.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#proposed-solution>Proposed
solution

   1.

   Have the compiler assume that all values with pointer type (the
   italicized types listed above) are non-null. This allows the representation
   of Optional.none for a pointer type to be a null pointer value.
   2.

   Drop NilLiteralConvertible conformance for all pointer types.
   3.

   Teach the Clang importer to treat _Nullable pointers as Optional (and
   _Null_unspecified pointers as ImplicitlyUnwrappedOptional).
   4.

   Deal with the fallout, i.e. adjust the compiler and the standard
   library to handle this new behavior.
   5.

   Test migration and improve the migrator as necessary.

This proposal does not include the removal of the NilLiteralConvertible
protocol altogether; besides still having two distinct optional types,
we've seen people wanting to use nil for their own types (e.g. JSON
values). (Changing this in the future is not out of the question; it's just
out of scope for this proposal.)

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#detailed-design>Detailed
design
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#api-changes>API
Changes

   -

   Conformance to NilLiteralConvertible is removed from all types except
   Optional, ImplicitlyUnwrappedOptional, and _OptionalNilComparisonType,
   along with the implementation of init(nilLiteral:).
   -

   init(bitPattern: Int) and init(bitPattern: UInt) on all pointer types
   become failable; if the bit pattern represents a null pointer, nil is
   returned.
   -

   Process.unsafeArgv is a pointer to a null-terminated C array of C
   strings, so its type changes from
   UnsafeMutablePointer<UnsafeMutablePointer<Int8>> to
   UnsafeMutablePointer<UnsafeMutablePointer<Int8>?>, i.e. the inner
   pointer type becomes optional. It is then an error to access
   Process.unsafeArgv before entering main. (Previously you would get a
   null pointer value.)
   -

   NSErrorPointer becomes optional:

-public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>+public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>?

   - A number of methods on String that came from NSString now have
   optional parameters:

   public func completePathIntoString(- outputName: UnsafeMutablePointer<String> = nil,+ outputName: UnsafeMutablePointer<String>? = nil,
     caseSensitive: Bool,- matchesIntoArray: UnsafeMutablePointer<[String]> = nil,+ matchesIntoArray: UnsafeMutablePointer<[String]>? = nil,
     filterTypes: [String]? = nil
   ) -> Int {

   public init(
     contentsOfFile path: String,- usedEncoding: UnsafeMutablePointer<NSStringEncoding> = nil+ usedEncoding: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {

   public init(
     contentsOfURL url: NSURL,- usedEncoding enc: UnsafeMutablePointer<NSStringEncoding> = nil+ usedEncoding enc: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {

   public func linguisticTags(
     in range: Range<Index>,
     scheme tagScheme: String,
     options opts: NSLinguisticTaggerOptions = [],
     orthography: NSOrthography? = nil,- tokenRanges: UnsafeMutablePointer<[Range<Index>]> = nil+ tokenRanges: UnsafeMutablePointer<[Range<Index>]>? = nil
   ) -> [String] {

   -

   NSZone's no-argument initializer is gone. (It probably should have
   been removed already as part of the Swift 3 naming cleanup.)
   -

   A small regression: optional pointers can no longer be passed using
   withVaList because it would require a conditional conformance to the
   CVarArg protocol. For now, using unsafeBitCast to reinterpret the
   optional pointer as an Int is the best alternative; Int has the same C
   variadic calling conventions as a pointer on all supported platforms.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#conversion-between-pointers>Conversion
between pointers

Currently each pointer type has initializers of this form:

init<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>)

This simply makes a pointer with a different type but the same address as
otherPointer. However, in making pointer nullability explicit, this now
only converts non-nil pointers to non-nil pointers. In my experiments, this
has led to this idiom becoming very common:

// Before:let untypedPointer = UnsafePointer<Void>(ptr)
// After:let untypedPointer = ptr.map(UnsafePointer<Void>.init)
// Usually the pointee type is actually inferred:
foo(ptr.map(UnsafePointer.init))

I consider this a bit more difficult to understand than the original code,
at least at a glance. We should therefore add new initializers of the
following form:

init?<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>?) {
  guard let nonnullPointer = otherPointer else {
    return nil
  }
  self.init(nonnullPointer)
}

The body is for explanation purposes only; we'll make sure the actual
implementation does not require an extra comparison.

(This would need to be an overload rather than replacing the previous
initializer because the "non-null-ness" should be preserved through the
type conversion.)

The alternative is to leave this initializer out, and require the nil case
to be explicitly handled or mapped away.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#open-issue-unsafebufferpointer>Open
Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region
with no ownership or lifetime semantics; it is logically a bare typed
pointer (its baseAddress) and a length (count). For a buffer with 0
elements, however, there's no need to provide the address of allocated
memory, since it can't be read from. Previously this case would be
represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to
access the base address: they need to consider the nil case explicitly,
where previously they wouldn't have had to. There are several possibilities
here, each with their own possible implementations:

   1.

   Like UnsafePointer, UnsafeBufferPointer should always have a valid
   base address, even when the count is 0. An UnsafeBufferPointer with a
   potentially-nil base address should be optional.
   1.

      UnsafeBufferPointer's initializer accepts an optional pointer and
      becomes failable, returning nil if the input pointer is nil.
      2.

      UnsafeBufferPointer's initializer accepts an optional pointer and
      synthesizes a non-null aligned pointer value if given nil as a base address.
      3.

      UnsafeBufferPointer's initializer only accepts non-optional
      pointers. Clients such as withUnsafeBufferPointermust synthesize a
      non-null aligned pointer value if they do not have a valid pointer to
      provide.
      4.

      UnsafeBufferPointer's initializer only accepts non-optional
      pointers. Clients *using* withUnsafeBufferPointermust handle a nil
      buffer.
      2.

   UnsafeBufferPointer should allow nil base addresses, i.e. the
   baseAddress property will be optional. Clients will need to handle
   this case explicitly.
   1.

      UnsafeBufferPointer's initializer accepts an optional pointer, but
      no other changes are made.
      2.

      UnsafeBufferPointer's initializer accepts an optional pointer.
      Additionally, any buffers initialized with a count of 0 will be
      canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer
and length probably shouldn't require the pointer to be non-null, but if
they do then perhaps there's a reason for it. It's also the least work.
Chris (Lattner) is leaning towards option (1ii), which treats
UnsafeBufferPointer similar to UnsafePointer while not penalizing the
common case of withUnsafeBufferPointer.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#impact-on-existing-code>Impact
on existing code

Any code that uses a pointer type (including Selector or NSZone) may be
affected by this change. For the most part our existing logic to handle
last year's nullability audit should cover this, but the implementer should
test migration of several projects to see what issues might arise.

Anecdotally, in migrating the standard library to use this new logic I've
been quite happy with nullability being made explicit. There are many
places where a pointer really *can't* be nil.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#alternatives-considered>Alternatives
considered
The primary alternative here would be to leave everything as it is today,
with UnsafePointer and friends including the null pointer as one of their
normal values. This has obviously worked just fine for nearly two years of
Swift, but it is leaving information on the table that can help avoid bugs,
and is strange in a language that makes fluent use of Optional. As a fairly
major source-breaking change, it is also something that we probably should
do sooner rather than later in the language's evolution.

--
-- Howard.


(Chris Lattner) #9

Fortunately, when a type has a single invalid value for which no operations are valid, Swift already has a solution: Optionals.

To me, this makes it sound like dereferencing an unsafe pointer is unsafe only if the pointer is nil. (Nil does have the merit that it's generally one of the only addresses known to be invalid, though.)

One thing that I would worry about, though, is the case where dereferencing address 0 is legal. My understanding is that Swift is aimed to be a systems language, and this case is relatively frequent on embedded devices.

This is definitely an important case to consider, but I don't think this is a problem for the proposal.

C compilers for these sorts of embedded systems have two choices: they can either reserve 1 byte for the null value (wasting a byte of physmem) or they can use a null value with a representation that is not equal to zero.

In either case, the null value (whatever it is) maps to the swift none case. Everything works out properly.

-Chris

···

On Mar 17, 2016, at 8:21 PM, Félix Cloutier via swift-evolution <swift-evolution@swift.org> wrote:

Félix

Le 17 mars 2016 à 21:59:49, Jordan Rose via swift-evolution <swift-evolution@swift.org> a écrit :

Hey, everyone. If you're like me, you're sick of the fact that 'UnsafePointer<Int>' doesn't tell you whether or not the pointer can be nil. Why do we need to suffer this indignity when reference types—including function pointers!—can distinguish "present" from "absent" with the standard type 'Optional'? Well, good news: here's a proposal to make pointer nullability explicit. 'UnsafePointer<Int>?' can be null (nil), while 'UnsafePointer<Int>' cannot. Read on for details!

https://github.com/jrose-apple/swift-evolution/blob/optional-pointers/proposals/nnnn-optional-pointers.md

Bonus good news: I've implemented this locally and updated nearly all the tests already. Assuming this is accepting, the actual changes will go through review as a PR on GitHub, although it's mostly going to be one big mega-patch because the core change has a huge ripple effect.

Jordan

---

Make pointer nullability explicit using Optional
Proposal: SE-NNNN
Author(s): Jordan Rose
Status: Awaiting review
Review manager: TBD
Introduction

In Objective-C, pointers (whether to objects or to a non-object type) can be marked as nullable or nonnull, depending on whether the pointer value can ever be null. In Swift, however, there is no such way to make this distinction for pointers to non-object types: an UnsafePointer<Int> might be null, or it might never be.

We already have a way to describe this: Optionals. This proposal makes UnsafePointer<Int> represent a non-nullable pointer, and UnsafePointer<Int>? a nullable pointer. This also allows us to preserve information about pointer nullability available in header files for imported C and Objective-C APIs.

Motivation

Today, UnsafePointer and friends suffer from a problem inherited from C: every pointer value could potentially be null, and code that works with pointers may or may not expect this. Failing to take the null pointer case into account can lead to assertion failures or crashes. For example, pretty much every operation on UnsafePointer itself requires a valid pointer (reading, writing, and initializing the pointee or performing arithmetic operations).

Fortunately, when a type has a single invalid value for which no operations are valid, Swift already has a solution: Optionals. Applying this to pointer types makes things very clear: if the type is non-optional, the pointer will never be null, and if it isoptional, the developer must take the "null pointer" case into account. This clarity has already been appreciated in Apple's Objective-C headers, which include nullability annotations for all pointer types (not just object pointers).

This change also allows developers working with pointers to take advantage of the many syntactic conveniences already built around optionals. For example, the standard library currently has a helper method on UnsafeMutablePointer called _setIfNonNil; with "optional pointers" this can be written simply and clearly:

ptr?.pointee = newValue
Finally, this change also reduces the number of types that conform to NilLiteralConvertible, a source of confusion for newcomers who (reasonably) associate nil directly with optionals. Currently the standard library includes the following NilLiteralConvertible types:

Optional
ImplicitlyUnwrappedOptional (subject of a separate proposal by Chris Willmore)
_OptionalNilComparisonType (used for optionalValue == nil)
UnsafePointer
UnsafeMutablePointer
AutoreleasingUnsafeMutablePointer
OpaquePointer
plus these Objective-C-specific types:

Selector
NSZone (only used to pass nil in Swift)
All of the italicized types would drop their conformance to NilLiteralConvertible; the "null pointer" would be represented by a nil optional of a particular type.

Proposed solution

Have the compiler assume that all values with pointer type (the italicized types listed above) are non-null. This allows the representation of Optional.none for a pointer type to be a null pointer value.

Drop NilLiteralConvertible conformance for all pointer types.

Teach the Clang importer to treat _Nullable pointers as Optional (and _Null_unspecified pointers as ImplicitlyUnwrappedOptional).

Deal with the fallout, i.e. adjust the compiler and the standard library to handle this new behavior.

Test migration and improve the migrator as necessary.

This proposal does not include the removal of the NilLiteralConvertible protocol altogether; besides still having two distinct optional types, we've seen people wanting to use nil for their own types (e.g. JSON values). (Changing this in the future is not out of the question; it's just out of scope for this proposal.)

Detailed design

API Changes

Conformance to NilLiteralConvertible is removed from all types except Optional, ImplicitlyUnwrappedOptional, and _OptionalNilComparisonType, along with the implementation of init(nilLiteral:).

init(bitPattern: Int) and init(bitPattern: UInt) on all pointer types become failable; if the bit pattern represents a null pointer, nil is returned.

Process.unsafeArgv is a pointer to a null-terminated C array of C strings, so its type changes from UnsafeMutablePointer<UnsafeMutablePointer<Int8>> to UnsafeMutablePointer<UnsafeMutablePointer<Int8>?>, i.e. the inner pointer type becomes optional. It is then an error to access Process.unsafeArgv before entering main. (Previously you would get a null pointer value.)

NSErrorPointer becomes optional:

-public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>
+public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>?
A number of methods on String that came from NSString now have optional parameters:
   public func completePathIntoString(
- outputName: UnsafeMutablePointer<String> = nil,
+ outputName: UnsafeMutablePointer<String>? = nil,
     caseSensitive: Bool,
- matchesIntoArray: UnsafeMutablePointer<[String]> = nil,
+ matchesIntoArray: UnsafeMutablePointer<[String]>? = nil,
     filterTypes: [String]? = nil
   ) -> Int {
   public init(
     contentsOfFile path: String,
- usedEncoding: UnsafeMutablePointer<NSStringEncoding> = nil
+ usedEncoding: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {

   public init(
     contentsOfURL url: NSURL,
- usedEncoding enc: UnsafeMutablePointer<NSStringEncoding> = nil
+ usedEncoding enc: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {
   public func linguisticTags(
     in range: Range<Index>,
     scheme tagScheme: String,
     options opts: NSLinguisticTaggerOptions = [],
     orthography: NSOrthography? = nil,
- tokenRanges: UnsafeMutablePointer<[Range<Index>]> = nil
+ tokenRanges: UnsafeMutablePointer<[Range<Index>]>? = nil
   ) -> [String] {
NSZone's no-argument initializer is gone. (It probably should have been removed already as part of the Swift 3 naming cleanup.)

A small regression: optional pointers can no longer be passed using withVaList because it would require a conditional conformance to the CVarArg protocol. For now, using unsafeBitCast to reinterpret the optional pointer as an Int is the best alternative; Int has the same C variadic calling conventions as a pointer on all supported platforms.

Conversion between pointers

Currently each pointer type has initializers of this form:

init<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>)
This simply makes a pointer with a different type but the same address as otherPointer. However, in making pointer nullability explicit, this now only converts non-nil pointers to non-nil pointers. In my experiments, this has led to this idiom becoming very common:

// Before:
let untypedPointer = UnsafePointer<Void>(ptr)

// After:
let untypedPointer = ptr.map(UnsafePointer<Void>.init)

// Usually the pointee type is actually inferred:
foo(ptr.map(UnsafePointer.init))
I consider this a bit more difficult to understand than the original code, at least at a glance. We should therefore add new initializers of the following form:

init?<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>?) {
  guard let nonnullPointer = otherPointer else {
    return nil
  }
  self.init(nonnullPointer)
}
The body is for explanation purposes only; we'll make sure the actual implementation does not require an extra comparison.

(This would need to be an overload rather than replacing the previous initializer because the "non-null-ness" should be preserved through the type conversion.)

The alternative is to leave this initializer out, and require the nil case to be explicitly handled or mapped away.

Open Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region with no ownership or lifetime semantics; it is logically a bare typed pointer (its baseAddress) and a length (count). For a buffer with 0 elements, however, there's no need to provide the address of allocated memory, since it can't be read from. Previously this case would be represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to access the base address: they need to consider the nil case explicitly, where previously they wouldn't have had to. There are several possibilities here, each with their own possible implementations:

Like UnsafePointer, UnsafeBufferPointer should always have a valid base address, even when the count is 0. An UnsafeBufferPointer with a potentially-nil base address should be optional.

UnsafeBufferPointer's initializer accepts an optional pointer and becomes failable, returning nil if the input pointer is nil.

UnsafeBufferPointer's initializer accepts an optional pointer and synthesizes a non-null aligned pointer value if given nil as a base address.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients such as withUnsafeBufferPointermust synthesize a non-null aligned pointer value if they do not have a valid pointer to provide.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients using withUnsafeBufferPointermust handle a nil buffer.

UnsafeBufferPointer should allow nil base addresses, i.e. the baseAddress property will be optional. Clients will need to handle this case explicitly.

UnsafeBufferPointer's initializer accepts an optional pointer, but no other changes are made.

UnsafeBufferPointer's initializer accepts an optional pointer. Additionally, any buffers initialized with a count of 0 will be canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer and length probably shouldn't require the pointer to be non-null, but if they do then perhaps there's a reason for it. It's also the least work.

Chris (Lattner) is leaning towards option (1ii), which treats UnsafeBufferPointer similar to UnsafePointer while not penalizing the common case of withUnsafeBufferPointer.
Impact on existing code

Any code that uses a pointer type (including Selector or NSZone) may be affected by this change. For the most part our existing logic to handle last year's nullability audit should cover this, but the implementer should test migration of several projects to see what issues might arise.

Anecdotally, in migrating the standard library to use this new logic I've been quite happy with nullability being made explicit. There are many places where a pointer really can't be nil.
Alternatives considered

The primary alternative here would be to leave everything as it is today, with UnsafePointer and friends including the null pointer as one of their normal values. This has obviously worked just fine for nearly two years of Swift, but it is leaving information on the table that can help avoid bugs, and is strange in a language that makes fluent use of Optional. As a fairly major source-breaking change, it is also something that we probably should do sooner rather than later in the language's evolution.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Félix Cloutier) #10

Well, they can also choose to not conform, or someone could use something lower-level than C to put something useful at address 0. I don't have any recent example, but for instance, in real mode, x86 has its interrupt descriptor table starting at 0.

It's true that it's a net win if you do app development, and it's probably a net win in most other cases too, but I just think that it's a little scary to make it 100% impossible to access anything at address 0. It's a tiny thing but it would make Swift the wrong tool if you ever needed to do it for whatever whacky reason that shouldn't exist in theory but ends up there in practice.

(My objection entirely goes away if `ptr!.pointee` tries to dereference the null address, but I think that this requires more compiler magic.)

Félix

···

Le 18 mars 2016 à 00:32:38, Chris Lattner <clattner@apple.com> a écrit :

On Mar 17, 2016, at 8:21 PM, Félix Cloutier via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Fortunately, when a type has a single invalid value for which no operations are valid, Swift already has a solution: Optionals.

To me, this makes it sound like dereferencing an unsafe pointer is unsafe only if the pointer is nil. (Nil does have the merit that it's generally one of the only addresses known to be invalid, though.)

One thing that I would worry about, though, is the case where dereferencing address 0 is legal. My understanding is that Swift is aimed to be a systems language, and this case is relatively frequent on embedded devices.

This is definitely an important case to consider, but I don't think this is a problem for the proposal.

C compilers for these sorts of embedded systems have two choices: they can either reserve 1 byte for the null value (wasting a byte of physmem) or they can use a null value with a representation that is not equal to zero.

In either case, the null value (whatever it is) maps to the swift none case. Everything works out properly.

-Chris

Félix

Le 17 mars 2016 à 21:59:49, Jordan Rose via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> a écrit :

Hey, everyone. If you're like me, you're sick of the fact that 'UnsafePointer<Int>' doesn't tell you whether or not the pointer can be nil. Why do we need to suffer this indignity when reference types—including function pointers!—can distinguish "present" from "absent" with the standard type 'Optional'? Well, good news: here's a proposal to make pointer nullability explicit. 'UnsafePointer<Int>?' can be null (nil), while 'UnsafePointer<Int>' cannot. Read on for details!

https://github.com/jrose-apple/swift-evolution/blob/optional-pointers/proposals/nnnn-optional-pointers.md

Bonus good news: I've implemented this locally and updated nearly all the tests already. Assuming this is accepting, the actual changes will go through review as a PR on GitHub, although it's mostly going to be one big mega-patch because the core change has a huge ripple effect.

Jordan


(Goffredo Marocchi) #11

+1 as well, seems like a proposal worth the source breaking changes it may bring.

[[iOS messageWithContent:webContent] broadcast]

···

On 18 Mar 2016, at 04:08, Russ Bishop via swift-evolution <swift-evolution@swift.org> wrote:

I’m very much +1 on this idea.

On Mar 17, 2016, at 6:59 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:

I consider this a bit more difficult to understand than the original code, at least at a glance. We should therefore add new initializers of the following form:

init?<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>?) {
  guard let nonnullPointer = otherPointer else {
    return nil
  }
  self.init(nonnullPointer)
}
The body is for explanation purposes only; we'll make sure the actual implementation does not require an extra comparison.

(This would need to be an overload rather than replacing the previous initializer because the "non-null-ness" should be preserved through the type conversion.)

The alternative is to leave this initializer out, and require the nil case to be explicitly handled or mapped away.

This should definitely be included. Everyone is used to “casting” by calling the appropriate initializers, and UnsafePointer<SpecificType>(someUnsafeVoidPointer) maps directly to what a C/Objective-C programmer would expect.

Open Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region with no ownership or lifetime semantics; it is logically a bare typed pointer (its baseAddress) and a length (count). For a buffer with 0 elements, however, there's no need to provide the address of allocated memory, since it can't be read from. Previously this case would be represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to access the base address: they need to consider the nil case explicitly, where previously they wouldn't have had to. There are several possibilities here, each with their own possible implementations:

Like UnsafePointer, UnsafeBufferPointer should always have a valid base address, even when the count is 0. An UnsafeBufferPointer with a potentially-nil base address should be optional.

UnsafeBufferPointer's initializer accepts an optional pointer and becomes failable, returning nil if the input pointer is nil.

UnsafeBufferPointer's initializer accepts an optional pointer and synthesizes a non-null aligned pointer value if given nil as a base address.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients such as withUnsafeBufferPointermust synthesize a non-null aligned pointer value if they do not have a valid pointer to provide.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients using withUnsafeBufferPointermust handle a nil buffer.

UnsafeBufferPointer should allow nil base addresses, i.e. the baseAddress property will be optional. Clients will need to handle this case explicitly.

UnsafeBufferPointer's initializer accepts an optional pointer, but no other changes are made.

UnsafeBufferPointer's initializer accepts an optional pointer. Additionally, any buffers initialized with a count of 0 will be canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer and length probably shouldn't require the pointer to be non-null, but if they do then perhaps there's a reason for it. It's also the least work.

Chris (Lattner) is leaning towards option (1ii), which treats UnsafeBufferPointer similar to UnsafePointer while not penalizing the common case of withUnsafeBufferPointer.

What’s the use of an UnsafeBufferPointer with zero count? Semantically that is making a claim that it can’t back up (“I have allocated memory at location X” which isn’t compatible with the idea of “zero count/size").

Without knowing more context I’d strongly favor (1i). If an array is empty the natural expectation for withUnsafeBufferPointer is you get UnsafeBufferPointer<Element>? = nil, which follows the behavior of the rest of the language and things like guard let make it trivial to handle properly. If someone really has a problem with it they can add ifUnsafeBufferPointer() that returns a non-optional pointer and skips executing the closure if the Array is empty (which is the behavior of your standard for loop).

Russ

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Jordan Rose) #12

Fortunately, when a type has a single invalid value for which no operations are valid, Swift already has a solution: Optionals.

To me, this makes it sound like dereferencing an unsafe pointer is unsafe only if the pointer is nil. (Nil does have the merit that it's generally one of the only addresses known to be invalid, though.)

Okay, you're right, it was a bad description of the problem. UnsafePointer remains "unsafe" for a lot of other reasons. :slight_smile:

One thing that I would worry about, though, is the case where dereferencing address 0 is legal. My understanding is that Swift is aimed to be a systems language, and this case is relatively frequent on embedded devices.

Good point. Technically in the C standard, there must be some pointer value that cannot be the address of anything valid, whether or not it has a bit pattern of 0; the intent was that Swift's 'nil' would follow whatever C did for that platform rather than being 0. But I imagine a lot of embedded C code doesn't actually follow this rule (i.e. "NULL" will still give you the 0 address).

I'm not very familiar with this space at all, but I'll look into it some more. If you know what existing compilers do here that'd be great to read.

Jordan

···

On Mar 17, 2016, at 20:21 , Félix Cloutier <felixcca@yahoo.ca> wrote:


(Jordan Rose) #13

The important use case here is that "array.withUnsafeBufferPointer" should always do something (i.e. it usually can't just skip the closure), and it turns out it's easiest if the zero-element case is treated the same as everything else. When converting over the standard library I found that very few of them wanted to do something different in the zero-element case, and then it would be bad to force Array to allocate memory just to not use it. That is, there aren't actually any clients interested in knowing whether the base address is valid, and all of the ones that do have to think about it (because they use it directly) aren't getting any value out of it.

Jordan

···

On Mar 17, 2016, at 21:08 , Russ Bishop <xenadu@gmail.com> wrote:

I’m very much +1 on this idea.

On Mar 17, 2016, at 6:59 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#open-issue-unsafebufferpointer>Open Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region with no ownership or lifetime semantics; it is logically a bare typed pointer (its baseAddress) and a length (count). For a buffer with 0 elements, however, there's no need to provide the address of allocated memory, since it can't be read from. Previously this case would be represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to access the base address: they need to consider the nil case explicitly, where previously they wouldn't have had to. There are several possibilities here, each with their own possible implementations:

Like UnsafePointer, UnsafeBufferPointer should always have a valid base address, even when the count is 0. An UnsafeBufferPointer with a potentially-nil base address should be optional.

UnsafeBufferPointer's initializer accepts an optional pointer and becomes failable, returning nil if the input pointer is nil.

UnsafeBufferPointer's initializer accepts an optional pointer and synthesizes a non-null aligned pointer value if given nil as a base address.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients such as withUnsafeBufferPointermust synthesize a non-null aligned pointer value if they do not have a valid pointer to provide.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients using withUnsafeBufferPointermust handle a nil buffer.

UnsafeBufferPointer should allow nil base addresses, i.e. the baseAddress property will be optional. Clients will need to handle this case explicitly.

UnsafeBufferPointer's initializer accepts an optional pointer, but no other changes are made.

UnsafeBufferPointer's initializer accepts an optional pointer. Additionally, any buffers initialized with a count of 0 will be canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer and length probably shouldn't require the pointer to be non-null, but if they do then perhaps there's a reason for it. It's also the least work.

Chris (Lattner) is leaning towards option (1ii), which treats UnsafeBufferPointer similar to UnsafePointer while not penalizing the common case of withUnsafeBufferPointer.

What’s the use of an UnsafeBufferPointer with zero count? Semantically that is making a claim that it can’t back up (“I have allocated memory at location X” which isn’t compatible with the idea of “zero count/size").

Without knowing more context I’d strongly favor (1i). If an array is empty the natural expectation for withUnsafeBufferPointer is you get UnsafeBufferPointer<Element>? = nil, which follows the behavior of the rest of the language and things like guard let make it trivial to handle properly. If someone really has a problem with it they can add ifUnsafeBufferPointer() that returns a non-optional pointer and skips executing the closure if the Array is empty (which is the behavior of your standard for loop).


(Jordan Rose) #14

I'm not sure what you mean by this. (I think we agree, but I'm not sure.) A null pointer (in C) is not a pointer with a zero representation; it's a pointer that is guaranteed not to point to any valid object. By using Optional, Swift gets that value out of the type called "pointer", but can still represent it efficiently by claiming a platform-dependent bit pattern that won't be mistaken for a valid pointer. (That is, we don't need a separate flag to represent the type 'UnsafePointer<Int>?', which is hardly an uncommon type.)

In practice, people often pass all sorts of non-pointer values through parameters of type 'void *', so Swift can't be too aggressive about assuming particular bit patterns are or are not valid. Swift does have to understand C's representation of an invalid pointer because you want to be able to assign to imported struct fields with pointer type.

And correct, using a very large subscript on a pointer to navigate to the zero value will not result in a precondition failure.

Jordan

···

On Mar 21, 2016, at 13:40 , David Waite <david@alkaline-solutions.com> wrote:

I’m +1 on this idea, but I think we should be more careful in the introduction. I don’t think at the swift level the concept of a null pointer should exist at all.

It is the platform and/or languages Swift integrates with which define a zero pointer as indicating ‘not pointing to anything’. It is the bridging between the platform or languages which should understand that a parameter or resulting value may optionally have on pointer, and how that pointer is represented by the underlying swift runtime.

I imagine that using RandomAccessIndexType methods to navigate to the zero value will not result in a precondition failure?

-DW

On Mar 17, 2016, at 7:59 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hey, everyone. If you're like me, you're sick of the fact that 'UnsafePointer<Int>' doesn't tell you whether or not the pointer can be nil. Why do we need to suffer this indignity when reference types—including function pointers!—can distinguish "present" from "absent" with the standard type 'Optional'? Well, good news: here's a proposal to make pointer nullability explicit. 'UnsafePointer<Int>?' can be null (nil), while 'UnsafePointer<Int>' cannot. Read on for details!

https://github.com/jrose-apple/swift-evolution/blob/optional-pointers/proposals/nnnn-optional-pointers.md

Bonus good news: I've implemented this locally and updated nearly all the tests already. Assuming this is accepting, the actual changes will go through review as a PR on GitHub, although it's mostly going to be one big mega-patch because the core change has a huge ripple effect.

Jordan

---

Make pointer nullability explicit using Optional

Proposal: SE-NNNN <https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-name.md>
Author(s): Jordan Rose <https://github.com/jrose-apple>
Status: Awaiting review
Review manager: TBD
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#introduction>Introduction

In Objective-C, pointers (whether to objects or to a non-object type) can be marked as nullable or nonnull, depending on whether the pointer value can ever be null. In Swift, however, there is no such way to make this distinction for pointers to non-object types: an UnsafePointer<Int> might be null, or it might never be.

We already have a way to describe this: Optionals. This proposal makes UnsafePointer<Int> represent a non-nullable pointer, and UnsafePointer<Int>? a nullable pointer. This also allows us to preserve information about pointer nullability available in header files for imported C and Objective-C APIs.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#motivation>Motivation

Today, UnsafePointer and friends suffer from a problem inherited from C: every pointer value could potentially be null, and code that works with pointers may or may not expect this. Failing to take the null pointer case into account can lead to assertion failures or crashes. For example, pretty much every operation on UnsafePointer itself requires a valid pointer (reading, writing, and initializing the pointee or performing arithmetic operations).

Fortunately, when a type has a single invalid value for which no operations are valid, Swift already has a solution: Optionals. Applying this to pointer types makes things very clear: if the type is non-optional, the pointer will never be null, and if it isoptional, the developer must take the "null pointer" case into account. This clarity has already been appreciated in Apple's Objective-C headers, which include nullability annotations for all pointer types (not just object pointers).

This change also allows developers working with pointers to take advantage of the many syntactic conveniences already built around optionals. For example, the standard library currently has a helper method on UnsafeMutablePointer called _setIfNonNil; with "optional pointers" this can be written simply and clearly:

ptr?.pointee = newValue
Finally, this change also reduces the number of types that conform to NilLiteralConvertible, a source of confusion for newcomers who (reasonably) associate nil directly with optionals. Currently the standard library includes the following NilLiteralConvertible types:

Optional
ImplicitlyUnwrappedOptional (subject of a separate proposal by Chris Willmore)
_OptionalNilComparisonType (used for optionalValue == nil)
UnsafePointer
UnsafeMutablePointer
AutoreleasingUnsafeMutablePointer
OpaquePointer
plus these Objective-C-specific types:

Selector
NSZone (only used to pass nil in Swift)
All of the italicized types would drop their conformance to NilLiteralConvertible; the "null pointer" would be represented by a nil optional of a particular type.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#proposed-solution>Proposed solution

Have the compiler assume that all values with pointer type (the italicized types listed above) are non-null. This allows the representation of Optional.none for a pointer type to be a null pointer value.

Drop NilLiteralConvertible conformance for all pointer types.

Teach the Clang importer to treat _Nullable pointers as Optional (and _Null_unspecified pointers as ImplicitlyUnwrappedOptional).

Deal with the fallout, i.e. adjust the compiler and the standard library to handle this new behavior.

Test migration and improve the migrator as necessary.

This proposal does not include the removal of the NilLiteralConvertible protocol altogether; besides still having two distinct optional types, we've seen people wanting to use nil for their own types (e.g. JSON values). (Changing this in the future is not out of the question; it's just out of scope for this proposal.)

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#detailed-design>Detailed design

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#api-changes>API Changes

Conformance to NilLiteralConvertible is removed from all types except Optional, ImplicitlyUnwrappedOptional, and _OptionalNilComparisonType, along with the implementation of init(nilLiteral:).

init(bitPattern: Int) and init(bitPattern: UInt) on all pointer types become failable; if the bit pattern represents a null pointer, nil is returned.

Process.unsafeArgv is a pointer to a null-terminated C array of C strings, so its type changes from UnsafeMutablePointer<UnsafeMutablePointer<Int8>> to UnsafeMutablePointer<UnsafeMutablePointer<Int8>?>, i.e. the inner pointer type becomes optional. It is then an error to access Process.unsafeArgv before entering main. (Previously you would get a null pointer value.)

NSErrorPointer becomes optional:

-public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>
+public typealias NSErrorPointer = AutoreleasingUnsafeMutablePointer<NSError?>?
A number of methods on String that came from NSString now have optional parameters:
   public func completePathIntoString(
- outputName: UnsafeMutablePointer<String> = nil,
+ outputName: UnsafeMutablePointer<String>? = nil,
     caseSensitive: Bool,
- matchesIntoArray: UnsafeMutablePointer<[String]> = nil,
+ matchesIntoArray: UnsafeMutablePointer<[String]>? = nil,
     filterTypes: [String]? = nil
   ) -> Int {
   public init(
     contentsOfFile path: String,
- usedEncoding: UnsafeMutablePointer<NSStringEncoding> = nil
+ usedEncoding: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {

   public init(
     contentsOfURL url: NSURL,
- usedEncoding enc: UnsafeMutablePointer<NSStringEncoding> = nil
+ usedEncoding enc: UnsafeMutablePointer<NSStringEncoding>? = nil
   ) throws {
   public func linguisticTags(
     in range: Range<Index>,
     scheme tagScheme: String,
     options opts: NSLinguisticTaggerOptions = [],
     orthography: NSOrthography? = nil,
- tokenRanges: UnsafeMutablePointer<[Range<Index>]> = nil
+ tokenRanges: UnsafeMutablePointer<[Range<Index>]>? = nil
   ) -> [String] {
NSZone's no-argument initializer is gone. (It probably should have been removed already as part of the Swift 3 naming cleanup.)

A small regression: optional pointers can no longer be passed using withVaList because it would require a conditional conformance to the CVarArg protocol. For now, using unsafeBitCast to reinterpret the optional pointer as an Int is the best alternative; Int has the same C variadic calling conventions as a pointer on all supported platforms.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#conversion-between-pointers>Conversion between pointers

Currently each pointer type has initializers of this form:

init<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>)
This simply makes a pointer with a different type but the same address as otherPointer. However, in making pointer nullability explicit, this now only converts non-nil pointers to non-nil pointers. In my experiments, this has led to this idiom becoming very common:

// Before:
let untypedPointer = UnsafePointer<Void>(ptr)

// After:
let untypedPointer = ptr.map(UnsafePointer<Void>.init)

// Usually the pointee type is actually inferred:
foo(ptr.map(UnsafePointer.init))
I consider this a bit more difficult to understand than the original code, at least at a glance. We should therefore add new initializers of the following form:

init?<OtherPointee>(_ otherPointer: UnsafePointer<OtherPointee>?) {
  guard let nonnullPointer = otherPointer else {
    return nil
  }
  self.init(nonnullPointer)
}
The body is for explanation purposes only; we'll make sure the actual implementation does not require an extra comparison.

(This would need to be an overload rather than replacing the previous initializer because the "non-null-ness" should be preserved through the type conversion.)

The alternative is to leave this initializer out, and require the nil case to be explicitly handled or mapped away.

<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#open-issue-unsafebufferpointer>Open Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region with no ownership or lifetime semantics; it is logically a bare typed pointer (its baseAddress) and a length (count). For a buffer with 0 elements, however, there's no need to provide the address of allocated memory, since it can't be read from. Previously this case would be represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to access the base address: they need to consider the nil case explicitly, where previously they wouldn't have had to. There are several possibilities here, each with their own possible implementations:

Like UnsafePointer, UnsafeBufferPointer should always have a valid base address, even when the count is 0. An UnsafeBufferPointer with a potentially-nil base address should be optional.

UnsafeBufferPointer's initializer accepts an optional pointer and becomes failable, returning nil if the input pointer is nil.

UnsafeBufferPointer's initializer accepts an optional pointer and synthesizes a non-null aligned pointer value if given nil as a base address.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients such as withUnsafeBufferPointermust synthesize a non-null aligned pointer value if they do not have a valid pointer to provide.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients using withUnsafeBufferPointermust handle a nil buffer.

UnsafeBufferPointer should allow nil base addresses, i.e. the baseAddress property will be optional. Clients will need to handle this case explicitly.

UnsafeBufferPointer's initializer accepts an optional pointer, but no other changes are made.

UnsafeBufferPointer's initializer accepts an optional pointer. Additionally, any buffers initialized with a count of 0 will be canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer and length probably shouldn't require the pointer to be non-null, but if they do then perhaps there's a reason for it. It's also the least work.

Chris (Lattner) is leaning towards option (1ii), which treats UnsafeBufferPointer similar to UnsafePointer while not penalizing the common case of withUnsafeBufferPointer.
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#impact-on-existing-code>Impact on existing code

Any code that uses a pointer type (including Selector or NSZone) may be affected by this change. For the most part our existing logic to handle last year's nullability audit should cover this, but the implementer should test migration of several projects to see what issues might arise.

Anecdotally, in migrating the standard library to use this new logic I've been quite happy with nullability being made explicit. There are many places where a pointer really can't be nil.
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#alternatives-considered>Alternatives considered

The primary alternative here would be to leave everything as it is today, with UnsafePointer and friends including the null pointer as one of their normal values. This has obviously worked just fine for nearly two years of Swift, but it is leaving information on the table that can help avoid bugs, and is strange in a language that makes fluent use of Optional. As a fairly major source-breaking change, it is also something that we probably should do sooner rather than later in the language's evolution.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution


(Dmitri Gribenko) #15

Thanks, Jordan! I have a concern about the following point:

+- `Process.unsafeArgv` is a pointer to a null-terminated C array of C strings,
+ so its type changes from `UnsafeMutablePointer<UnsafeMutablePointer<Int8>>` to
+ `UnsafeMutablePointer<UnsafeMutablePointer<Int8>?>`, i.e. the inner pointer
+ type becomes optional. It is then an error to access `Process.unsafeArgv`
+ before entering `main`. (Previously you would get a null pointer value.)

I don't think that this error is defensible -- if you are writing some
library code, you just can't know whether you are being executed
before `main` or not. This is a precondition that reusable code just
can't guarantee.

Dmitri

···

On Mon, Mar 21, 2016 at 4:14 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:

I've recorded everyone's feedback so far (mostly on the UnsafeBufferPointer
issue) and opened a pull request:
https://github.com/apple/swift-evolution/pull/219.

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/


(Russ Bishop) #16

Does optional chaining (ptr?.count ?? 0) or the guard check (guard let ptr = ptr else { return }) impose a performance (or cognitive) burden here? I’m OK with (2i), it just seems less Swift-ish than (1i).

I don’t use UnsafeBufferPointer a lot so I’ll happily live with whatever the choice is.

Russ

···

On Mar 18, 2016, at 9:49 AM, Jordan Rose <jordan_rose@apple.com> wrote:

On Mar 17, 2016, at 21:08 , Russ Bishop <xenadu@gmail.com <mailto:xenadu@gmail.com>> wrote:

I’m very much +1 on this idea.

On Mar 17, 2016, at 6:59 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
<https://github.com/jrose-apple/swift-evolution/tree/optional-pointers#open-issue-unsafebufferpointer>Open Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region with no ownership or lifetime semantics; it is logically a bare typed pointer (its baseAddress) and a length (count). For a buffer with 0 elements, however, there's no need to provide the address of allocated memory, since it can't be read from. Previously this case would be represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to access the base address: they need to consider the nil case explicitly, where previously they wouldn't have had to. There are several possibilities here, each with their own possible implementations:

Like UnsafePointer, UnsafeBufferPointer should always have a valid base address, even when the count is 0. An UnsafeBufferPointer with a potentially-nil base address should be optional.

UnsafeBufferPointer's initializer accepts an optional pointer and becomes failable, returning nil if the input pointer is nil.

UnsafeBufferPointer's initializer accepts an optional pointer and synthesizes a non-null aligned pointer value if given nil as a base address.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients such as withUnsafeBufferPointermust synthesize a non-null aligned pointer value if they do not have a valid pointer to provide.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients using withUnsafeBufferPointermust handle a nil buffer.

UnsafeBufferPointer should allow nil base addresses, i.e. the baseAddress property will be optional. Clients will need to handle this case explicitly.

UnsafeBufferPointer's initializer accepts an optional pointer, but no other changes are made.

UnsafeBufferPointer's initializer accepts an optional pointer. Additionally, any buffers initialized with a count of 0 will be canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer and length probably shouldn't require the pointer to be non-null, but if they do then perhaps there's a reason for it. It's also the least work.

Chris (Lattner) is leaning towards option (1ii), which treats UnsafeBufferPointer similar to UnsafePointer while not penalizing the common case of withUnsafeBufferPointer.

What’s the use of an UnsafeBufferPointer with zero count? Semantically that is making a claim that it can’t back up (“I have allocated memory at location X” which isn’t compatible with the idea of “zero count/size").

Without knowing more context I’d strongly favor (1i). If an array is empty the natural expectation for withUnsafeBufferPointer is you get UnsafeBufferPointer<Element>? = nil, which follows the behavior of the rest of the language and things like guard let make it trivial to handle properly. If someone really has a problem with it they can add ifUnsafeBufferPointer() that returns a non-optional pointer and skips executing the closure if the Array is empty (which is the behavior of your standard for loop).

The important use case here is that "array.withUnsafeBufferPointer" should always do something (i.e. it usually can't just skip the closure), and it turns out it's easiest if the zero-element case is treated the same as everything else. When converting over the standard library I found that very few of them wanted to do something different in the zero-element case, and then it would be bad to force Array to allocate memory just to not use it. That is, there aren't actually any clients interested in knowing whether the base address is valid, and all of the ones that do have to think about it (because they use it directly) aren't getting any value out of it.

Jordan


(Félix Cloutier) #17

GCC may have some hints:

-fdelete-null-pointer-checks
Assume that programs cannot safely dereference null pointers, and that no code or data element resides at address zero. This option enables simple constant folding optimizations at all optimization levels. In addition, other optimization passes in GCC use this flag to control global dataflow analyses that eliminate useless checks for null pointers; these assume that a memory access to address zero always results in a trap, so that if a pointer is checked after it has already been dereferenced, it cannot be null.
Note however that in some environments this assumption is not true. Use -fno-delete-null-pointer-checks to disable this optimization for programs that depend on that behavior.

This option is enabled by default on most targets. On Nios II ELF, it defaults to off. On AVR and CR16, this option is completely disabled.

Passes that use the dataflow information are enabled independently at different optimization levels.

I recall that my first time being bit by null being valid was on AVR32 (though with GCC 3.something), and this seems to confirm it.

Félix

···

Le 18 mars 2016 à 12:40:35, Jordan Rose <jordan_rose@apple.com> a écrit :

On Mar 17, 2016, at 20:21 , Félix Cloutier <felixcca@yahoo.ca <mailto:felixcca@yahoo.ca>> wrote:

Fortunately, when a type has a single invalid value for which no operations are valid, Swift already has a solution: Optionals.

To me, this makes it sound like dereferencing an unsafe pointer is unsafe only if the pointer is nil. (Nil does have the merit that it's generally one of the only addresses known to be invalid, though.)

Okay, you're right, it was a bad description of the problem. UnsafePointer remains "unsafe" for a lot of other reasons. :slight_smile:

One thing that I would worry about, though, is the case where dereferencing address 0 is legal. My understanding is that Swift is aimed to be a systems language, and this case is relatively frequent on embedded devices.

Good point. Technically in the C standard, there must be some pointer value that cannot be the address of anything valid, whether or not it has a bit pattern of 0; the intent was that Swift's 'nil' would follow whatever C did for that platform rather than being 0. But I imagine a lot of embedded C code doesn't actually follow this rule (i.e. "NULL" will still give you the 0 address).

I'm not very familiar with this space at all, but I'll look into it some more. If you know what existing compilers do here that'd be great to read.

Jordan


(Chris Lattner) #18

Why would you have to allocate memory for this case? The pointer only needs to be non-null, not valid and dereferencable. You could use the address of a global or even 0x1.

-Chris

···

On Mar 18, 2016, at 9:49 AM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:

On Mar 17, 2016, at 21:08 , Russ Bishop <xenadu@gmail.com> wrote:

I’m very much +1 on this idea.

On Mar 17, 2016, at 6:59 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:
Open Issue: UnsafeBufferPointer

The type UnsafeBufferPointer represents a bounded typed memory region with no ownership or lifetime semantics; it is logically a bare typed pointer (its baseAddress) and a length (count). For a buffer with 0 elements, however, there's no need to provide the address of allocated memory, since it can't be read from. Previously this case would be represented as a nil base address and a count of 0.

With optional pointers, this now imposes a cost on clients that want to access the base address: they need to consider the nil case explicitly, where previously they wouldn't have had to. There are several possibilities here, each with their own possible implementations:

Like UnsafePointer, UnsafeBufferPointer should always have a valid base address, even when the count is 0. An UnsafeBufferPointer with a potentially-nil base address should be optional.

UnsafeBufferPointer's initializer accepts an optional pointer and becomes failable, returning nil if the input pointer is nil.

UnsafeBufferPointer's initializer accepts an optional pointer and synthesizes a non-null aligned pointer value if given nil as a base address.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients such as withUnsafeBufferPointermust synthesize a non-null aligned pointer value if they do not have a valid pointer to provide.

UnsafeBufferPointer's initializer only accepts non-optional pointers. Clients using withUnsafeBufferPointermust handle a nil buffer.

UnsafeBufferPointer should allow nil base addresses, i.e. the baseAddress property will be optional. Clients will need to handle this case explicitly.

UnsafeBufferPointer's initializer accepts an optional pointer, but no other changes are made.

UnsafeBufferPointer's initializer accepts an optional pointer. Additionally, any buffers initialized with a count of 0 will be canonicalized to having a base address of nil.

I'm currently leaning towards option (2i). Clients that expect a pointer and length probably shouldn't require the pointer to be non-null, but if they do then perhaps there's a reason for it. It's also the least work.

Chris (Lattner) is leaning towards option (1ii), which treats UnsafeBufferPointer similar to UnsafePointer while not penalizing the common case of withUnsafeBufferPointer.

What’s the use of an UnsafeBufferPointer with zero count? Semantically that is making a claim that it can’t back up (“I have allocated memory at location X” which isn’t compatible with the idea of “zero count/size").

Without knowing more context I’d strongly favor (1i). If an array is empty the natural expectation for withUnsafeBufferPointer is you get UnsafeBufferPointer<Element>? = nil, which follows the behavior of the rest of the language and things like guard let make it trivial to handle properly. If someone really has a problem with it they can add ifUnsafeBufferPointer() that returns a non-optional pointer and skips executing the closure if the Array is empty (which is the behavior of your standard for loop).

The important use case here is that "array.withUnsafeBufferPointer" should always do something (i.e. it usually can't just skip the closure), and it turns out it's easiest if the zero-element case is treated the same as everything else. When converting over the standard library I found that very few of them wanted to do something different in the zero-element case, and then it would be bad to force Array to allocate memory just to not use it. That is, there aren't actually any clients interested in knowing whether the base address is valid, and all of the ones that do have to think about it (because they use it directly) aren't getting any value out of it.


(Chris Lattner) #19

Felix, I understand what you're saying, but I still don't see how it shapes this discussion. There are many things you can't do at that level of system programming in pure swift at the moment, and there are better ways to solve those problems later than to make UnsafePointer bear the burden for an extremely narrow use case.

Swift had no notion of volatile pointers for example.

-Chris

···

On Mar 17, 2016, at 10:09 PM, Félix Cloutier <felixcca@yahoo.ca> wrote:

Well, they can also choose to not conform, or someone could use something lower-level than C to put something useful at address 0. I don't have any recent example, but for instance, in real mode, x86 has its interrupt descriptor table starting at 0.

It's true that it's a net win if you do app development, and it's probably a net win in most other cases too, but I just think that it's a little scary to make it 100% impossible to access anything at address 0. It's a tiny thing but it would make Swift the wrong tool if you ever needed to do it for whatever whacky reason that shouldn't exist in theory but ends up there in practice.

(My objection entirely goes away if `ptr!.pointee` tries to dereference the null address, but I think that this requires more compiler magic.)

Félix

Le 18 mars 2016 à 00:32:38, Chris Lattner <clattner@apple.com> a écrit :

On Mar 17, 2016, at 8:21 PM, Félix Cloutier via swift-evolution <swift-evolution@swift.org> wrote:

Fortunately, when a type has a single invalid value for which no operations are valid, Swift already has a solution: Optionals.

To me, this makes it sound like dereferencing an unsafe pointer is unsafe only if the pointer is nil. (Nil does have the merit that it's generally one of the only addresses known to be invalid, though.)

One thing that I would worry about, though, is the case where dereferencing address 0 is legal. My understanding is that Swift is aimed to be a systems language, and this case is relatively frequent on embedded devices.

This is definitely an important case to consider, but I don't think this is a problem for the proposal.

C compilers for these sorts of embedded systems have two choices: they can either reserve 1 byte for the null value (wasting a byte of physmem) or they can use a null value with a representation that is not equal to zero.

In either case, the null value (whatever it is) maps to the swift none case. Everything works out properly.

-Chris

Félix

Le 17 mars 2016 à 21:59:49, Jordan Rose via swift-evolution <swift-evolution@swift.org> a écrit :

Hey, everyone. If you're like me, you're sick of the fact that 'UnsafePointer<Int>' doesn't tell you whether or not the pointer can be nil. Why do we need to suffer this indignity when reference types—including function pointers!—can distinguish "present" from "absent" with the standard type 'Optional'? Well, good news: here's a proposal to make pointer nullability explicit. 'UnsafePointer<Int>?' can be null (nil), while 'UnsafePointer<Int>' cannot. Read on for details!

https://github.com/jrose-apple/swift-evolution/blob/optional-pointers/proposals/nnnn-optional-pointers.md

Bonus good news: I've implemented this locally and updated nearly all the tests already. Assuming this is accepting, the actual changes will go through review as a PR on GitHub, although it's mostly going to be one big mega-patch because the core change has a huge ripple effect.

Jordan


(David Waite) #20

From "[swift-evolution] Notes from Swift core team 2016-03-23 design discussion”:

Make pointer nullability explicit using Optional <file:///Users/alexmartini/DevPubs%20Git%20Repositories/Swift%20Language%20Review/_build/html/LR_MeetingNotes/2016-03-23.html#make-pointer-nullability-explicit-using-optional>
https://github.com/apple/swift-evolution/pull/219
Biggest open issue is what to do with UnsafeBufferPointer which has a base address and a count of the number of elements at that address. The most common use is to do fast things with an array. The problem is when you have an empty array.

We have a statically initialized empty array, so this doesn’t apply to array. But slices and Cocoa arrays can do it.

Half of the use cases are subscripting off of the buffer, so they don’t actually use the base address. They can’t actually subscript an empty array, but it’s not a syntax error — the loop is run zero times, so it doesn’t matter. The other half pass the pointers down to a C API that takes an address and count.

Someone might expect that the base address doesn’t change when something is initialized.

We can’t easily use the zero pointer because SIL already uses it for nil. But there are issues with using the same representation as C to avoid bridging costs.

We’re mapping two things in C onto one thing in Swift. In C, the buffer pointer would be __nullable long * and the length is ulong.

Given everything else in the system, it’s more like pointer. We didn’t call it a buffer because that tends to imply ownership.

Sketching out the state space:

Pointer Length Static type
null 0 UBP?
valid >= 0 UBP
valid < 0 X
vull != 0 ???
This issue would go away if we got rid of the base address on UnsafeBufferPointer, but that would get rid of a number of valid C operations like calling memcopy.

It seems like withUnsafeBufferPointer should never produce nil. With that in mind, why should UnsafeBufferPointer need to?

We do need a properly-aligned “valid” invalid pointer. LLVM makes assumptions about things being aligned.

Dominant feedback on the list has been for people want something that round trips cleanly. Making the base address non-optional adds overhead and removes the ability to round trip.

It’s unfortunate that we don’t have a way to represent in the type system a buffer pointer that isn’t nullable, from within withUnsafeBufferPointer which wouldn’t even call its closure if the buffer has a null base address.

In my mind UBP is primarily meant to be a collection. In that case, I imagine (nil, 0) as an input wouldn’t necessarily represent a nil UBP? - it could represent an empty UBP.

My question is whether a valid pointer, length 0 is a valid UBP or not - I have trouble imagining a API which wants a UBP which would differentiate this value over the (nil, 0) one and not have it either be an abuse of UBP (using it to transport just a pointer and not representing a buffer) or an error. I suspect it actually would be ok to always represent a length 0 UBP as having a nil base address.

Alternatively expressing the way I see it in mock data structures:

enum InnerUnsafeBufferPointerRepresentation<T> {
   case empty
   case some(UnsafePointer<T>, Int) // where Int is always > 0
}

-DW