Proposal: Add Initializers For Converting UnsafePointers to Int and Unit

UnsafePointer and UnsafeMutablePointer both contain initializers to convert
from Int and Uint, but Int and UInt lack initializers to convert from
UnsafePointer and UnsafeMutablePointer. Over in swift-dev, Dimitri Gribenko
proposed adding something like the following initializers to stdlib.

extension UInt {
  init<T>(bitPattern: UnsafePointer<T>) {
    self = UInt(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }

  init<T>(bitPattern: UnsafeMutablePointer<T>) {
    self = UInt(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }
}

extension Int {
  init<T>(bitPattern: UnsafePointer<T>) {
    self = Int(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }

  init<T>(bitPattern: UnsafeMutablePointer<T>) {
    self = Int(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }
}

https://bugs.swift.org/browse/SR-131

There are two motivations behind this proposal. First, it would allow more
complicated pointer arithmetic, such as checking pointer alignment. Second,
some C functions take intptr_t and uintptr_t parameters, which are not
compatible with UnsafePointer.

As noted in UnsafePointer.swift.gyb the conversions from UInt and Int are
fundamentally unsafe, and these proposed conversions would also be unsafe.
They would also allow for more unsafe pointer arithmetic operations. For
example, it would allow users to convert two UnsafePointers to UInt8s in
order to find the distance between them, which would be a problem if the
user subtracted the larger pointer from the smaller pointer.

Because of this, and because it's part of the Swift evolution process, it's
worth considering alternatives. Pointer arithmetic could be taken care of
by adding functions to UnsafePointer, and these may be worth proposing
separately. I'm thinking something like this.

func distanceToBoundary(_ boundary: Int) -> Int {
    return Int(Builtin.ptrtoint_Word(self._rawValue)) % boundary
}

func isAlignedToBoundary(_ boundary: Int) -> Bool {
    return distanceToBoundary(boundary) == 0
}

As for intptr_t parameters, the only alternative I can think of is to
require users to write adaptor functions in C. Requiring users to write C
code also works for pointer alignment and other pointer arithmetic, but it
doesn't feel like a great solution to the problem.

Any feedback, additional motivations, or additional alternatives welcome.
Thanks.

Hi Michael,

Thank you for working on this proposal!

UnsafePointer and UnsafeMutablePointer both contain initializers to
convert from Int and Uint, but Int and UInt lack initializers to convert
from UnsafePointer and UnsafeMutablePointer. Over in swift-dev, Dimitri
Gribenko proposed adding something like the following initializers to
stdlib.

extension UInt {
  init<T>(bitPattern: UnsafePointer<T>) {
    self = UInt(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }

  init<T>(bitPattern: UnsafeMutablePointer<T>) {
    self = UInt(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }
}

extension Int {
  init<T>(bitPattern: UnsafePointer<T>) {
    self = Int(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }

  init<T>(bitPattern: UnsafeMutablePointer<T>) {
    self = Int(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }
}

[SR-131] Initializers for converting UnsafePointers to integers · Issue #42753 · apple/swift · GitHub

There are two motivations behind this proposal. First, it would allow more
complicated pointer arithmetic, such as checking pointer alignment. Second,
some C functions take intptr_t and uintptr_t parameters, which are not
compatible with UnsafePointer.

As noted in UnsafePointer.swift.gyb the conversions from UInt and Int are
fundamentally unsafe, and these proposed conversions would also be unsafe.
They would also allow for more unsafe pointer arithmetic operations. For
example, it would allow users to convert two UnsafePointers to UInt8s in
order to find the distance between them, which would be a problem if the
user subtracted the larger pointer from the smaller pointer.

Another use case is tagged pointers, which there are at least two flavors
of: Lisp-style (low bits are the tag), or NSString-style (the pointer
payload contains character data).

We could consider adding APIs for tagged pointers, too (although I'm
concerned that an API that exposes the operations with zero overhead would
be probably as unsafe as direct arithmetic and possibly more clumsy and
less readable).

A meta point is, though, that Swift, being a systems programming language,
should allow expressing these patterns (working with alignment, creating
tagged pointers), and any other things people might imagine (e.g., a XOR
linked list) directly in the language, without builtins.

Because of this, and because it's part of the Swift evolution process, it's

worth considering alternatives. Pointer arithmetic could be taken care of
by adding functions to UnsafePointer, and these may be worth proposing
separately. I'm thinking something like this.

func distanceToBoundary(_ boundary: Int) -> Int {
    return Int(Builtin.ptrtoint_Word(self._rawValue)) % boundary
}

func isAlignedToBoundary(_ boundary: Int) -> Bool {
    return distanceToBoundary(boundary) == 0
}

Then one would also need two other methods, I think: to align the pointer
to the nearest boundary, up and down.

As for intptr_t parameters, the only alternative I can think of is to

require users to write adaptor functions in C. Requiring users to write C
code also works for pointer alignment and other pointer arithmetic, but it
doesn't feel like a great solution to the problem.

Even writing wrappers for intptr_t doesn't feel great. One of the goals is
to be able to wrap C libraries and improve their API by augmenting with
Swift code to the extent possible (for multiple reasons).

Dmitri

···

On Tue, Dec 8, 2015 at 7:54 PM, Michael Buckley via swift-evolution < swift-evolution@swift.org> wrote:

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/

extension UInt {
  init<T>(bitPattern: UnsafePointer<T>) {
    self = UInt(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }

  init<T>(bitPattern: UnsafeMutablePointer<T>) {
    self = UInt(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }
}

I was helping someone in my NSCoder Night group struggle with UnsafePointer just last night. He was working with MIDIMetaEvent, a C struct with a variable-sized buffer at the end of it. There was just no good way to handle the allocation and use of this type; I ended up writing an extension full of bizarre magic just so he could access and use it..

Anyway, all this is to say I would have used these initializers if they’d existed, and the alignment stuff you’re suggesting probably wouldn’t have cut the mustard for some of what I was trying to do.

···

--
Brent Royal-Gordon
Architechies

On the off chance you're trying to do loads from unaligned UnsafePointers, that's undefined in the current interface. You'll need to memcpy to well-aligned memory first.

-Joe

···

On Dec 9, 2015, at 12:56 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

extension UInt {
init<T>(bitPattern: UnsafePointer<T>) {
   self = UInt(Builtin.ptrtoint_Word(bitPattern._rawValue))
}

init<T>(bitPattern: UnsafeMutablePointer<T>) {
   self = UInt(Builtin.ptrtoint_Word(bitPattern._rawValue))
}
}

I was helping someone in my NSCoder Night group struggle with UnsafePointer just last night. He was working with MIDIMetaEvent, a C struct with a variable-sized buffer at the end of it. There was just no good way to handle the allocation and use of this type; I ended up writing an extension full of bizarre magic just so he could access and use it..

Anyway, all this is to say I would have used these initializers if they’d existed, and the alignment stuff you’re suggesting probably wouldn’t have cut the mustard for some of what I was trying to do.

On the off chance you're trying to do loads from unaligned UnsafePointers, that's undefined in the current interface. You'll need to memcpy to well-aligned memory first.

The buffer I’m accessing is UInt8, which I believe is always aligned, right?

C:

  typedef struct MIDIMetaEvent
  {
    UInt8 metaEventType;
    UInt8 unused1;
    UInt8 unused2;
    UInt8 unused3;
    UInt32 dataLength;
    UInt8 data[1];
  } MIDIMetaEvent;

Swift:

  public struct MIDIMetaEvent {
      
      public var metaEventType: UInt8
      public var unused1: UInt8
      public var unused2: UInt8
      public var unused3: UInt8
      public var dataLength: UInt32
      public var data: (UInt8)
      public init()
      public init(metaEventType: UInt8, unused1: UInt8, unused2: UInt8, unused3: UInt8, dataLength: UInt32, data: (UInt8))
  }

For obvious reasons, Swift doesn’t understand the actual size of this data structure and only copies the amount of data it knows about. (It also has to be allocated with malloc(), because UnsafePointer.alloc() takes a number of objects, not a number of bytes. He was apparently in contact with someone at Apple about this MIDI stuff, and the “the guy on my team who knows Swift” *didn’t* know this.) We had to always access it through an UnsafePointer, and construct an UnsafeBufferPointer from &data and dataLength. That all actually worked pretty straightforwardly, but then I started playing with the idea of a general purpose type to manage this kind of pointer, and it was at that point that I started doing ugly pointer arithmetic (to try to access the variable-length buffer at he end of the type).

···

--
Brent Royal-Gordon
Architechies

FWIW, this problem looks very much like one for which we did the
CoreAudio overlay: stdlib/public/SDK/CoreAudio/CoreAudio.swift

Dmitri

···

On Wed, Dec 9, 2015 at 1:15 PM, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

On the off chance you're trying to do loads from unaligned UnsafePointers, that's undefined in the current interface. You'll need to memcpy to well-aligned memory first.

The buffer I’m accessing is UInt8, which I believe is always aligned, right?

C:

        typedef struct MIDIMetaEvent
        {
                UInt8 metaEventType;
                UInt8 unused1;
                UInt8 unused2;
                UInt8 unused3;
                UInt32 dataLength;
                UInt8 data[1];
        } MIDIMetaEvent;

Swift:

        public struct MIDIMetaEvent {

            public var metaEventType: UInt8
            public var unused1: UInt8
            public var unused2: UInt8
            public var unused3: UInt8
            public var dataLength: UInt32
            public var data: (UInt8)
            public init()
            public init(metaEventType: UInt8, unused1: UInt8, unused2: UInt8, unused3: UInt8, dataLength: UInt32, data: (UInt8))
        }

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/

Thanks Dimitri, Brent, and Joe for the feedback. Your additional comments
on the motivation were very helpful, Dimitri. If I'm not mistaken, there
has been no negative feedback expressed towards the proposal, nor have
there been any other alternatives proposed. I believe the next step in the
Swift evolution process is to draft a proposal in the swift-evolution
format and circulate it on this mailing list for refinement. So, without
further ado,

Introduction

Just as users can create Unsafe[Mutable]Pointers from Ints and UInts, they
should be able to create Ints and UInts from Unsafe[Mutable]Pointers. This
will allow users to call C functions with intptr_t and uintptr_t
parameters, and will allow users to perform more advanced pointer
arithmetic than is allowed by UnsafePointers.

Motivation

Swift currently lacks the ability to perform many complex operations on
pointers, such as checking pointer alignment, tagging pointers, or XORing
pointers (for working with XOR linked lists, for example). As a systems
programming language, Swift ought to be able to solve these problems
natively and concisely.

Additionally, since some C functions take intptr_t and uintptr_t
parameters, Swift currently has no ability to call these functions
directly. Users must wrap calls to these functions in C code.

Proposed solution

Initializers will be added to Int and UInt to convert from UnsafePointer
and UnsafeMutablePointer.

Currently, the only workaround which can solve these problems is to write
any code that requires pointer arithmetic in C. Writing this code in Swift
will be no safer than it is in C, as this is a fundamentally unsafe
operation. However, it will be cleaner in that users will not be forced to
write C code.

Detailed design

The initializers will be implemented using the built-in ptrtoint_Word
function.

extension UInt {
  init<T>(_ bitPattern: UnsafePointer<T>) {
    self = UInt(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }

  init<T>(_ bitPattern: UnsafeMutablePointer<T>) {
    self = UInt(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }
}

extension Int {
  init<T>(_ bitPattern: UnsafePointer<T>) {
    self = Int(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }

  init<T>(_ bitPattern: UnsafeMutablePointer<T>) {
    self = Int(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }
}

As an example, these initializers will allow the user to get the next
address of an XOR linked list in Swift.

struct XORLinkedList<T> {
  let address: UnsafePointer<T>

  ...

  func successor(_ predecessor: XORLinkedList<T>) -> XORLinkedList<T> {
    return XorLinkedList(UnsafePointer<T>(UInt(address) ^
UInt(predecessor.address)))
  }
}

Impact on existing code

There is no impact on existing code.

Alternatives considered

Three alternatives were considered.

The first alternative was to add an intValue function to
Unsafe[Mutable]Pointer. This alternative was rejected because it is
preferred that type conversions be implemented as initializers where
possible.

The next alternative was to add functions to Unsafe[Mutable]Pointer which
covered the identified pointer arithmetic cases. This alternative was
rejected because it either would have required us to imagine every use-case
of pointer arithmetic and write functions for them, which is an impossible
task, or it would have required adding a full suite of arithmetic and
bitwise operators to Unsafe[Mutable]Pointer. Because some of these
operations are defined only on signed integers, and others on unsigned, it
would have required splitting Unsafe[Mutable]Pointer into signed and
unsigned variants, which would have complicated things for users who did
not need to do pointer arithmetic. Additionally, the implementations of
these operations would have probably converted the pointers to integers,
perform a single operation, and then convert them back. When chaining
operations, this would create a lot of unnecessary conversions.

The last alternative was to forgo these initializers and force users to
write all their complicated pointer code in C. This alternative was
rejected because it makes Swift less useful as a systems programming
language.

···

On Wed, Dec 9, 2015 at 1:55 PM, Dmitri Gribenko via swift-evolution < swift-evolution@swift.org> wrote:

On Wed, Dec 9, 2015 at 1:15 PM, Brent Royal-Gordon via swift-evolution > <swift-evolution@swift.org> wrote:
>> On the off chance you're trying to do loads from unaligned
UnsafePointers, that's undefined in the current interface. You'll need to
memcpy to well-aligned memory first.
>
> The buffer I’m accessing is UInt8, which I believe is always aligned,
right?
>
> C:
>
> typedef struct MIDIMetaEvent
> {
> UInt8 metaEventType;
> UInt8 unused1;
> UInt8 unused2;
> UInt8 unused3;
> UInt32 dataLength;
> UInt8 data[1];
> } MIDIMetaEvent;
>
> Swift:
>
> public struct MIDIMetaEvent {
>
> public var metaEventType: UInt8
> public var unused1: UInt8
> public var unused2: UInt8
> public var unused3: UInt8
> public var dataLength: UInt32
> public var data: (UInt8)
> public init()
> public init(metaEventType: UInt8, unused1: UInt8, unused2:
UInt8, unused3: UInt8, dataLength: UInt32, data: (UInt8))
> }

FWIW, this problem looks very much like one for which we did the
CoreAudio overlay: stdlib/public/SDK/CoreAudio/CoreAudio.swift

Dmitri

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Thanks Michael! This looks good, please submit a pull request to
swift-evolution.

Dmitri

···

On Thu, Dec 10, 2015 at 9:04 PM, Michael Buckley via swift-evolution < swift-evolution@swift.org> wrote:

Thanks Dimitri, Brent, and Joe for the feedback. Your additional comments
on the motivation were very helpful, Dimitri. If I'm not mistaken, there
has been no negative feedback expressed towards the proposal, nor have
there been any other alternatives proposed. I believe the next step in the
Swift evolution process is to draft a proposal in the swift-evolution
format and circulate it on this mailing list for refinement. So, without
further ado,

Introduction

Just as users can create Unsafe[Mutable]Pointers from Ints and UInts, they
should be able to create Ints and UInts from Unsafe[Mutable]Pointers. This
will allow users to call C functions with intptr_t and uintptr_t
parameters, and will allow users to perform more advanced pointer
arithmetic than is allowed by UnsafePointers.

Motivation

Swift currently lacks the ability to perform many complex operations on
pointers, such as checking pointer alignment, tagging pointers, or XORing
pointers (for working with XOR linked lists, for example). As a systems
programming language, Swift ought to be able to solve these problems
natively and concisely.

Additionally, since some C functions take intptr_t and uintptr_t
parameters, Swift currently has no ability to call these functions
directly. Users must wrap calls to these functions in C code.

Proposed solution

Initializers will be added to Int and UInt to convert from UnsafePointer
and UnsafeMutablePointer.

Currently, the only workaround which can solve these problems is to write
any code that requires pointer arithmetic in C. Writing this code in Swift
will be no safer than it is in C, as this is a fundamentally unsafe
operation. However, it will be cleaner in that users will not be forced to
write C code.

Detailed design

The initializers will be implemented using the built-in ptrtoint_Word
function.

extension UInt {
  init<T>(_ bitPattern: UnsafePointer<T>) {
    self = UInt(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }

  init<T>(_ bitPattern: UnsafeMutablePointer<T>) {
    self = UInt(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }
}

extension Int {
  init<T>(_ bitPattern: UnsafePointer<T>) {
    self = Int(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }

  init<T>(_ bitPattern: UnsafeMutablePointer<T>) {
    self = Int(Builtin.ptrtoint_Word(bitPattern._rawValue))
  }
}

As an example, these initializers will allow the user to get the next
address of an XOR linked list in Swift.

struct XORLinkedList<T> {
  let address: UnsafePointer<T>

  ...

  func successor(_ predecessor: XORLinkedList<T>) -> XORLinkedList<T> {
    return XorLinkedList(UnsafePointer<T>(UInt(address) ^
UInt(predecessor.address)))
  }
}

Impact on existing code

There is no impact on existing code.

Alternatives considered

Three alternatives were considered.

The first alternative was to add an intValue function to
Unsafe[Mutable]Pointer. This alternative was rejected because it is
preferred that type conversions be implemented as initializers where
possible.

The next alternative was to add functions to Unsafe[Mutable]Pointer which
covered the identified pointer arithmetic cases. This alternative was
rejected because it either would have required us to imagine every use-case
of pointer arithmetic and write functions for them, which is an impossible
task, or it would have required adding a full suite of arithmetic and
bitwise operators to Unsafe[Mutable]Pointer. Because some of these
operations are defined only on signed integers, and others on unsigned, it
would have required splitting Unsafe[Mutable]Pointer into signed and
unsigned variants, which would have complicated things for users who did
not need to do pointer arithmetic. Additionally, the implementations of
these operations would have probably converted the pointers to integers,
perform a single operation, and then convert them back. When chaining
operations, this would create a lot of unnecessary conversions.

The last alternative was to forgo these initializers and force users to
write all their complicated pointer code in C. This alternative was
rejected because it makes Swift less useful as a systems programming
language.

On Wed, Dec 9, 2015 at 1:55 PM, Dmitri Gribenko via swift-evolution < > swift-evolution@swift.org> wrote:

On Wed, Dec 9, 2015 at 1:15 PM, Brent Royal-Gordon via swift-evolution >> <swift-evolution@swift.org> wrote:
>> On the off chance you're trying to do loads from unaligned
UnsafePointers, that's undefined in the current interface. You'll need to
memcpy to well-aligned memory first.
>
> The buffer I’m accessing is UInt8, which I believe is always aligned,
right?
>
> C:
>
> typedef struct MIDIMetaEvent
> {
> UInt8 metaEventType;
> UInt8 unused1;
> UInt8 unused2;
> UInt8 unused3;
> UInt32 dataLength;
> UInt8 data[1];
> } MIDIMetaEvent;
>
> Swift:
>
> public struct MIDIMetaEvent {
>
> public var metaEventType: UInt8
> public var unused1: UInt8
> public var unused2: UInt8
> public var unused3: UInt8
> public var dataLength: UInt32
> public var data: (UInt8)
> public init()
> public init(metaEventType: UInt8, unused1: UInt8, unused2:
UInt8, unused3: UInt8, dataLength: UInt32, data: (UInt8))
> }

FWIW, this problem looks very much like one for which we did the
CoreAudio overlay: stdlib/public/SDK/CoreAudio/CoreAudio.swift

Dmitri

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/