Proposed amendment to SE-0138: Normalize UnsafeRawBufferPointer Slices

Andrew_Trick · 2017-03-21T02:21:21.000Z

This proposal amends SE-0138: Normalize UnsafeRawBufferPointer Slices
to fix a design bug: Amend SE-0138: UnsafeRawBufferPointer - Normalize slice type. by atrick · Pull Request #651 · apple/swift-evolution · GitHub

The issue was discussed on swift-evolution in Nov/Dec:
See [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers
https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20161128/029108.html

The implementation of this fix is in PR #8222:

github.com/apple/swift

SE-0138: Proposed amendment to SE-0138: Normalize UnsafeRawBufferPointer Slices

apple:master ← atrick:rawbufslice

opened 02:14AM - 21 Mar 17 UTC

atrick

+700 -373

See the swift-evolution PR to amend SE-0138: https://github.com/apple/swift-evo…lution/pull/651 Fix: Change `Unsafe${Mutable}RawBufferPointer`'s `SubSequence` type Original: `Unsafe${Mutable}RawBufferPointer.SubSequence = Unsafe${Mutable}RawBufferPointer` Fixed: `Unsafe${Mutable}RawBufferPointer.SubSequence = ${Mutable}RandomAccessSlice<Unsafe${Mutable}RawBufferPointer>` This is a source breaking bug fix that only applies to post-3.0.1. It's extremely unlikely that any Swift 3 code would rely on the Subsequence type beyond the simple use case of passing a raw buffer subrange to an another raw buffer argument: `takesRawBuffer(buffer[i..<j])` A diagnostic message now instructs users to convert the slice to a buffer using a `rebasing` initializer: `takesRawBuffer(UnsafeRawBufferPointer(rebasing: buffer[i..<j]))` To support this, the following `rebasing` initializers are added: ``` extension UnsafeRawBufferPointer { public init(rebasing slice: RandomAccessSlice<UnsafeRawBufferPointer>) public init( rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer> ) } extension UnsafeMutableRawBufferPointer { public init( rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer> ) } ``` The source compatibility test builds are unnaffected by this change.

Fix: Change Unsafe[Mutable]RawBufferPointer's SubSequence type

Original: Unsafe[Mutable]RawBufferPointer.SubSequence = Unsafe[Mutable]RawBufferPointer

Fixed: Unsafe[Mutable]RawBufferPointer.SubSequence = [Mutable]RandomAccessSlice<Unsafe[Mutable]RawBufferPointer>

This is a source breaking bug fix that only applies to
post-3.0.1. It's extremely unlikely that any Swift 3 code would rely
on the SubSequence type beyond the simple use case of passing a
raw buffer subrange to an another raw buffer argument:

`takesRawBuffer(buffer[i..<j])`

A diagnostic message now instructs users to convert the slice to a
buffer using a `rebasing` initializer:

`takesRawBuffer(UnsafeRawBufferPointer(rebasing: buffer[i..<j]))`

To support this, the following `rebasing` initializers are added:

extension UnsafeRawBufferPointer {
  public init(rebasing slice: RandomAccessSlice<UnsafeRawBufferPointer>)
  public init(
    rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer>
  )
}

extension UnsafeMutableRawBufferPointer {
  public init(
    rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer>
  )
}

The source compatibility test builds are unnaffected by this change.

-Andy

Karl · 2017-03-24T02:22:31.000Z

The convenience initialiser should exist on all of the unsafe buffers, not just the raw (untyped) ones.

I’ve run in to this problem a few times, and I think it would get worse if we adopted a ContiguouslyStored protocol to formalise accessing the raw-pointers of generic collections. It would mean that you couldn’t write code that works with UnsafeRawBufferPointer/Data/DispatchData generically, or with UnsafeBufferPointer<T>/Array<T>.

Also, there seem to be some implicit conversions for the unsafe-pointer types, but UMBP -> UBP requires an awkward initialiser. We should introduce an implicit conversion for that case or add an “immutable” computed property to UMBP.

And while we’re on the subject, memory allocation/deallocation functions are weirdly dispersed. In order to allocate an UnsafeMutableBufferPointer<T>, for instance, you have to do:

var buffer: UnsafeMutableBufferPointer<T>
init(length: Int) {
let b = UnsafeMutablePointer<T>.allocate(capacity: length)
buffer = UnsafeMutableBufferPointer(start: b, count: length)
}

Also, the deallocate API feels weird - since it deallocates n items from the head of the pointer, it is a consuming operation and I feel like it should return a new pointer (with @discardableResult). Once you’ve deallocated a memory address, you can never re-allocate that specific location so there is no reason to know about it any more.

- Karl

···

On 21 Mar 2017, at 03:21, Andrew Trick via swift-evolution <swift-evolution@swift.org> wrote:

This proposal amends SE-0138: Normalize UnsafeRawBufferPointer Slices
to fix a design bug: Amend SE-0138: UnsafeRawBufferPointer - Normalize slice type. by atrick · Pull Request #651 · apple/swift-evolution · GitHub

The issue was discussed on swift-evolution in Nov/Dec:
See [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers
[swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

The implementation of this fix is in PR #8222:
SE-0138: Proposed amendment to SE-0138: Normalize UnsafeRawBufferPointer Slices by atrick · Pull Request #8222 · apple/swift · GitHub

Fix: Change Unsafe[Mutable]RawBufferPointer's SubSequence type

Original: Unsafe[Mutable]RawBufferPointer.SubSequence = Unsafe[Mutable]RawBufferPointer

Fixed: Unsafe[Mutable]RawBufferPointer.SubSequence = [Mutable]RandomAccessSlice<Unsafe[Mutable]RawBufferPointer>

This is a source breaking bug fix that only applies to
post-3.0.1. It's extremely unlikely that any Swift 3 code would rely
on the SubSequence type beyond the simple use case of passing a
raw buffer subrange to an another raw buffer argument:

`takesRawBuffer(buffer[i..<j])`

A diagnostic message now instructs users to convert the slice to a
buffer using a `rebasing` initializer:

`takesRawBuffer(UnsafeRawBufferPointer(rebasing: buffer[i..<j]))`

To support this, the following `rebasing` initializers are added:

extension UnsafeRawBufferPointer {
public init(rebasing slice: RandomAccessSlice<UnsafeRawBufferPointer>)
public init(
rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer>
)
}

extension UnsafeMutableRawBufferPointer {
public init(
rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer>
)
}

The source compatibility test builds are unnaffected by this change.

-Andy
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Andrew_Trick · 2017-03-24T06:49:39.000Z

This is all valuable feedback. I also have a bunch of convenience APIs in
mind but haven't been pushing them yet because:

- the Swift 4 schedule is tight

- I don't want to speculatively add API surface until enough users
have had experience with the feature. Maybe there are better ideas.
…

The convenience initialiser should exist on all of the unsafe buffers, not just the raw (untyped) ones.

I think it's fair to add the UnsafeBuffer.init(rebasing:) initializer
as part of this SE-0138 amendment. It's not so much a new API as a
consistency fix. I’ll update the current proposal.

I’ve run in to this problem a few times, and I think it would get worse if we adopted a ContiguouslyStored protocol to formalise accessing the raw-pointers of generic collections. It would mean that you couldn’t write code that works with UnsafeRawBufferPointer/Data/DispatchData generically, or with UnsafeBufferPointer<T>/Array<T>.

Also, there seem to be some implicit conversions for the unsafe-pointer types, but UMBP -> UBP requires an awkward initialiser. We should introduce an implicit conversion for that case or add an “immutable” computed property to UMBP.

UnsafeBufferPointer should have init from mutable. We already had a bug for that.
Thanks for reminding me: [SR-3929] UnsafeBufferPointer should have init from mutable · Issue #46514 · apple/swift · GitHub

I think that needs a new proposal, but it's a trivial addition. I can work on that.

I agree that implicit conversions would be nice but does merit some
discussion and involves a bit of type system work. You have to be a
bit careful with implicit conversion because of potentially ambigous
overloads. Implicit conversion is primarily for C interop. Eventually
I do think it would be nice to optionally import some pointer+length
arguments as UnsafeBuffer. We just don't do it now. Regardless of
interop, this would also be handy way to pass an inout argument as a
buffer without wrapping it in a closure.

Your specific concern seems to be about the mutable -> immutable
conversion being implicit though. But that just comes down to writing out
the type name. As bad as the type name is, to me that's not worth an
evolution proposal at the moment. I suggest filing a bug for any
implicit conversion issues, or `immutable` property suggestions for now.

And while we’re on the subject, memory allocation/deallocation functions are weirdly dispersed. In order to allocate an UnsafeMutableBufferPointer<T>, for instance, you have to do:

var buffer: UnsafeMutableBufferPointer<T>
init(length: Int) {
let b = UnsafeMutablePointer<T>.allocate(capacity: length)
buffer = UnsafeMutableBufferPointer(start: b, count: length)
}

We've had a bug open on this since introducing raw buffers:

github.com/apple/swift

[SR-3088] UnsafeMutableBufferPointer doesn't have an allocating init

opened 02:16AM - 31 Oct 16 UTC

closed 01:09AM - 07 Dec 17 UTC

karwa

improvement standard library

| | | |------------------|-----------------|… |Previous ID | SR-3088 | |Radar | rdar://problem/31236583 | |Original Reporter | @karwa | |Type | Improvement | |Status | Closed | |Resolution | Done | <details> <summary>Additional Detail from JIRA</summary> | | | |------------------|-----------------| |Votes | 0 | |Component/s | Standard Library | |Labels | Improvement | |Assignee | None | |Priority | Medium | md5: 49bfddea8da991130821d438f3d14f45 </details> **Issue Description:** We have: - UnsafeMutablePointer\<T\>.allocate(...), and - UnsafeMutableRawBufferPointer.allocate(...) but no UnsafeMutableBufferPointer\<T\>.allocate(...) The workaround at the moment is to allocate a UMP and create a UMBP which points to it: ``` java let buffer = UnsafeMutablePointer<T>.allocate(capacity: n) let bufferPtr = UnsafeMutableBufferPointer<T>(start: buffer, count: n) ```

This is obvious enough that I can probably sneak it into Swift 4 if I just get around to writing the proposal.

Also, the deallocate API feels weird - since it deallocates n items from the head of the pointer, it is a consuming operation and I feel like it should return a new pointer (with @discardableResult). Once you’ve deallocated a memory address, you can never re-allocate that specific location so there is no reason to know about it any more.

Now we're outside UnsafeBufferPointer territory.
UnsafePointer.deallocate(capacity:) needs to be passed the
same capacity as allocate. If that's confusing, please file a
documentation bug.

Or you may be referring to UnsafePointer.deinitialize(count:). That
returns a raw pointer to the same memory address to indicate that you
can now initialize the same memory as a different type.

With move-only types, I think we'll be able to revisit some of the
pointer initialize/deinitialize API to be more robust.

-Andy

···

On Mar 23, 2017, at 7:22 PM, Karl Wagner <razielim@gmail.com> wrote:

- Karl

On 21 Mar 2017, at 03:21, Andrew Trick via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

This proposal amends SE-0138: Normalize UnsafeRawBufferPointer Slices
to fix a design bug: Amend SE-0138: UnsafeRawBufferPointer - Normalize slice type. by atrick · Pull Request #651 · apple/swift-evolution · GitHub

The issue was discussed on swift-evolution in Nov/Dec:
See [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers
[swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

The implementation of this fix is in PR #8222:
SE-0138: Proposed amendment to SE-0138: Normalize UnsafeRawBufferPointer Slices by atrick · Pull Request #8222 · apple/swift · GitHub

Fix: Change Unsafe[Mutable]RawBufferPointer's SubSequence type

Original: Unsafe[Mutable]RawBufferPointer.SubSequence = Unsafe[Mutable]RawBufferPointer

Fixed: Unsafe[Mutable]RawBufferPointer.SubSequence = [Mutable]RandomAccessSlice<Unsafe[Mutable]RawBufferPointer>

This is a source breaking bug fix that only applies to
post-3.0.1. It's extremely unlikely that any Swift 3 code would rely
on the SubSequence type beyond the simple use case of passing a
raw buffer subrange to an another raw buffer argument:

`takesRawBuffer(buffer[i..<j])`

A diagnostic message now instructs users to convert the slice to a
buffer using a `rebasing` initializer:

`takesRawBuffer(UnsafeRawBufferPointer(rebasing: buffer[i..<j]))`

To support this, the following `rebasing` initializers are added:

extension UnsafeRawBufferPointer {
public init(rebasing slice: RandomAccessSlice<UnsafeRawBufferPointer>)
public init(
rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer>
)
}

extension UnsafeMutableRawBufferPointer {
public init(
rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer>
)
}

The source compatibility test builds are unnaffected by this change.

-Andy
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Karl · 2017-03-24T02:34:33.000Z

Oh, one more thing (about this specific change): If we do this, we should add an “offset” parameter to UnsafeMutableBufferPointer.[move]initialize/assign (with a default of 0 for source compatibility). Otherwise, it becomes awkward to initialise a region of a buffer from another buffer.

What I want to write:
buffer.suffix(from: filled).initialize(from: newData)

If SubSequence is not another unsafe pointer, I’d have to do this:
buffer.baseAddress!.advanced(by: filled).initialize(from: newData) // Warning: deprecated in Swift 4.0

So instead, we should have the ability to write this:
buffer.initialize(startingAt: filled, from: newData) // awkward labels, but src compat...

But yeah, this post just reminded me that there are a number of small consistency tweaks we could make to the unsafe-buffer API.

- Karl

···

On 24 Mar 2017, at 03:22, Karl Wagner <karl.swift@springsup.com> wrote:

The convenience initialiser should exist on all of the unsafe buffers, not just the raw (untyped) ones.

I’ve run in to this problem a few times, and I think it would get worse if we adopted a ContiguouslyStored protocol to formalise accessing the raw-pointers of generic collections. It would mean that you couldn’t write code that works with UnsafeRawBufferPointer/Data/DispatchData generically, or with UnsafeBufferPointer<T>/Array<T>.

Also, there seem to be some implicit conversions for the unsafe-pointer types, but UMBP -> UBP requires an awkward initialiser. We should introduce an implicit conversion for that case or add an “immutable” computed property to UMBP.

And while we’re on the subject, memory allocation/deallocation functions are weirdly dispersed. In order to allocate an UnsafeMutableBufferPointer<T>, for instance, you have to do:

var buffer: UnsafeMutableBufferPointer<T>
init(length: Int) {
  let b = UnsafeMutablePointer<T>.allocate(capacity: length)
  buffer = UnsafeMutableBufferPointer(start: b, count: length)
}

Also, the deallocate API feels weird - since it deallocates n items from the head of the pointer, it is a consuming operation and I feel like it should return a new pointer (with @discardableResult). Once you’ve deallocated a memory address, you can never re-allocate that specific location so there is no reason to know about it any more.

- Karl

On 21 Mar 2017, at 03:21, Andrew Trick via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

This proposal amends SE-0138: Normalize UnsafeRawBufferPointer Slices
to fix a design bug: Amend SE-0138: UnsafeRawBufferPointer - Normalize slice type. by atrick · Pull Request #651 · apple/swift-evolution · GitHub

The issue was discussed on swift-evolution in Nov/Dec:
See [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers
[swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

The implementation of this fix is in PR #8222:
SE-0138: Proposed amendment to SE-0138: Normalize UnsafeRawBufferPointer Slices by atrick · Pull Request #8222 · apple/swift · GitHub

Fix: Change Unsafe[Mutable]RawBufferPointer's SubSequence type

Original: Unsafe[Mutable]RawBufferPointer.SubSequence = Unsafe[Mutable]RawBufferPointer

Fixed: Unsafe[Mutable]RawBufferPointer.SubSequence = [Mutable]RandomAccessSlice<Unsafe[Mutable]RawBufferPointer>

This is a source breaking bug fix that only applies to
post-3.0.1. It's extremely unlikely that any Swift 3 code would rely
on the SubSequence type beyond the simple use case of passing a
raw buffer subrange to an another raw buffer argument:

`takesRawBuffer(buffer[i..<j])`

A diagnostic message now instructs users to convert the slice to a
buffer using a `rebasing` initializer:

`takesRawBuffer(UnsafeRawBufferPointer(rebasing: buffer[i..<j]))`

To support this, the following `rebasing` initializers are added:

extension UnsafeRawBufferPointer {
public init(rebasing slice: RandomAccessSlice<UnsafeRawBufferPointer>)
public init(
   rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer>
)
}

extension UnsafeMutableRawBufferPointer {
public init(
   rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer>
)
}

The source compatibility test builds are unnaffected by this change.

-Andy
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Andrew_Trick · 2017-03-24T06:49:51.000Z

Oh, one more thing (about this specific change): If we do this, we should add an “offset” parameter to UnsafeMutableBufferPointer.[move]initialize/assign (with a default of 0 for source compatibility). Otherwise, it becomes awkward to initialise a region of a buffer from another buffer.

What I want to write:
buffer.suffix(from: filled).initialize(from: newData)

If SubSequence is not another unsafe pointer, I’d have to do this:
buffer.baseAddress!.advanced(by: filled).initialize(from: newData) // Warning: deprecated in Swift 4.0

Oh no. Don't do that. You're supposed to do:

UnsafeMutableRawBufferPointer(rebasing: buffer.suffix(from: filled)).initialize(from: newData)

I don't blame you for not wanting to write out the type name.

So instead, we should have the ability to write this:
buffer.initialize(startingAt: filled, from: newData) // awkward labels, but src compat…

I agree that's worth doing as a result of changing the slice type. You
shouldn't need to "rebase" the buffer each time you want to initialize
a region. My concern is that there are a handful of APIs to fix
(initialize, initializeMemory, copyBytes, load). We should handle them
all in one proposal, and we may want time to bikeshed the API. I'm
unconvinced this is worth pursuing for Swift 4. Can you file a bug?

But yeah, this post just reminded me that there are a number of small consistency tweaks we could make to the unsafe-buffer API.

Yeah, but these are convenience issues, not correctness issues.

I think these are the ones that are important, obvious, and trivial
enough that might make sense in Swift 4.

** UnsafeBufferPointer should have init from mutable

github.com/apple/swift

[SR-3929] UnsafeBufferPointer should have init from mutable

opened 12:49AM - 11 Feb 17 UTC

closed 01:09AM - 07 Dec 17 UTC

177d8476-2756-4152-91d7-984f74d3896c

improvement feature standard library

| | | |------------------|-----------------|… |Previous ID | SR-3929 | |Radar | rdar://problem/31236319 | |Original Reporter | @milseman | |Type | Improvement | |Status | Closed | |Resolution | Done | <details> <summary>Additional Detail from JIRA</summary> | | | |------------------|-----------------| |Votes | 0 | |Component/s | Standard Library | |Labels | Improvement, LanguageFeatureRequest | |Assignee | @atrick | |Priority | Medium | md5: 6cfa9b4be4e1783cece7fcf8f1833adf </details> **Issue Description:** UnsafeBufferPointer should have an initializer that takes a UnsafeMutableBufferPointer of the same type, and vice versa.

** UnsafeMutableBufferPointer doesn't have an allocating init

github.com/apple/swift

[SR-3088] UnsafeMutableBufferPointer doesn't have an allocating init

opened 02:16AM - 31 Oct 16 UTC

closed 01:09AM - 07 Dec 17 UTC

karwa

improvement standard library

| | | |------------------|-----------------|… |Previous ID | SR-3088 | |Radar | rdar://problem/31236583 | |Original Reporter | @karwa | |Type | Improvement | |Status | Closed | |Resolution | Done | <details> <summary>Additional Detail from JIRA</summary> | | | |------------------|-----------------| |Votes | 0 | |Component/s | Standard Library | |Labels | Improvement | |Assignee | None | |Priority | Medium | md5: 49bfddea8da991130821d438f3d14f45 </details> **Issue Description:** We have: - UnsafeMutablePointer\<T\>.allocate(...), and - UnsafeMutableRawBufferPointer.allocate(...) but no UnsafeMutableBufferPointer\<T\>.allocate(...) The workaround at the moment is to allocate a UMP and create a UMBP which points to it: ``` java let buffer = UnsafeMutablePointer<T>.allocate(capacity: n) let bufferPtr = UnsafeMutableBufferPointer<T>(start: buffer, count: n) ```

** UnsafeBufferPointer needs a withMemoryRebound method

github.com/apple/swift

[SR-4340] UnsafeBufferPointer needs a withMemoryRebound method

opened 06:56AM - 24 Mar 17 UTC

closed 01:08AM - 07 Dec 17 UTC

atrick

improvement feature standard library

| | | |------------------|-----------------|… |Previous ID | SR-4340 | |Radar | rdar://problem/31236179 | |Original Reporter | @atrick | |Type | Improvement | |Status | Closed | |Resolution | Done | <details> <summary>Additional Detail from JIRA</summary> | | | |------------------|-----------------| |Votes | 0 | |Component/s | Standard Library | |Labels | Improvement, LanguageFeatureRequest | |Assignee | None | |Priority | Medium | md5: 97500090b48ef769f616d5192ac5f06c </details> **Issue Description:** Currently, rebinding memory requires manually dropping down to \`baseAddress\` + \`count\`: ``` java let count = p.count p.baseAddress!.withMemoryRebound(to: Int64.self, capacity: count) { bar(UnsafeBufferPointer(start: $0, count: count)) } ``` We should obviously add a \`withMemoryRebound(capacity🙂\` to UnsafeBufferPointer and UnsafeMutableBufferPointer. While we're at it, UnsafeRawBufferPointer should have a bindMemory API.

** Implicit Conversion: &Tuple to UnsafePointer<Tuple.Element>

github.com/apple/swift

[SR-3590] Implicitly convert &Tuple to UnsafePointer<Tuple.Element>

opened 06:27AM - 10 Jan 17 UTC

closed 05:56PM - 16 Jul 19 UTC

atrick

duplicate compiler feature swift evolution proposal needed

| | | |------------------|-----------------|… |Previous ID | SR-3590 | |Radar | rdar://problem/31236224 | |Original Reporter | @atrick | |Type | New Feature | |Status | Closed | |Resolution | Duplicate | <details> <summary>Additional Detail from JIRA</summary> | | | |------------------|-----------------| |Votes | 0 | |Component/s | Compiler | |Labels | New Feature, LanguageFeatureRequest | |Assignee | None | |Priority | Medium | md5: 324b77b6bdc9f425f17f3e1bfc47b11b </details> **duplicates**: * [SR-11147](https://bugs.swift.org/browse/SR-11147) Add withUnsafeBufferPointer<T: HomogeneousAggregate, Result> **relates to**: * [SR-4929](https://bugs.swift.org/browse/SR-4929) Support non-inout homogeneous tuple argument conversion to UnsafePointer<Element> **Issue Description:** For convenience add implicit casting from a tuple address to an unsafe pointer to its elements. This is a natural thing to do and should not require dropping down to a raw pointer and rebinding memory. See SR-2471. Example use case: ``` java import Foundation var bytes: (CChar, CChar, CChar, CChar) = (0x61, 0x62, 0x63, 0) let name: String = withUnsafePointer(to: &bytes) { (ptr: UnsafePointer<CChar>) -> String in return String(utf8String: ptr)! } ```

-Andy

···

On Mar 23, 2017, at 7:34 PM, Karl Wagner <razielim@gmail.com> wrote:

- Karl

On 24 Mar 2017, at 03:22, Karl Wagner <karl.swift@springsup.com <mailto:karl.swift@springsup.com>> wrote:

The convenience initialiser should exist on all of the unsafe buffers, not just the raw (untyped) ones.

I’ve run in to this problem a few times, and I think it would get worse if we adopted a ContiguouslyStored protocol to formalise accessing the raw-pointers of generic collections. It would mean that you couldn’t write code that works with UnsafeRawBufferPointer/Data/DispatchData generically, or with UnsafeBufferPointer<T>/Array<T>.

Also, there seem to be some implicit conversions for the unsafe-pointer types, but UMBP -> UBP requires an awkward initialiser. We should introduce an implicit conversion for that case or add an “immutable” computed property to UMBP.

And while we’re on the subject, memory allocation/deallocation functions are weirdly dispersed. In order to allocate an UnsafeMutableBufferPointer<T>, for instance, you have to do:

var buffer: UnsafeMutableBufferPointer<T>
init(length: Int) {
  let b = UnsafeMutablePointer<T>.allocate(capacity: length)
  buffer = UnsafeMutableBufferPointer(start: b, count: length)
}

Also, the deallocate API feels weird - since it deallocates n items from the head of the pointer, it is a consuming operation and I feel like it should return a new pointer (with @discardableResult). Once you’ve deallocated a memory address, you can never re-allocate that specific location so there is no reason to know about it any more.

- Karl

On 21 Mar 2017, at 03:21, Andrew Trick via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

This proposal amends SE-0138: Normalize UnsafeRawBufferPointer Slices
to fix a design bug: Amend SE-0138: UnsafeRawBufferPointer - Normalize slice type. by atrick · Pull Request #651 · apple/swift-evolution · GitHub

The issue was discussed on swift-evolution in Nov/Dec:
See [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers
[swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

The implementation of this fix is in PR #8222:
SE-0138: Proposed amendment to SE-0138: Normalize UnsafeRawBufferPointer Slices by atrick · Pull Request #8222 · apple/swift · GitHub

Fix: Change Unsafe[Mutable]RawBufferPointer's SubSequence type

Original: Unsafe[Mutable]RawBufferPointer.SubSequence = Unsafe[Mutable]RawBufferPointer

Fixed: Unsafe[Mutable]RawBufferPointer.SubSequence = [Mutable]RandomAccessSlice<Unsafe[Mutable]RawBufferPointer>

This is a source breaking bug fix that only applies to
post-3.0.1. It's extremely unlikely that any Swift 3 code would rely
on the SubSequence type beyond the simple use case of passing a
raw buffer subrange to an another raw buffer argument:

`takesRawBuffer(buffer[i..<j])`

A diagnostic message now instructs users to convert the slice to a
buffer using a `rebasing` initializer:

`takesRawBuffer(UnsafeRawBufferPointer(rebasing: buffer[i..<j]))`

To support this, the following `rebasing` initializers are added:

extension UnsafeRawBufferPointer {
public init(rebasing slice: RandomAccessSlice<UnsafeRawBufferPointer>)
public init(
   rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer>
)
}

extension UnsafeMutableRawBufferPointer {
public init(
   rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer>
)
}

The source compatibility test builds are unnaffected by this change.

-Andy
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Karl · 2017-03-24T08:40:12.000Z

Oh, one more thing (about this specific change): If we do this, we should add an “offset” parameter to UnsafeMutableBufferPointer.[move]initialize/assign (with a default of 0 for source compatibility). Otherwise, it becomes awkward to initialise a region of a buffer from another buffer.

What I want to write:
buffer.suffix(from: filled).initialize(from: newData)

If SubSequence is not another unsafe pointer, I’d have to do this:
buffer.baseAddress!.advanced(by: filled).initialize(from: newData) // Warning: deprecated in Swift 4.0

Oh no. Don't do that. You're supposed to do:

UnsafeMutableRawBufferPointer(rebasing: buffer.suffix(from: filled)).initialize(from: newData)

I don't blame you for not wanting to write out the type name.

So instead, we should have the ability to write this:
buffer.initialize(startingAt: filled, from: newData) // awkward labels, but src compat…

I agree that's worth doing as a result of changing the slice type. You
shouldn't need to "rebase" the buffer each time you want to initialize
a region. My concern is that there are a handful of APIs to fix
(initialize, initializeMemory, copyBytes, load). We should handle them
all in one proposal, and we may want time to bikeshed the API. I'm
unconvinced this is worth pursuing for Swift 4. Can you file a bug?

It might be related to the existing bug about unsafely initialising an Array’s storage (SR-3087). We could add these methods as a protocol extension or refined protocol on “UnsafelyInitializable”.

Then, in a similar fashion to SR-3631, which would introduce ContiguouslyStored and remove ArraySlice, Slice<T> could conditionally conform to that protocol when its base does. That would make the first example — buffer.suffix().initialize(from:), work.

github.com/apple/swift

[SR-3087] No way to arbitrarily initialise an Array's storage

opened 02:11AM - 31 Oct 16 UTC

closed 05:45AM - 24 Mar 19 UTC

karwa

improvement feature standard library swift evolution implemented

| | | |------------------|-----------------|… |Previous ID | SR-3087 | |Radar | rdar://problem/21880692 | |Original Reporter | @karwa | |Type | Improvement | |Status | Resolved | |Resolution | Done | <details> <summary>Additional Detail from JIRA</summary> | | | |------------------|-----------------| |Votes | 0 | |Component/s | Standard Library | |Labels | Improvement, LanguageFeatureRequest | |Assignee | @natecook1000 | |Priority | Medium | md5: 78d2891e46bb0f408db15202b1a1b736 </details> **relates to**: * [SR-3631](https://bugs.swift.org/browse/SR-3631) Introduce ContiguouslyStored protocol and kill ArraySlice type **Issue Description:** The standard library does not currently expose a way to allocate an Array's storage and (unsafely) initialise its elements, as we can with UnsafeMutablePointer. An example use-case is to implement a concurrent map. See: <https://lists.swift.org/pipermail/swift-users/Week-of-Mon-20161024/003769.html> Even though we expose a UMP from Array, it cannot be used to unsafely initialise the Array elements. The workaround at the moment is to allocate a UMP, initialise its elements unsafely, and then copy the contents in to an Array via the Array(Sequence) constructor, which is less than optimal.

github.com/apple/swift

[SR-3631] Introduce ContiguouslyStored protocol and kill ArraySlice type

opened 09:52PM - 13 Jan 17 UTC

closed 07:39PM - 11 Nov 22 UTC

dabrahams

improvement good first issue standard library affects ABI swift evolution implemented

| | | |------------------|-----------------|… |Previous ID | SR-3631 | |Radar | rdar://problem/30541077 | |Original Reporter | @dabrahams | |Type | Improvement | Attachment: [Download](https://user-images.githubusercontent.com/2727770/164961835-16c81216-05b1-4765-8dcb-0567f07f08a5.gz) <details> <summary>Additional Detail from JIRA</summary> | | | |------------------|-----------------| |Votes | 0 | |Component/s | Standard Library | |Labels | Improvement, AffectsABI, StarterProposal | |Assignee | None | |Priority | Medium | md5: 0b822da3966dafcaf6527038fc805b6a </details> **relates to**: * [SR-3633](https://bugs.swift.org/browse/SR-3633) Support for segmented data structures * [SR-3087](https://bugs.swift.org/browse/SR-3087) No way to arbitrarily initialise an Array's storage **Issue Description:** Sketch: ``` java protocol ContiguouslyStored : RandomAccessCollection { func withUnsafeBufferPointer<R>(...) -> R mutating func withUnsafeMutableBufferPointer<R>(...) -> R } extension Slice : ContiguouslyStored where Base : ContiguouslyStored { func withUnsafeBufferPointer<R>(...) -> R { ... } mutating func withUnsafeMutableBufferPointer<R>(...) -> R { ... } } typealias ArraySlice<T> = Slice<Array<T>> ``` (This is related to <https://bugs.swift.org/browse/SR-3633>)

Fleshing out those unsafe-initialisation protocols would be useful in general, not just for slices of unsafe buffers. For example, if you’re initialising such any such Collection from something which is ContiguouslyStored, we could optimise that operation quite a lot. Currently, the standard library has private, underscored methods on Sequence to do that: “_copyContentsToContiguousArray” and “_copyContents(initializing:)”.

I hope unsafe-initialisation is something which can make Swift 4. It would align well with other features in that release.

Where ownership is concerned, I feel we still need to express how code which wishes to avoid copies should get its data in to standard-library datatypes: the ownership manifesto only covers data at the Swift-type level, and provides no way to express ownership over an arbitrary buffer of allocated memory (that you might, for example, be able to safely wrap as an Array or String). Unsafe initialisation is a great answer to this - the standard library type owns its backing, which remains a private implementation detail, but you get the opportunity to have it allocate a buffer for you to fill up directly, using whichever unsafe operations you like and eliminating copies.

In particular, the new String model will expose its backing as a contiguous RandomAccessCollection of CodeUnits. It would be totally badass if we could unsafely initialise a standard Swift.String, using its backing storage directly as the buffer for file read/network recv operations.

- Karl

···

On 24 Mar 2017, at 07:49, Andrew Trick <atrick@apple.com> wrote:

On Mar 23, 2017, at 7:34 PM, Karl Wagner <razielim@gmail.com <mailto:razielim@gmail.com>> wrote:

But yeah, this post just reminded me that there are a number of small consistency tweaks we could make to the unsafe-buffer API.

Yeah, but these are convenience issues, not correctness issues.

I think these are the ones that are important, obvious, and trivial
enough that might make sense in Swift 4.

** UnsafeBufferPointer should have init from mutable
[SR-3929] UnsafeBufferPointer should have init from mutable · Issue #46514 · apple/swift · GitHub

** UnsafeMutableBufferPointer doesn't have an allocating init
[SR-3088] UnsafeMutableBufferPointer doesn't have an allocating init · Issue #45678 · apple/swift · GitHub

** UnsafeBufferPointer needs a withMemoryRebound method
[SR-4340] UnsafeBufferPointer needs a withMemoryRebound method · Issue #46919 · apple/swift · GitHub

** Implicit Conversion: &Tuple to UnsafePointer<Tuple.Element>
[SR-3590] Implicitly convert &Tuple to UnsafePointer<Tuple.Element> · Issue #46175 · apple/swift · GitHub

-Andy

- Karl

On 24 Mar 2017, at 03:22, Karl Wagner <karl.swift@springsup.com <mailto:karl.swift@springsup.com>> wrote:

The convenience initialiser should exist on all of the unsafe buffers, not just the raw (untyped) ones.

I’ve run in to this problem a few times, and I think it would get worse if we adopted a ContiguouslyStored protocol to formalise accessing the raw-pointers of generic collections. It would mean that you couldn’t write code that works with UnsafeRawBufferPointer/Data/DispatchData generically, or with UnsafeBufferPointer<T>/Array<T>.

Also, there seem to be some implicit conversions for the unsafe-pointer types, but UMBP -> UBP requires an awkward initialiser. We should introduce an implicit conversion for that case or add an “immutable” computed property to UMBP.

And while we’re on the subject, memory allocation/deallocation functions are weirdly dispersed. In order to allocate an UnsafeMutableBufferPointer<T>, for instance, you have to do:

var buffer: UnsafeMutableBufferPointer<T>
init(length: Int) {
  let b = UnsafeMutablePointer<T>.allocate(capacity: length)
  buffer = UnsafeMutableBufferPointer(start: b, count: length)
}

Also, the deallocate API feels weird - since it deallocates n items from the head of the pointer, it is a consuming operation and I feel like it should return a new pointer (with @discardableResult). Once you’ve deallocated a memory address, you can never re-allocate that specific location so there is no reason to know about it any more.

- Karl

On 21 Mar 2017, at 03:21, Andrew Trick via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

This proposal amends SE-0138: Normalize UnsafeRawBufferPointer Slices
to fix a design bug: Amend SE-0138: UnsafeRawBufferPointer - Normalize slice type. by atrick · Pull Request #651 · apple/swift-evolution · GitHub

The issue was discussed on swift-evolution in Nov/Dec:
See [swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers
[swift-evolution] [Pitch] Normalize Slice Types for Unsafe Buffers

The implementation of this fix is in PR #8222:
SE-0138: Proposed amendment to SE-0138: Normalize UnsafeRawBufferPointer Slices by atrick · Pull Request #8222 · apple/swift · GitHub

Fix: Change Unsafe[Mutable]RawBufferPointer's SubSequence type

Original: Unsafe[Mutable]RawBufferPointer.SubSequence = Unsafe[Mutable]RawBufferPointer

Fixed: Unsafe[Mutable]RawBufferPointer.SubSequence = [Mutable]RandomAccessSlice<Unsafe[Mutable]RawBufferPointer>

This is a source breaking bug fix that only applies to
post-3.0.1. It's extremely unlikely that any Swift 3 code would rely
on the SubSequence type beyond the simple use case of passing a
raw buffer subrange to an another raw buffer argument:

`takesRawBuffer(buffer[i..<j])`

A diagnostic message now instructs users to convert the slice to a
buffer using a `rebasing` initializer:

`takesRawBuffer(UnsafeRawBufferPointer(rebasing: buffer[i..<j]))`

To support this, the following `rebasing` initializers are added:

extension UnsafeRawBufferPointer {
public init(rebasing slice: RandomAccessSlice<UnsafeRawBufferPointer>)
public init(
   rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer>
)
}

extension UnsafeMutableRawBufferPointer {
public init(
   rebasing slice: MutableRandomAccessSlice<UnsafeMutableRawBufferPointer>
)
}

The source compatibility test builds are unnaffected by this change.

-Andy
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution