Get `UnsafeRawBufferPointer(s)` protocols

oscbyspro · 2023-02-23T10:08:17.743Z

Has promoting ContiguousBytes and/or DataProtocol to the standard library been discussed before? I find myself reaching for those, without touching anything else in Foundation, for the sole purpose of abstraction. While some Collection<UInt8> works, I would prefer 12 ways of getting a pointer to memory over 12 specializations of doLotsOfStuffWithLotsOfBytes(_:). I find it awkward that there's no semantic way of requesting UnsafeRawBufferPointer(s). I mean, [UInt8]/withUnsafe[Mutable]Bytes(_:) exists in the standard library, but not the abstraction for it.

Also, if Swift's stance is "Give me UnsafeRawBufferPointer(s) protocols belong in Foundation!" - is there a way to mimic these protocols without losing interoperability? Like, if I make my own MyDataProtocol - is there a way to say A) "if Foundation is imported, MyDataProtocol is DataProtocol" or B) "if Foundation is imported, all types conforming to DataProtocol also conform to MyDataProtocol"?

lukasa · 2023-02-23T10:52:20.119Z

Have you tried using Collection's withContiguousStorageIfAvailable?

oscbyspro · 2023-02-23T14:17:45.138Z

No, I was unaware of it, and I appreciate that you brought it to my attention. Although, it seems that it’s not quite equivalent since ContiguousBytes guarantees contiguous storage and DataProtocol is agnostic to it. As I’m unfamiliar with withContiguousStorageIfAvailable(_:), I assume it requires the caller to fall back on copying the storage when it’s discontiguous, which is bit unfortunate. My use case is similar to HashFunction’s use of combine(bufferPointer:) as the base for the more generic combine(_: some DataProtocol) in CryptoKit.

lukasa · 2023-02-24T14:06:01.852Z

oscbyspro:

requires the caller to fall back on copying the storage when it’s discontiguous, which is bit unfortunate.

As a practical matter there are almost no discontiguous collections in Swift.

bbrk24 · 2023-02-24T17:22:32.163Z

This statement surprises me. Off the top of my head, there’s at least two groups of collection types that aren’t stored linearly:

The various Range types, which store a start and end value rather than the whole collection
Algorithm types like ReversedCollection

I’m also not sure off the top of my head whether Set and Dictionary are contiguous; they might well be but I’d be unsurprised if not.

lukasa · 2023-02-24T17:23:20.420Z

That's fair, let me rephrase: there are almost no discontiguous collections that still qualify for DataProtocol conformance.

johannesweiss · 2023-02-25T09:52:01.747Z

The issue you'll get is that Data (and many other DataProtocol-implementing types) don't (and sometimes can't) implement withContiguousStorageIfAvailable (as in, it'll always return nil).

Data doesn't guarantee that the memory is bound to UInt8 -- in fact it specifically allows the user to bind it to whatever they need -- and hence can't implement withContiguousStorageIfAvailable. SwiftNIO's ByteBuffer is the only implementation that I'm aware of that implements both DataProtocol as well as withContiguousStorageIfAvailable. That works because ByteBuffer guarantees to always be bound to UInt8 and the user is disallowed from rebinding.

Karl · 2023-02-25T11:04:32.788Z

johannesweiss:

Data doesn't guarantee that the memory is bound to UInt8 -- in fact it specifically allows the user to bind it to whatever they need -- and hence can't implement withContiguousStorageIfAvailable.

I think that was relaxed recently:

Pitch: Add implementation for `withContiguousStorageIfAvailable()` to `Foundation.ContiguousBytes`-conforming sequence types

As of Feb 5, 2022, it is reasonably safe to implement Foundation.Data's withContiguousStorageIfAvailable by using withMemoryRebound: SE-0333 Expand usability of withMemoryRebound

CharlesS · 2023-02-25T17:09:23.582Z

Karl:

johannesweiss:

Data doesn't guarantee that the memory is bound to UInt8 -- in fact it specifically allows the user to bind it to whatever they need -- and hence can't implement withContiguousStorageIfAvailable.

I think that was relaxed recently:

Pitch: Add implementation for `withContiguousStorageIfAvailable()` to `Foundation.ContiguousBytes`-conforming sequence types

As of Feb 5, 2022, it is reasonably safe to implement Foundation.Data's withContiguousStorageIfAvailable by using withMemoryRebound: SE-0333 Expand usability of withMemoryRebound

This does not seem to be the case:

  1> import Foundation
  2> var data = "Foo Bar".data(using: .ascii)!
data: Foundation.Data = 7 bytes
  3> data.withContiguousStorageIfAvailable { _ in return 1 }
$R0: Int? = nil

I, too, would love to see a way to access byte buffers from UInt8 collections (including Data) in a library without forcing clients to link against Foundation.

The lack of this is actually obnoxious enough that I made my own library just to reimplement DataProtocol without requiring Foundation. It's not an ideal solution, though, since in order for Data and DispatchData to be properly supported, a client that actually does link against Foundation has to be sure to link against the CSDataProtocol+Foundation version of the library, or else any Datas that get passed in will end up taking the slow path. It would be great if we didn't have to do this.

DataProtocol is so small, lightweight, and fundamental in purpose that it really makes no sense for it not to be part of the standard library, IMO.

johannesweiss · 2023-02-25T21:00:06.107Z

@Karl Oh nice, I missed that, thanks!

@CharlesS yes, even on today's main it's not fixed yet:

$ echo 'import Foundation; print(Data("hello".utf8).withContiguousStorageIfAvailable { _ in "yay" } ?? "nay")' \
    | docker run -i --rm swiftlang/swift:nightly-main-jammy swift -
nay