[Pitch] Safe loading of integer values from `RawSpan`

[glessard (Guillaume Lessard)]

I just opened a PR to propose new safe-loading API to load concrete integer values from RawSpan and family: Safe loading of integer values for `RawSpan` by glessard · Pull Request #3065 · swiftlang/swift-evolution · GitHub

Motivation

In [SE-0447][SE-0447], we introduced RawSpan along with some unsafe functions to load values of arbitrary types. While it is safe to load any of the native integer types with those functions, the unsafe annotation introduces an element of doubt for users of the standard library. Furthermore, controlling the endiannness for the loading operation is not available at the point of serialization, introducing further confusion. This proposal adds the ability to safely load integer values with ergonomic endianness control, without the doubt introduced by unsafe functions.

Proposed solution

RawSpan

RawSpan will gain a series of concretely typed load(as:) functions to obtain numeric values from the underlying memory, with no alignment requirement. These load(as:) functions can be safe because they return values from fully-inhabited types, meaning that these types have a valid value for every bit pattern of their underlying bytes.

The load(as:) functions will be bounds-checked, being a safe RawSpan API. For example,

extension RawSpan {
  func load(fromByteOffset: Int = 0, as: UInt8.Type) -> UInt8

  func load(
    fromByteOffset: Int = 0, as: UInt16.Type, endianness: Endianness? = nil
  ) -> UInt16
}

@frozen
public enum Endianness: Equatable, Hashable, Sendable {
  case big, little
}

The loadable types will be UInt8, Int8, UInt16, Int16, UInt32, Int32, UInt64, Int64, UInt, Int, Float32 (aka Float) and Float64 (aka Double). On platforms that support them, loading Float16, Float80, UInt128, and Int128 values will also be supported. These are not atomic operations.

The concrete load(as:) functions will not have equivalents with unchecked byte offset. If that functionality is needed, the generic unsafeLoad(fromUncheckedByteOffset:as:) is already available.

The load(as:) functions will also be available for MutableRawSpan and OutputRawSpan.

MutableRawSpan and OutputRawSpan

MutableRawSpan will gain a series of concretely typed storeBytes() functions that accept an endianness parameter, while OutputRawSpan will have matching append() functions:

extension MutableRawSpan {
  mutating func storeBytes(
    of value: UInt16,
    toByteOffset offset: Int = 0,
    as type: UInt16.Type,
    endianness: Endianness
  )
}

extension OutputRawSpan {
  mutating func append(
    _ value: UInt16,
    as type: UInt16.Type,
    endianness: Endianness
  )
}

These functions do not have a default value for their endianness argument, as the existing generic MutableSpan.storeBytes(of:toByteOffset:as:) and OutputRawSpan.append(_:as:) functions use the native endianness, addressing this need.

These concrete implementations will support UInt16, Int16, UInt32, Int32, UInt64, Int64, UInt, Int, Float32 (aka Float) and Float64 (aka Double). On platforms that support them, Float16, Float80, UInt128, and Int128 values will also be supported. These are not atomic operations.

The concrete storeBytes(of:as:) functions will not have an equivalent with unchecked byte offset. If that functionality is needed, the generic storeBytes(of:toUncheckedByteOffset:as:) is already available.

Span

Span will gain a series of concrete initializers init(viewing: RawSpan) to allow viewing a range of untyped memory as a typed Span, when Span.Element is a numeric type. These conversions will check for alignment and bounds. For example,

extension Span {
  @_lifetime(borrow span)
  init(viewing bytes: borrowing RawSpan) where Element == UInt32
}

The supported element types will be UInt8, Int8, UInt16, Int16, UInt32, Int32, UInt64, Int64, UInt, Int, Float32 (aka Float) and Float64 (aka Double). On platforms that support them, initializing Span instances with Float16, Float80, UInt128, and Int128 elements will also be implemented.

The conversions from RawSpan to Span only support well-aligned views with the native endianness. The [swift-binary-parsing][swift-binary-parsing] package provides a more fully-featured ParserSpan type.

Please see the full proposal draft for the detailed design, alternatives and future directions.

[xwu (Xiaodi Wu)]

@glessard

@frozen
public enum Endianness: Equatable, Hashable, Sendable {
  case big, little
}

Two comments on this—

1. I think many peer languages call this just Endian rather than Endianness, which would also be supported by our naming guidelines since it'd be more succinct (and easier to say) without any loss of meaning.
   • The naming guidelines also tell us not to repeat the type in the argument label, so load(..., endianness: Endianness?) is not ideal; load(..., endianness: Endian) would be better. (And if we named the cases bigEndian and littleEndian as some other peer languages do, we could omit the label altogether.)
2. Using Endianness? = nil as the convention to mean native endianness feels awkward. Since the enum is equatable, this would also mean that native endianness compares not-equal to both big and little, which doesn't seem right. Peer languages have a .native, which we could also have here as a static property.

---

@glessard

RawSpan will gain a series of concretely typed load(as:) functions

Making these additions generic over FixedWidthInteger & BitwiseCopyable and BinaryFloatingPoint & BitwiseCopyable

These are not sufficient constraints, since they do not mandate that their conformers must be fully inhabited. This approach would also require extremely defensive implementations to account for the existence of conformers from outside of the standard library. It would be possible to make the additions generic over a new protocol, which would have to be public. Two new issues arise from creating a new protocol: backdeployment and the possibility of unsafety due to future defective conformers from outside the standard library. A layout constraint would achieve a similar outcome while being less risky.

Side note: We could make these additions generic over FixedWidthInteger & _ExpressibleByBuiltinIntegerLiteral, or some other similar internal-only constraint, no? Then it would be a matter of simply stating as policy that the standard library itself will never vend non-fully-inhabited conformers (which is doable).

[fclout]

I am not sure that we have ascertained the endianness options on RawSpan are the way we want to go:

• There is no Span variant that lets you control endianness; it’s annoying that we would say “use Span if you’re cool with native endianness, and if you don’t, the blessed pattern is to use RawSpan and count your offsets by hand”
• There is no way to control endianness for anything other than integers, which creates a weird cliff if you’re trying to load a struct (assuming that one day we do get to that fully inhabited protocol)
  • There are binary formats that use different endianness in different parts (or even for two integers next to one another), like Photoshop documents, so there is just no coherent way that specifying endianness for a struct can work in all cases
• Integers already have init(bigEndian:) and init(littleEndian:) for endianness conversion, which are good ways to deal with the problem (floating point types don’t, but they should)

I think this pushes RawSpan into parsing territory more than being the primitive for working with untyped memory, so I’m less hot for it.

[fclout]

When it comes to the core functionality: we do need safe load methods on RawSpan, and how to stage the functionality is not really my business. However, I think there’s a ton of value in a layout constraint or marker protocol that a type is fully inhabited. Loading things from a RawSpan is one super important use case of some kind of FullyInhabited constraint, but it’s not the only one. For instance, we should also have bitCast<From: BitwiseCopyable, To: FullyInhabited>. So, the main thing I’d like to check here is that if we think that FullyInhabited is going to happen one way or another, we’re not adding API surface to RawSpan that will make it unwieldy to prefer/transition to that later on. (I think it’s fine, but I wanted to bring it up in case somebody else sees something I’m not seeing.)

[glessard (Guillaume Lessard)]

@xwu

1. Using Endianness? = nil as the convention to mean native endianness feels awkward. Since the enum is equatable, this would also mean that native endianness compares not-equal to both big and little, which doesn't seem right. Peer languages have a .native, which we could also have here as a static property.

If we include a .native case, then the Endianness type wouldn’t be usable by the binary-parser package, for example. It would be perfectly fine to overload a one-parameter function with another that has two, including a non-defaulted Endianness parameter. I feel fine with the comparison awkwardness, because we don’t want to encourage endianness-dependent code.

A different question is whether we want to have “native” as a default. It seems to me this question has already been answered by the existing store(_:as:) functions, which have no endianness control.

[glessard (Guillaume Lessard)]

Naming: ByteOrder might be clearer and less jargony than re-using a joke from Jonathan Swift.

[glessard (Guillaume Lessard)]

@fclout

• Integers already have init(bigEndian:) and init(littleEndian:) for endianness conversion, which are good ways to deal with the problem (floating point types don’t, but they should)

I have a problem with these (and the related .bigEndian and .littleEndian properties,) in that they conflate types with their serialized representation. Endianness should be considered while serializing, which is where we store/load values to/from untyped buffers.

@fclout

• There is no way to control endianness for anything other than integers

That seems fine; if endianness is a concern for anything other than integers, then it has become a parsing problem, and values should be reconstructed safely with initializers.

@fclout

• There is no Span variant that lets you control endianness

Span's goal is to represent the computer’s memory representation as directly as possible. To control endianness would be a transformation, intrinsically indirect. It would be perfectly fine to write a transforming wrapper that does something else, but it shouldn’t be Span that does it.

@fclout

I think this pushes RawSpan into parsing territory more than being the primitive for working with untyped memory, so I’m less hot for it.

Parsing-adjacent use cases were always a major justification for RawSpan; parsing is mentioned in the first sentence introducing RawSpan in SE-0447!

[xwu (Xiaodi Wu)]

@glessard

If we include a .native case, then the Endianness type wouldn’t be usable by the binary-parser package, for example.

Can you explain?

[glessard (Guillaume Lessard)]

.native is only useful for parsing a stream that comes from your own process or another live process on your machine. When doing the kinds of tasks intended for the binary parsing package, .native would be an undesirable option to have to handle; they’d either trap on that branch or simply not use our type at all.

[xwu (Xiaodi Wu)]

@glessard

.native is only useful for parsing a stream that comes from your own process or another live process on your machine. When doing the kinds of tasks intended for the binary parsing package, .native would be an undesirable option to have to handle; they’d either trap on that branch or simply not use our type at all.

I'm not sure I entirely understand. As I said above, I'm thinking of .native as a static property which aliases either .big or .little depending on the platform; there would be nothing to 'handle' from the callee's side.

---

Naming: ByteOrder might be clearer and less jargony

That's nice: I think there's good precedent here given CFByteOrder.

The cases would naturally be spelled ByteOrder.bigEndian and ByteOrder.littleEndian, and no argument labels would then be needed for the functions as the call sites will read something like: load(..., as: Int.self, .bigEndian).

And on that note, an optional value would be a self-documenting and idiomatic way to model in modern Swift what CF has as a separate case, CFByteOrderUnknown. But I'm queasy about nil representing an actual know(n|able) endianness.

[glessard (Guillaume Lessard)]

@xwu

As I said above, I'm thinking of .native as a static property which aliases either .big or .little depending on the platform; there would be nothing to 'handle' from the callee's side.

I see. That might be okay. I’d personally prefer not to have an even more prominent way to conditionalize on endianness, but to be fair we can already ask whether 1.bigEndian == 1.

[YOCKOW]

I think this is a good case to introduce Endianness type, although I prefer ByteOrder to Endianness for its name.
I'd also want .native (or .current or .host) property, while I can't confidently say it's necessary.

---

Context

Endianness type including .native property was once proposed as a part of another proposal:

@Michael_Ilseman

Endianness

Endianness (byte-ordering or memory ordering), denotes whether the first byte received contains the high bits or low bits of a code unit. It is relevant for multi-byte encodings (UTF-16 and UTF-32, but not UTF-8).

public enum Endianness {
  case little
  case big

  /// The platform's native byte-ordering in memory 
  public static var native: Self
}

Alternative: Endianness could be considered in combination with the encoding, yielding e.g. UTF16BE and UTF16LE encodings. Or, it could be considered a property of a serialization format.

Alternative: An alternate name for Endianness could be ByteOrder.

...and I personally thought it was worth discussing such a type separately:

Tiny Pitch: `ByteOrder` type

Latest version is available at gist: Thread: https://forums.swift.org/t/tiny-pitch-byteorder-type/74027/ · GitHub ByteOrder type Introduction You may often take endianness (or byte-order) into account when you get involved in, for example, stream programming. While network byte-order is defined as big endian, you may have to handle some other stream whose endian is unknown in advance. In such case, you want a type that identifies endianness to write code like if theStream.byteOrder == .littleEn…

One of comments on that thread says:

@ktoso

Wouldn't it be better to hold off introducing such types for when we tackle the long standing issue of introducing a "good" ByteBuffer type? I think we're getting close to being able to do so with the introduction of move-only and non-escaping types. Such new type may have APIs which benefit from having a ByteOrder type then.

*Span types are "good ByteBuffer" ones, aren't they?

However, on the other hand, some concern was raised:

@johannesweiss

@Karl

Whenever I see code that asks what the native byte order is, it's almost certain the code is either wrong or misguided. [...]

I agree with @Karl here. Querying the native byte order is almost always wrong.

[scanon (Steve Canon)]

_Querying_ native byte order is often (though not always) wrong, but using the native byte order can be correct (usually when you have to interact with other software that is not aware of byte order).

[glessard (Guillaume Lessard)]

I had forgotten about @YOCKOW‘s tiny pitch. I’ll add a link to that thread in the proposal.

[YOCKOW]

@scanon

Querying native byte order is often (though not always) wrong, but using the native byte order can be correct (usually when you have to interact with other software that is not aware of byte order).

Thank you for clarifying.
Then, as an API design, func load(..., endianness: Endianness = .native) is not so odd. Am I right?
I want to echo @xwu's comments:

@xwu

Using Endianness? = nil as the convention to mean native endianness feels awkward. Since the enum is equatable, this would also mean that native endianness compares not-equal to both big and little, which doesn't seem right. Peer languages have a .native, which we could also have here as a static property.

@xwu

And on that note, an optional value would be a self-documenting and idiomatic way to model in modern Swift what CF has as a separate case, CFByteOrderUnknown. But I'm queasy about nil representing an actual know(n|able) endianness.

I agree that nil(Optional<Endianness>.none) seems to correspond to CFByteOrderUnknown.

[grynspan (Jonathan Grynspan)]

Why only integer types? Why not anything BitwiseCopyable? Are the loads aligned or unaligned? If aligned, can we also get unaligned equivalents?

[xwu (Xiaodi Wu)]

@grynspan

Why not anything BitwiseCopyable?

Future directions

A "fully inhabited" layout constraint or marker protocol

This document proposes functions to load numeric values only. The salient property they share is that no possible bit pattern loaded into them is invalid. We could try to encode this property in a "fully inhabited" layout constraint, or in a more limited way in a marker protocol. We note that loading an aggregate that contains padding bytes is not inherently unsafe. However, attempting to load a value that has invalid bit patterns is inherently unsafe. Enums generally fall into the second category. [...]

Alternatives considered

[...]

Waiting for a "fully inhabited" layout constraint

The need for the functionality in this proposal is urgent and can be achieved with standard library additions. The fully-inhabited layout constraint would also require significant compiler work, and we believe that it is not worth waiting for it. It is also unclear whether the layout constraint might is a sound approach.

Making these additions generic over FixedWidthInteger & BitwiseCopyable and BinaryFloatingPoint & BitwiseCopyable

These are not sufficient constraints, since they do not mandate that their conformers must be fully inhabited.

See also:

Possible unsoundness with RawSpan and MutableRawSpan

The RawSpan and MutableRawSpan APIs make it possible to arbitrarily change the bit pattern of a BitwiseCopyable value without using unsafe code. For example, the following function returns a Bool value with an invalid bit pattern: func createInvalidBool() -> Bool { var dst: [1 of Bool] = [false] var dstSpan = dst.mutableSpan var dstBytes = dstSpan.mutableBytes dstBytes.storeBytes(of: 2, toByteOffset: 0, as: UInt8.self) return dst[0] } I've also noticed that the RawSpan and…

[grynspan (Jonathan Grynspan)]

I missed that future direction.

[glessard (Guillaume Lessard)]

I’ve updated the document with a number of clarifications, and a rename of the Endianness enum to ByteOrder, along with the necessary changes in prose.

[fclout]

My read is that we’re kind of trying to have it both ways:

• There is no way to control endianness for anything other than integers

That seems fine; if endianness is a concern for anything other than integers, then it has become a parsing problem, and values should be reconstructed safely with initializers.

I think this pushes RawSpan into parsing territory more than being the primitive for working with untyped memory, so I’m less hot for it.

Parsing-adjacent use cases were always a major justification for RawSpan; parsing is mentioned in the first sentence introducing RawSpan in SE-0447!

I expect that more people will have Jonathan’s “why does this only work for integer types” reaction.