I think there actually should be a general-purpose feature for parsing an integer from bytes, including choice of byte order and support for arbitrary BinaryInteger types. It's a common enough operation that it should be supported in the standard library, even if advanced deserialization isn't.
We can look at String for comparison. The String type doesn't have facilities for advanced parsing, but we still have the ability to parse an integer from a string using the FixedWidthInteger.init?(_: String) initializer. There's even a way to specify an arbitrary radix.
We can also look at Python for prior art. In Python, advanced deserialization is delegated to the struct module in the standard library. But parsing an integer from bytes is common enough that the built-in int type has a from_bytes static method.
Given that floating-point numbers have a standardized binary representation, I think it would also be useful to support general-purpose deserialization for those as well, including choice of byte order and support for arbitrary BinaryFloatingPoint types, including Float80.
I sympathize with the argument that RawSpan should be limited to manipulating raw memory, not deserialization, especially because it's ambiguous at present whether a RawSpan should be guaranteed to point to fully-initialized memory (if so, it would currently be unsound). I think it's okay if RawSpan ends up being limited to lower-level unsafe operations over memory that may be partially uninitialized. But abstractions over fully-initialized binary data, with safe operations like the ones proposed, would still be useful even if RawSpan doesn't end up filling that role. For prior art, we could look at Python's bytes, bytearray, and memoryview types, and JavaScript's ArrayBuffer, TypedArray, and DataView types.