Why must i build ALL of NIOCore just to say the name ByteBuffer?

[taylorswift]

@lukasa

For the above reason, I don't think anyone thinks that: it's an alternative type to Data that has the baggage of NIOCore, which is exactly what promoted @taylorswift to start this thread in the first place.

i don't know why BSON uses ByteBuffer as its backing storage, only the author of BSON knows that. but i don't think ByteBuffer has the same baggage as Data, because NIOCore ultimately must be linked into the final binary product anyways (not sure if there are any good use cases for BSON that don't involve NIO), whereas Foundation is not necessary and just adds bloat to the binary and consumes extra memory.

a dependency on NIOCore is annoying, because it takes a long time to compile, but it is not a blocker. a dependency on Foundation is a blocker.

@lukasa

None, but it predates the introduction of ByteBufferView. Additionally, there are a suite of related functions that ByteBufferView cannot model, including withUnsafeMutableReadableBytes, readWithUnsafeReadableBytes & readWithUnsafeMutableReadableBytes.

@lukasa

The problem with this idea is that it makes manipulating the indices extremely painful. In particular, the "read" operation (consume some bytes and move the reader index) conceptually cannot occur on the ByteBufferView without allowing a BBV to be stored into ByteBuffer.readableBytesView, and that opens us up to awkward questions like "what happens if someone stores a BBV that was not derived from this ByteBuffer?".

i think this is analogous to assigning to Dictionary.values; the type system allows it but it really doesn't make sense to do, and doing it is considered a programming error.

@lukasa

There are four reasons we don't do that:

1. Swift does not promise that structs defined in Swift have a specific layout, so performing this unsafe bitcast is not generally safe. If the type is frozen it's safer, but still not safe as tools versions may change the layout, and we can't force that as a generic constraint anyway.

so, i think over the years, "flexible layout" has evolved into "flexible layout with swiftian characteristics". that is to say, declaration-order layout has become the facts on the ground, even though we ceremonially preface that with "the compiler has the right to reorder struct fields in any way it sees fit". otherwise there would not be a right and a wrong way to list the fields in source. and if the compiler were ever to change that behavior anything that blits buffers of things like RGBA<T>, Vector4<T>, OpenGL structures, etc. would degrade in performance to an extent that it would become a bug.

@lukasa

It forces users to understand structure padding. Imagine a 9 byte header made up of a single 8-bit flags field and a 64-bit integer. You may want to define this type as:

@frozen
struct MyBinaryHeader {
    var flags: UInt8
    var identifier: UInt64
}

The problem here is that the size of this structure is 16 bytes, not 9. Unsafebitcasting to this will both read too many bytes and throw away 7 of the header bytes.

right so my experience with binary protos is that most of them do actually respect alignment, and will try to buddy-up integers in a way that matches their natural alignment boundaries. sometimes we have awkward shapes like (UInt8, UInt64) in your example, but most of the time that is because the 8-byte integer is actually a fixed-length string or something, and we just took a shortcut making it into a UInt64 when the designer of the proto intended for it to be something like (UInt8, UInt8, UInt8, UInt8, UInt8, UInt8, UInt8, UInt8).

so the problem then becomes getting users to feel confident that

@frozen
struct MyBinaryHeader 
{
    var a:UInt16
    var b:UInt16
    var c:UInt32
}

has no interior padding, which requires them to understand the layout behavior of swift structs. but i do not think it is possible to do serious work with binary formats and not be familiar with this in the first place; alignment and padding is a pretty core concept to binary serialization.

@lukasa

There is no way to express a generic constraint that forces this to be a trivial type. Users will naturally be tempted to write a struct containing an Array, and an unsafeBitCast will therefore produce a wild pointer with a crash. Unacceptable for a safe API.

which is why i named the strawman method readTrivial so there would be a reminder every time someone were to use it. i suppose whether this is sufficient or not is up to your discretion.

@lukasa

Defining an @BigEndian property wrapper whose only effect is to change how NIO encodes something is, in my view, an abstraction violation. It pushes an implementation detail (this type is decoded from a ByteBuffer) into your API surface. I've never been a fan of this when people use it for Codable, and I don't love it here either.

i envisioned that the structs using the @BigEndian wrapper would merely be shims to quarry data out of a ByteBuffer without a performance hit. it adds work for the user, but reduces the amount of gybbed API in the library, since it's unlikely all sixteen overloads would be used in any particular deserializer.

@lukasa

Yes, but it matters when someone who thinks that we've made a dumb choice publicly rants about how dumb our choice is. If I hadn't stepped in it is highly likely you'd have been unopposed, and so folks who haven't used our type would read your post and assume that our silence implies that you are right. It becomes a form of misinformation that encourages other people to draw the same incorrect conclusion as you.

Put another way, I don't care whether you think NIO is good or not, your opinion is your own. But I do care when a) you think it for reasons that are ill-informed, and b) you write posts that encourage others to agree with you. This is why I wanted you to begin by asking questions instead of making assertions: it makes the whole thread far less combative, with the side benefit that third party observers might learn something.

oh my, i think you overestimate my influence on these forums and the community at large. i don't work at Apple (as i'm often reminded) and i'm not even part of the server group, so i certainly don't post anticipating anyone will take me very seriously. but, i understand why this is a concern for you, so i will keep that in mind in the future.

@lukasa

Yup, I agree this is a pain in the neck. I just don't think the proposed remedy is the right one. It implies that if we ever build something useful we need to factor it into its own library in case someone exposes it in their own API. We actually went through this earlier with CircularBuffer where we repeatedly received pressure to move it to a separate library, which we resisted.

Happily, with the release of swift-collections containing Deque, all pressure to make CircularBuffer generally available vanished. Deque is both a better type and available without NIO's baggage, so users have rapidly transitioned over to it. I'd like to see something similar happen with ByteBuffer.

right, i am really struggling to think of any "good" solutions to this problem in the short-medium term, because i agree with you now that this is not SwiftNIO's fault, and i don't think it's BSON's fault either because BSON is part of MongoKitten. so i think assessments like

@Avi

I don't think it's fair to hold NIO responsible for how third parties abuse their framework. There's nothing stopping those library authors from extracting the ByteBuffer code into their own library and using it internally. The problem only begins when third parties want to use it as a currency type, and that's just not what it's designed for.

The NIO team isn't responsible for others' laziness.

are accurate but not constructive because even if i had time to write a self-contained BSON library (which i do not), it wouldn't do any good because MongoKitten and MSD rely on their own framework-specific BSON implementations that do use it as a currency type.