How can I encode a struct to Data/binary?

I know Swift has the Codable interface that allows a type be encoded or decoded. However, I only know of two encoders, namely JSONEncoder and PropertyListEncoder. What do I do if all I want to binary (aka Data) encode?

NSKeyedArchiver is you can stomach it. (forget it, it won't work with structs).

If you need high performance encoding and decoding, I personally don't think Codable is a good fit. It doesn't support streaming, for example, and was built primarily with high-level serialization formats in mind, specifically JSON. Although, you'll gain an ability to switch to different encoders and decoders later if needed, if you build on top of Codable.

The most comprehensive set of functions that work with binary data are in Swift NIO's ByteBuffer. Those handle endianness, bit width, and alignment correctly for you on all platforms, and I recommend relying on it (or something similar) instead of using withUnsafeBytes directly on your own.

2 Likes

I haven't used it in earnest beyond playing around with it in a playground, but Paul Kraft's "DataKit" looks very interesting and might be useful for your purposes:

It's great to see more packages appearing that address the need for binary serialization!

I see that it uses result builders, and I'd like to warn that for more complex model types I found that to be a major disadvantage when debugging and maintaining the serialization code.

One can't simply put a breakpoint at a result builder element to examine serialization input or output, which is the consequence of how result builders work. Result builders are usually evaluated at a different point in time than your serialization code, and they express logic at a certain level of indirection, which makes it hard to debug state at serialization time.

IMO macros are a much better fit for this, but I'm not aware of libraries exploring that space yet.

1 Like

BTW, is there a good and concise example of making a custom Encoder / Decoder?

I can quickly cook a very simple one that encodes, say, an Int or a String, but struggle with a more complex one that encodes a struct of multiple fields.


Edit: found this one from @Lantua but it's probably good to be used "as is" rather than an educational source how to make one. Will try to carve it down to a minimal educational example later on.

1 Like

XMLCoder is one example, but after maintaining it in the past, I can say that Codable is suitable mainly for serialization formats that are strictly tree-like. XML is not one of those with its choice of coding as an attribute or an element, which makes things quite complicated and slow as a consequence, a lot of branching for these coding decisions has to be done dynamically at run time. If I were developing XMLCoder from scratch for Swift 5.9 I'd go with macros, skipping the current implementation of Codable altogether.

This could also be the case for the binary format that the OP would be using, same warning applies.

2 Likes

I have used BSON with Codable as serialization method for binary data before, and if I need to do something along the line again I might go with ProtoBuf depending on the use case.

Some years ago Mattt Thomson did a series of books about advanced topics in Swift called Flight School.

The first one was about codable and in the final chapter he made a custom MessagePack encoder.

The book is free now so maybe that can help you get an idea of what it takes to write a custom Encoder/Decoder :smile:

2 Likes

PropertyListEncoder can save binary data (as well as XML) and works well with structs. You need to do extra work for classes if you also use inheritance, but it's not so difficult. I really like Codable.

1 Like

JSONEncoder returns Data... :wink:

On a serious note, are you after encoding speed, decoding speed, encoded size, the desired "obfuscated" encoded representation, round trip (bit-for-bit) fidelity, key order stability, all or some of these, or something else altogether?


For example JSONEncoder can't roundtrip float values bit for bit:

let x = Float(-0.0)
let data = try! JSONEncoder().encode(x)
print(String(data: data, encoding: .ascii)!) // -0
let y = try! JSONDecoder().decode(Float.self, from: data)
print(x == y ? "x == y" : "x != y") // x == y
print(x.bitPattern == y.bitPattern ? "x.bitPattern == y.bitPattern" : "x.bitPattern != y.bitPattern") // x.bitPattern != y.bitPattern
print(String(x.bitPattern, radix: 2)) // 10000000000000000000000000000000
print(String(y.bitPattern, radix: 2)) // 0

Ditto (actually worse) for NaN, Inf and subnormals.

So if bit for bit roundtrip fidelity is important you'd need to choose among coders that preserve it.

1 Like

I was looking for something like this a year ago, Thanks for the link

2 Likes

Thanks! This was what I was looking for:

let encoder = PropertyListEncoder()
encoder.outputFormat = .binary

I wanted to send objects via the network and deserialize them at the opposite path.

Ah, ok, and when I decode, it decodes back to the actual type. Lol! My entire life was a lie! I was operating under the impression it encodes to string or something.

I quite liked Routing with Codable – Encoding - Swift Talk - objc.io

I used it as foot hold to work on my own which is very much not heading towards being concise: