I'm writing a custom binary Coder implementation for high-performance XPC serialization (it's very rough and actively being worked on, but I put it online if anyone is curious: GitHub - saagarjha/Cod: A binary Swift Coder implementation), so this discussion is extremely relevant to me. Because of my use case, my focus will mostly be on writing custom Coders that perform well–but in generally I agree with the points brought up above about the need for better key transformations, handle versioning and decoding failures, and other ways to deal with the realities of parsing data that isn't ideal. (One thing do want to address is the "my decoder can't access my default value" complaint: personally, I feel this is a problem waiting for const generics, not better serialization. In my mind the ideal way to express these is
@Defaultable<5>
var foo: Int
which would allow a decoder to pull this value out at decoding time since it's part of the type itself. In the meantime I've been doing the janky workaround for a lack of const generics, which is to lift values into the type system with a bit of boilerplate:
struct Five {
let value = 5
}
@Defaultable<Five>
var foo: Int
)
Anyways, on to actually implementing a coder: my opinion is that this is possible to do, but it is really lacking in documentation and gets progressively more challenging the further you get away from a JSON-type encoding. I think the design makes it technically possible to do pretty much anything, but at a certain point you're keeping around a god-object containing context for the entire encoding tree and just satisfying protocol requirements because you have to, not because it matches the design of your serializer. (Why would you continue to use Codable when you're clearly no longer on board with how it wants you to do serialization? Because it's the only way to get access to compiler-generated reflection you need for serialization. But that's going into "we need hygienic macros" and I don't want to digress too far.) @dwaite mentions that pretty much every implementation requires implementers to hand-roll a vtable in the generic encode<T: Encodable>(_: T) function; it is really disappointing that we have to do this because the compiler cannot help us write this particular switch statement. The extra functions should either be trimmed for not pulling their weight, or they should be rethought so that it is possible to statically dispatch to them automatically.
Performance wise: @lukasa is spot-on. Codable, as it is currently designed, cannot be performant. I can get actual numbers once I add some more optimizations in my implementation, but extrapolating from my measurements the performance ceiling is capped at a couple hundred instructions per byte, which is orders of magnitude slower than fast serialization implementations. Heap allocations are basically required to happen everywhere by design, and there is way too much "are you an x" going on that requires walking through really slow runtime type checks.
Unkeyed containers are critical for performance (at least, they are for me, since they back the big arrays and data) but they really are not designed for that at all. A good binary encoding can encode an array efficiently both in space and in time–copying out bytes (maybe with a quick application of variable length encoding or endianness fix) should be really optimized. With UnkeyedEncodingContainer it just can't be: you get a bunch of possibly-heterogeneous data piecemeal rather than all at once. Even if you specialize your format by detecting this common case (which is not ergonomic at all, mind you) the "here is one element → check the type of this element → store the data and its type information in some internal context so you can write it out later" cycle is pretty much the opposite of efficient. And if you happen to get strings or a structure that has variable size (has an array member, etc.) every element is going to be variable sized, so you have to throw away your optimizations halfway through and generate an index table of byte offsets to handle this.
Likewise, unkeyed containers have some fat that could be trimmed as well. The type system generally has an idea of the shape of data; it's not going to be missing random fields or have nulls in inconvenient places (the Decoder might, but coming from Swift everything should be well-formed). But the Encoder interface has no way of representing "this is a nice, complete type that I can hand to you together", which means that an implementation needs to keep track of this itself (and again, piecemeal as members arrive). Like, consider this struct:
struct Foo: Codable {
let bar: Int
let baz: Double
}
A KeyedEncodingContainer needs to have logic in its encoding method to keep a list of all the types it encounters during encoding, and for this one it will go "ok in my encoding functions I noticed that I was passed a Int and Double, I've seen that combination before (but I don't actually know what it is) so I don't need to write out a record for it". The other inefficiency is that the autogenerated CodingKey are Strings, not integers, and these can be a substantial overhead in the size the data you produce since you need to store at least some data describing the keys (it is unclear if keyed containers can rely on ordering being consistent?) I'm probably going to migrate to some sort of compressed prefix trie thing, but having access to integer keys for performance would have been significantly nicer.
Another strange quirk is that Codable is clearly meant to be fairly opaque to end users, who are meant to just use the APIs without looking under the hood at how it works, but if you do that you can accidentally leave significant performance on the table. For example, encoding a [[Int]] usually requires ever single byte to be looked at twice, because the array is going to encode each member individually (arrays go through the unkeyed coder), then combine them together and encode the whole thing another time (again, each byte is going through the unkeyed coder). You'd think the encoder would be "smart" and just see "oh, this is nested, I can encode it and pass it through without looking at it again" but most coders will not specialize this and it's not clear whether they even should (especially for types they don't own). So you can hit random performance cliffs if you aren't an expert in how the implementation actually works, but pretty much nobody is.
Anyways, this is getting a bit long and rambly for an initial post, so I'll stop here, but hopefully this this helps provide more details about what parts of the API seem to be slow or difficult to work with from my side.
) but it's primary failing is that it just isn't enough. So it needs more. More what? More everything!