Serialization in Swift

Evolution Discussion
15

I’m excited that this can of worms is finally being opened, thanks @tomerd!

TLDR

Given our past performance, I find it unlikely that we will come up with a singular serialization solution which addresses the full scope of the community's needs. As such, it makes sense to shift focus to providing a lower level toolkit that can both be used to solve the prominent use cases (like serialization for common formats), as well as enable folks to address the long tail of weird things that have made consensus on this topic difficult to achieve. This kernel of functionality is static structural reflection.

My Experience

I've used Codable extensively over the years, primarily for mobile applications but also a nontrivial amount of command-line tools and server applications, I also authored ResilientDecoding. I've ran into a bunch of the pain points already discussed here (like performance concerns), but often it was beneficial to eat the cost of Codable in exchange for the utility it provided. Here are a couple of the weirder use cases I've come across (a bit simplified for this discussion):

Transformations on unstructured data At a large company I used to work for, we needed to interface with a backend API that contained a fair bit of legacy cruft as well as a number of unfortunate compromises from supporting a variety of different platforms. Using `Codable` directly with this API was a pain, and instead I wrote some code that applied some general transformations on the `NSJSONSerialization` output, then instantiated a `JSONDecoder` with the transformed object. How did I accomplish this? I copied the implementation of JSONDecoder into our project, and exposed an initializer that took the unstructured data as an `Any`. It wasn't pretty, but it made the downstream usage of `Codable` significantly more ergonomic.
Abusing `Codable` for non-serialization tasks In my current role, I defined a protocol which had a `Codable` associated type and a method which took an instance of that type and performed some meaningful work. The trick here was that I needed to create an instance of the associated type in a generic way, and the logic to do that required a unique, semantic name for each of the leaf values of that type. I achieved this by creating a custom decoder which, when it reached a leaf value, passed the coding path to an initializer for that value (which had to conform to a specific protocol).

Comments

Performance

At a high level, I think the bar for performance should be "on par with generated code", meaning that if we need to write code generators to achieve good performance for things like protobuf we will have failed (as users will be likely to eschew our system and just lean on code generators). It seems entirely possible to come up with a system where the only thing we would need to generate from protobuf is a bunch of (field, offset) pairs and have the serialization infrastructure and compiler synthesize the necessary serialization logic. This will also make it much easier to, in the future, be able to import things like protobuf definitions the same way we import C code.
Much of this has been discussed upthread, but a less obvious concern is binary size. There will be a natural tension between specializing the decoding logic per-type (increasing binary size and performance) and having more general logic (decreasing binary size and performance). The compiler already has to deal with this for generics, but the magnitude of the effect will likely be much bigger for serialization, and since the logic is synthesized, this effect is often better hidden from the developer.

Ergonomics

  • Composability: Much of the infrastructure surrounding Codable is not composable. A simple example is date format selection, which is currently implemented directly on JSON{De,En}coder. A better solution would allow that logic to be modularized and independent of serialization, as well as be applied to a subsection of a serialized response. The same observation applies to snake-case-keys and other "strategies". It would also be nice if the encoding and decoding side were semantically connected (as opposed to having DateDecodingStrategy and DateEncodingStrategy).
  • Customization: Right now we can achieve heavy-weight customization by manually implementing init(from:) or encode(to:). Property wrappers provide some lighter-weight customization but are fairly limited in what they can achieve. A big limitation is that we can't pass value arguments to the encoding/decoding infrastructure; in theory something like @Resilient(defaultValue: 5) var value: Int should be possible (You can currently write this, but there is no way to access the value 5 during decoding). Another avenue for adding customization is adding keypaths to decoding, which would enable something like protocol Resilient { static func defaultValue<T>(for keyPath: KeyPath<Self, T>) -> T }

Static Structural Reflection

At the top of this post, I mentioned "Static Structural Reflection" as the kernel of generic Codable functionality with which we can solve the big serialization use cases, as well as the long tail of interesting things the community might need. Automatic Requirement Satisfaction in plain Swift discusses this, though I share some concerns about its type-system-oriented approach (something closer to how function builders work might be preferable). Such a system would need to have the following capabilities:

  • Be able to express structural constraints on a type (for instance, all leaf properties must conform to the Decodable protocol). This could be implicit from the code doing the structural reflecting. For instance, if we ended up with a system similar to function builders, we could mandate that an overload exists for a buildExpression-style method.
  • Given a type, we should be able to access a collection of stored properties, along with their name, key path and value type; we should also be able to look this information up for a specific property (for instance, by name).
  • Use the information in the previous point to create an instance of that type from a set of values
  • As a bonus, such a system could replicate much of the functionality of Mirror with less runtime overhead and more static checks.
13 Likes