The future of serialization & deserialization APIs

Hello Swift Community,

I’m happy to announce that I've been hard at work planning a potential future for serialization & deserialization APIs in Swift. It's clear from community adoption and feedback that Codable has had a lot of success in the years since it was added to Swift 4, but that it doesn’t satisfy some important needs. One of the foremost of those needs is performance more in line with programming environments that compete with Swift. As such, the main goal for this effort is to unlock higher levels of performance during both serialization and deserialization without sacrificing the ease of use that Codable provides.

This is a large project with a lot of moving parts and I'm not quite ready for an official Pitch yet. However, I want to collect the community’s initial thoughts and reactions to the direction we’re taking so far to make sure that we’re successfully addressing the right set of problems.

Here are the core tenets of the effort:

A new API is required to escape the implicit Codable performance ceiling

Even with all of its strengths, the existing API’s design has some unavoidable performance penalties. For instance, its use of existentials implies additional runtime and memory costs as existential values are boxed, unboxed, retained, released, and dynamic dispatch is performed.

Also, because a client can decode dictionary values in arbitrary orders, a KeyedDecodingContainer is effectively required to proactively parse the payload into some kind of intermediate representation, necessitating allocations for internal temporary dictionaries, and String values. In fact, during JSONDecoder optimization work, I discovered that ALL containers need to do this because some Decodable types retain the decoder or one of its containers after init(from:Decoder) returns to perform deferred decoding—which was not an intended usage of the interface.

Dynamic casting is prevalent in both Property List and JSON encoders and decoders. They have special support for Foundation types like Data, Date, and others that are not members of the core set of types Codable supports at the standard library level. These dynamic casts are unavoidable with the existing design and have a measurable impact on performance.

Because of these and other performance penalties that are inherent to the existing APIs, the best path forward is to design a new API that avoids them as much as possible.

Rust’s Serde feature is an attractive design to imitate for performance

The design of Rust’s Serde neatly avoids many of the issues described above, which helps it achieve its impressive levels of performance.

For instance, nowhere in its design does it use dyn trait types, which are the closest parallel to Swift’s any existentials. Most notably, Serde’s deserialization design employs a Visitor pattern which allows the parser to drive the deserialization process instead of being required to service requests from the client in arbitrary orders. This design can be more easily optimized for performance. For instance, when deserializing the contents of an encoded dictionary, the deserializer vends keys and values to the client in the order they occur in the payload. This allows the deserializer to elide building an intermediate representation of the dictionary in temporary memory. These are features that we can imitate in a Swift-friendly design quite easily.

Serde’s visitor pattern and deep use of lifetime annotations also enables incredible opportunities for borrowing data from the payload instead of copying. While Swift’s ~Escapable and lifetime dependency features are still in their infancy, there are still some opportunities to use them for significant performance boosts, like pattern matching dictionary keys. I hope that in due time we’ll be able to leverage those features in this design to do more borrowing and less copying like Rust allows.

Deep macro reliance will limit the need for manual implementations

Rust Serde’s deserialization Visitor pattern, mentioned above, requires more verbose boilerplate code than Swift’s Decodable. For instance, consider a simple Swift Decodable type's init(from:) implementation that is a total of three lines:

struct Person: Decodable {
    let name: String
    let age: Int
    
    init(from decoder: any Decoder) throws {
        let container: KeyedDecodingContainer<CodingKeys> = try decoder.container(keyedBy: CodingKeys.self)
        self.name = try container.decode(String.self, forKey: .name)
        self.age = try container.decode(Int.self, forKey: .age)
    }
}

Contrast that with a minimal version of the same type that uses a hypothetical Visitor pattern which grows to 12 lines (asymptotically a factor of 3x as many lines):

   func visit(_ decoder: inout some StructDecoder) throws -> Person {
       var name_tmp: String?
       var age_tmp: Int?
       while let key = decoder.nextKey() {
           switch key {
           case "name": name_tmp = decoder.decodeValue(String.self)
           case "age": age_tmp = decoder.decodeValue(Int.self)
           default: break
           }
       }
       guard let name = name_tmp else { /* throw an error */ }
       guard let age = age_tmp else { /* throw an error */ }
       return Person(name: name, age: age)
   }

Serde provides Serialize and Deserialize derive macros to generate this (de)serialization code directly from custom type definitions. This isn’t really a new concept to those familiar with Swift's automatic synthesis of Codable types.

The advantage of Serde’s macro design is that it enables much more comprehensive customizations than Codable synthesis does, allowing it to be leveraged more often and requiring fewer custom implementations.

In Swift, when a client needs to do more than just alter the default CodingKey representations, developers are often faced with a large cliff where they’re forced to manually replicate the whole Codable implementation just to do so. This situation is somewhat ameliorated by property wrappers, but the kind of customizations those can achieve is very limited. Xcode is now capable of emitting the synthesized Codable implementation directly into source files which eliminates the initial manual implementation work, but it must still be manually maintained as code evolves.

In this new design I aim to leverage Swift’s macro features to meet or exceed Serde’s level of support for customization of synthesized conformances. Moving code synthesis from the compiler to a macro will enable us to use attribute-like macros as targeted customization mechanisms, which was not something we could easily accomplish with the compiler-based Codable synthesis. The compiler implementation is also frankly difficult to evolve. A macro, by its very nature, will be much easier to enhance as needed over time.

Dual approach: format-agnostic and format-specialized protocols

One of the advantages of both Serde's and Codable's design is some level of format-agnosticism. In other words, you can implement Serialize or Encodable for a type and expect to get a valid encoding regardless of what Serializer / Encoder you use.

The need for format-agnostic encoding interfaces is incontrovertible, but it does present some problems. One these problems that Serde doesn’t solve is support for types that don’t fit neatly into its data model, which consists of strings, numbers, byte sequences, arrays, dictionaries, etc. Some encoded types can’t cleanly translate to these types at all, or they may have native and more optimal representations in a given serialization format.

One example that is particularly close to home is date values in Property Lists. Property List has native support for date—either as a <date> tag in XML plist, or a dedicated type specifier in binary plist. This is how Foundation.Date values are expected to be encoded in property lists. By contrast, Rust Date-like types typically encode themselves in floating point or string representations, as those are the most appropriate types available in the Serde data model. It is technically possible to convince a Property List Serde serializer to encode a native date value, but it requires coupling the date type’s implementation to that of the serializer’s, which is antagonistic to Serde’s intention of Serialize trait types being format-agnostic.

Actually, a similar problem exists with Codable. There is no encode(_: Date) function present in the Encoder interface, which means PropertyListEncoder has to attempt to dynamically cast every some Encodable type it receives to Date in order to handle these natively. This helps keep the Encodable type format-agnostic, but it has a negative impact on performance, even if you never actually encode any Dates.

I believe that fully and formally embracing format-specialization where appropriate is the best solution to this problem. Specifically, we should encourage each serialization format that has native support for data types that aren't represented in the format-agnostic interface to produce its own protocol variant that includes explicit support for these types, e.g. JSONCodable or PropertyListCodable. These format-specialized protocols are expected to be entirely distinct from the format-agnostic one, but they should share the same basic structure and patterns. For maximum compatibility, encoders for a specific format are also expected to support types that only conform to the format-agnostic protocol.

This dual-protocol approach should simultaneously enable broad compatibility for simple types that have no knowledge of what format they’ll be encoded in, as well as high performance and ideal encoded representations when a client knows they are encoding their types into a specific serialization format.

Unfortunately, each format and specialized protocol will need to provide its own main macro, which is a lot of work. However, in due time I hope to facilitate implementing these macros by virtue of a package that contains the core pieces of such an implementation that will simplify this work.

Compatibility with Codable

While I anticipate the new APIs to largely supplant Codable, we can’t neglect its legacy by expecting everyone to move over to new system right away. To this end, we encourage all encoders and decoders to not only accept types conforming only to the new format-agnostic protocols, but also Encodable and Decodable types, if possible.

Implementing this support may seem daunting, but for self-describing serialization formats, I am developing generic Encoder, Decoder, and Container types that operate on format-specific primitive values, e.g. JSONPrimitive or PropertyListPrimitive. Thus supporting Codable becomes as simple as implementing functions that convert between these primitive values and Codable’s data model.

Non-goals

  • While we should support as many different serialization formats as possible—even more than original Codable, ideally—there will inevitably be some that don't fit this particular model. We should focus the design around the formats that are most common to the Swift ecosystem.
  • The end result for this design should keep all definitions in code. There are many popular serialization formats (e.g. Protobuf, Flatbuffers, etc.) that use auxiliary definition or schema files, external code generator tools, and third party libraries. Regardless, for this effort, the only tool that should be required to adopt is the swift compiler, and the only dependencies should be the standard library, and any library that defines a format-specialized protocol.
  • This design does not include support for encoding and decoding cyclical objects graphs. Relatedly, there's still no intention to include encoding of runtime type information in serialization formats for any purpose—all concrete types must be specified by the client doing the encoding or decoding.

Example

To help roughly illustrate our vision for these APIs, here's a small example of a macro-annotated type, along with what it would expand to. (Please recognize any names and designs are placeholders, and that macro implementations are still theoretical at this stage.)

// Written code:
@JSONCodable
struct BlogPost {
    let title: String
    let subtitle: String?
    @CodingKey("date_published") @CodingFormat(.iso8601)
    let publishDate: Date
    let body: String
    @CodingDefault([])
    let tags: [String]
}
// Synthesized code:
extension BlogPost {
    enum CodingFields: Int {
        case title
        case subtitle
        case publishDate
        case body
        case tags
        case unknown
    }
}
extension BlogPost.CodingFields: DecodingField {
    static func field(for key: UTF8Span) throws -> Self {
        switch key {
        case "title": .title
        case "subtitle": .subtitle
        case "date_published": .publishDate
        case "body": .body
        case "tags": .tags
        default: .unknown
        }
    }
}
extension BlogPost.CodingFields: EncodingField {
    var key: String {
        switch key {
        case .title: "title"
        case .subtitle: "subtitle"
        case .publishDate: "date_published"
        case .body: "body"
        case .tags: "tags"
        case .unknown: fatalError("Cannot encode unknown field")
        }
    }
}
extension BlogPost: JSONEncodable {
    func encode(to encoder: inout JSONEncoder2) throws -> Person {
        try encoder.encodeStructFields() { fieldEncoder in
            try fieldEncoder.encode(field: CodingFields.title, value: self.title)
            try fieldEncoder.encode(field: CodingFields.subtitle, value: self.subtitle)
            let publishDateValue = self.publishDate.formatted(.iso8601)
            try fieldEncoder.encode(field: CodingFields.publishDate, value: publishDateValue)
            try fieldEncoder.encode(field: CodingFields.body, value: self.body)
            let tagsValue = self.tags ?? []
            try fieldEncoder.encode(field: CodingFields.tags, value: tagsValue)
        }
    }
}
extension BlogPost: JSONDecodable {
    static func decode(from decoder: inout JSONDecoder2) throws -> Person {
        try decoder.decodeWithStructHint(visitor: Visitor())
    }
    
    struct Visitor: JSONDecodingStructVisitor {
        typealias DecodedValue = BlogPost
        func visit(decoder: inout JSONDecoder2.StructDecoder) throws -> BlogPost {
            var title_tmp: String?
            var subtitle_tmp: String?
            var publishDate_tmp: Date?
            var body_tmp: String?
            var tags_tmp: [String]?
            while let field = try decoder.nextField(CodingFields.self) {
                switch field {
                case .title: title_tmp = try decoder.decodeValue(String.self)
                case .subtitle: subtitle_tmp = try decoder.decodeValue(String.self)
                case .publishDate:
                    let formatted = try decoder.decodeValue(String.self)
                    publishDate_tmp = try Date.ISO8601FormatStyle().parse(formatted)
                case .body: body_tmp = try decoder.decodeValue(String.self)
                case .tags: tags_tmp = try decoder.decodeValue([String].self)
                case .unknown: try decoder.skipValue()
            }
            guard let title = title_tmp else { throw <missing required field error> }
            let subtitle = subtitle_tmp
            guard let publishDate = publishDate_tmp else { throw <missing required field error> }
            guard let body = body_tmp else { throw <missing required field error> }
            let tags = tags_tmp ?? []
            return BlogPost(title, subtitle, publishDate, body, tags)
        }
    }
}

Conclusion

Again, I look forward to incorporating feedback of the broader Swift community to help in designing a highly performant serialization system that is easy to use and customize and meets the feature needs of more developers than ever.

The next concrete steps will include an official Pitch for the stdlib APIs and swift-foundation evolution proposals for JSON and PropertyList protocols & encoders/decoders. Proposals for macro definitions will follow later.

54 Likes

One immediate reaction is that there should be a single @Coding macro, if at all possible, that takes the relevant parameters. @Coding(key: "date_published", format: .iso8601, default: .now) or something similar. I'd also hope the macros can keep the typed keys from Codable so we can easily reuse them.

12 Likes

This code sample almost brought me to tears. In a good way.

3 Likes

Swift Testing needs to encode and decode JSON, but also needs to not link Foundation. Will this new API be part of Foundation or be elsewhere in the stack?

2 Likes

What would CodingFormat(.iso8601) mean, if anything, in a property list encoder? Or more broadly, how does it interact with types that have a specialized encoding for a given encoder?

On a similar note, I have to wonder how these APIs compose together if I want the same type to support both JSON and XML serialization?

3 Likes

It would be great if this could follow Subprocess and be a separate package.

As I think through what this means for TOMLDecoder, I'm both dreadful for the new implementation challenges , and excited for the performance potential!

Re-stating some of the points from OP that resonates with me: having to work with the keyed/unkeyed container model is a huge performance hit; especially when one have to store everything as Any. In contrast, TOML libraries that don't have this API constraint can lightly lex through the input during de-serialization, separately record entries (in list or table) by type of the value, and defer most of the work of converting the raw input for value to actual typed instances until retrieval time. This is a way more efficient scheme. I'm happy to see this new design enabling that in Swift.

1 Like

If we could adopt visitor pattern, would async-decoding be possible? (I'm imagining fetching a large XML file via HTTP. We might want to disconnect if an invalid format is detected.)

One more bike-shedding: Do we want to rename Codable to e.g. @Serializable taking this opportunity? [1]


  1. Typealias Stridable = Strideable - #5 by dabrahams ↩︎

2 Likes

One of the big flaws of Codable is that it was built on the wrong abstraction. 99.9% of the time, developers who are interested in serializing a struct to data and back are doing so to a single, well-known format. However, the Codable API was built so that the abstraction point is the encoder itself, under the assumption that you would want to serialize a type to multiple formats. This is not the case.

That design flaw has been the #1 source of Codable's woes. It makes properly implementing custom coders almost impossible; no one implements superEncoder properly, since most people don't deal with inheritance of reference types, and some formats are fundamentally incompatible with the Encoder/Decoder APIs. (XML and CSV are two that spring to mind off the top of my head)

There are two main kinds of serialization that types need to do:

  1. Serialization to-and-from a specific format
  2. Serialization to-and-from an opaque format

Codable tries to do both and, from what I can tell, this proposal is aiming to replace it with something that also tries and does both. I think that's a mistake. IMO we should be encouraging packages that provide format-specific coders (JSONCodable, PlistCodable, CSVCodable, XMLCodable, etc) so that each encoder and decoder can provide format-specific functionality. Then we should provide a system level API to ask types to encode into an opaque format (ie "please turn yourself into a Data and back again").

So I'm very interested in the @JSONCodable macros provided in the example... but those should be part of a JSON-specific package. And if I truly want a single struct that can be encoded to JSON and XML and ProtoBuf and CSV, then I should have to indicate that separately on the struct by conforming to each of those separate protocols.


Edit: or to put it more plainly... Foundation should provide an updated replacement for NSCoding and leave the type-specific encoders to type-specific packages to implement.

14 Likes

This is a good point, this system should be designed so the core requirements don't preclude streaming parsing. Most users won't need to stream but when you do, you really need it.

3 Likes

I agree, thanks for pointing out this was tagged foundation. It definitely seems like such a system:

  • Should NOT be part of Foundation.
  • Should at least be developed as a separate package, regardless of where it lives in a stable format. This would allow for iteration on design and thorough performance testing (using swift-benchmark) before 1.0.
  • Should focus on enabling core functionality like performant serialization so it's easy for third parties to write their own encoders and decoders. This package shouldn't reach 1.0 until that goal is demonstrated by outside developers.
  • Should design for composability, or at least find some way so the multiformat goal is achievable, if not necessarily automatic.
  • Dave's point about an opaque format is interesting. A high performance local serialization format might be valuable for those who don't need external readers.
10 Likes

I think there's one common use case which is not covered by the current Codable design: heterogeneous/dynamic decoding/encoding.

Many times in my developing, I wanted to decode part of a json into an intermediate representation, and later further decode that thing into a specific type.

struct Outer: Decodable {
    let value: Int
    let partialData: PartialJSONValue  // PartialJSONValue is a placeholder for the heterogeneous json value type I need
}

let fullObject = try SomeDecoder().decode(Outer.self, from: someData)
if someDynamicCondition {
    let a = SomeDecoder().decode(A.self, from: fullObject.partialData)
} else {
    let b = SomeDecoder().decode(B.self, from: fullObject.partialData)
}

Because the current Codable mechanism lacks ways to express PartialJSONValue, I often resorted to something like [String: Any], which has its own fatal flaws (such as implementation difficulties and performance problems).

3 Likes

I think this was by design… that Coding was meant for well-formed data. NSJSONSerialization and NSPropertyListSerialization were the more correct choice for data that was not well formed.

I would love for this vision to support some form of opt-in async serialization that can a) decrease the memory footprint when very large objects are serialized/deserialized over the wire (potentially with back pressure!), and b) decrease latency for consuming the first parts of a message.

Both of these are incredibly important for both server and client use cases that I don’t think a “version 2” of Codable can reasonably ignore, at least in its design vision.

I would assume unkeyed structures can support this out of the box as an AsyncSequence, but keyed structures with optional members could potentially as well, for instance an actor could provide observable members that would send out updates as its members are received.

2 Likes

I would also love it if Codable came with a format-agnostic type for collecting and re-serializing arbitrary trees. If I can throw out GitHub - mochidev/DynamicCodable: Easily preserve arbitrary Codable data structures when unarchiving so I never need to worry about preserving something like metadata fields, I would in a heartbeat :heart:

I don't totally agree that Codable is an effort to serialise a type to multiple formats. Ideally, looking at Codable without any custom codable functions, it is really an effort to serialise to and from a single format: key-value coded Swift structs.

The problem with Codable – and what I think you're getting at when you suggest we need JSONCodable/PlistCodable – is there's no sane custom implementation of init(from:) and encode(to:) without being archive-specific. These functions are generally a mashup of two different ideas:

  1. migration and versioning
  2. archive-specific choices like which fields to include and what order

But moreover, while you might make archive-specific choices, you don't always have archive-specific knowledge. Take for example, trying to decode a value from an archive that might include any common "property tree" value:

public init(from decoder: Decoder) throws {
    let container = try decoder.singleValueContainer()
    if container.decodeNil() {
        self = .null
    } else if let stringValue = try? container.decode(String.self) {
        self = .string(stringValue)
    } else if let intValue = try? container.decode(Int.self) {
        self = .integer(intValue)
    } else if let doubleValue = try? container.decode(Double.self) {
        self = .double(doubleValue)
    } else if let boolValue = try? container.decode(Bool.self) {
        self = .boolean(boolValue)
    } else if let array = try? container.decode([DatabaseLiteralValue].self) {
        self = .array(try JSONEncoder().encode(array))
    } else {
        self = .object(try JSONEncoder().encode(container.decode([String: DatabaseLiteralValue].self)))
    }
}

We have no lookahead. We can't peek to see if the next char is a double-quote, a digit or a bracket. Without overloading the Decoder to emit lookahead metadata as decodable types, you simply need to try each possibility, in turn, incurring the overhead and disruption of thrown errors.

I'd hope the visitor pattern here would be able to come back and say: I've already decoded the Bool type and just eliminate this need for archive-specific knowledge and ad-hoc parsing within the Decoding implementation.

1 Like

Great to see this progress so early, thank for sharing it! I have a couple of questions about somewhat niche Codable patterns, which I wonder if they could be easily supported in this new library. If the answer is "not really" I don't think that's a problem, as these really are slightly oddball, but I would be personally delighted if they turn out to be easier in this paradigm than with Codable.

  1. Let's call this one "coding mixins". This is where an API documents something like "all objects served via this API have the metadata properties @@type, @@url, @@last_changed_datetime, and @@id." With Codable and knowing I was decoding JSON I could do a quite dubious generic-composition dance to put the decoding rules for these properties in just one place. It seems like that wouldn't be possible under this new proposal because the container controls the order keys are delivered in. The best alternative I saw myself was a protocol requiring that all my API-model types have these properties, but I think I would still need to manually annotate each model type with the coding details (the non-standard keys and the encoding of the date field). Is there some better alternative (even if only sketchy, or in prior art elsewhere) that I'm missing?
  2. My other niche interest is format-shifting. Suppose I want to reuse the same model but encode/decode it into multiple formats (speculative use cases: json/plist, with the different date representations implied; different API versions that change the encoding spec but preserve the semantics as in "we now use camelCase for all properties"; a rich representation for snapshot-testing and a stripped representation for public API traffic). Since the serialisation control lives as macros (@CodingDefault and @CodingKey and so on, whatever their eventual spelling turns out to be) on the model-type's properties, it seems like I would need a separate model type for each serialisation format. (This is possible, but keeping those types in sync is harder than I would prefer if they include optionals or properties with default values.) But I'm wondering about the possibility to decouple two distinct types: (a) the Visitor.DecodedValue type which is produced by decoding, but also (b) a "coding specification" type which provides the keys, formats, defaults, etc. In the normal case these are the same type, but the format-shifting use case would be helped if it were possible to separate them (even if that meant forgoing some of the concise representation), e.g.
@JSONCodableExplicit(decodedValue=BlogPost.self)
struct BlogPostJSONCoding {
    let title: String // the macro would complain if these properties don't match BlogPost
    let subtitle: String?
    @CodingKey("date_published") @CodingFormat(.iso8601)
    let publishDate: Date
    // etc
}

@PListCodableExplicit(decodedValue=BlogPost.self)
struct BlogPostJSONCoding { ... }

Or alternatively, perhaps an optional specifier in the @CodingXYZ family of macros allowing to stack multiple copies to be used by different en/decoders:

@CodingKey("date_published", for: .json)
@CodingKey("datePublished", for: .thatAPIVersion2) // I would need to be able to name my own refinements for ad-hoc cases like this
@CodingFormat(.iso8601, for: .json)
@CodingFormat(.native, for: .plist)
let publishDate: Date

Heterogenous does not imply malformed.

JSON is derived from JavaScript - a highly polymorphic and duck-typed language. Many well-formed JSON APIs carry this assumption with them, and as such are hard to work with in Swift because they have objects that can have one of several (well-defined) layouts, and properties that that can be, for example, either a string or number.

Such things can be represented in Swift using protocols, but there is no support in Codable for decoding a value that could be "any type that conforms to this protocol".

Instead we must provide an enum wrapper for all supported types, which would be fine in most cases except that the default encoding for enums with heterogeneous payload values does not align with the format anyone designing a typical JSON API would use, so this requires a tonne of manual boilerplate.

It's also common to have plug-in architectures, where people can extend an API with new types that may also need to be serialized by a core library that doesn't know about them, and this can again be represented easily in Swift using protocols, but isn't supported by Codable at all (except via private runtime hacks such as the one employed by SwiftUI's NavigationPath).

10 Likes

I was really disappointed to see this, because these are probably my two major pain points with Codable.

If we are going to the trouble of making a brand new, backwards-incompatible replacement for Codable then it should try to correct all the major deficiencies of the existing design, not just performance.

NSCoding (for all its faults) supports both or heterogeneous data and cyclical references. If this new system doesn't support those then we are saying from the outset that it is still isn't going to be capable of dealing with a lot of real-world use-cases.

I understand the security implications of including type information in serialized data, but it's a super-common real-world use-case and if it's not a built-in feature then it should at the very least have first-class support for people who want to create their own such systems without starting from scratch.

Two very common patterns that I'd like to see supported out of the box are:

@JSONCodable
enum PolymorphicPrimitive {
case string(String)
case int(String)
}

and:

@JSONCodable
protocol PolymorphicObject {
@CodableTypeKey
let type: String
...
}

If CodableTypeKey needs to be an enum, or accept a fixed array of types as a parameter then so be it (although it would ideally allow any type that conforms to the protocol to support runtime registration new types via plugins, etc).

7 Likes