Add support for Encoding and Decoding nested JSON keys

Introduction

Today, decoding JSON using JSONDecoder with a synthesized Codable implemenation requires that your object graph has a one-to-one mapping to the object graph of the source JSON. This decreases the control that authors have over their Codable models, and can require the creation of unnecessary boilerplate objects.

I propose that we add support for Encoding and Decoding nested JSON keys using dot notation.

A previous Swift-evolution thread on this topic: Support nested custom CodingKeys for Codable types

Motivation

Application authors typically have little to no control over the structure of the JSON payloads they receive. It is often desirable to rename or reorganize fields of the payload at the time of deocoding.

Here is a theoretical JSON payload representing a Swift Evolution proposal (SE-0274):

{
    "id": "SE-0274",
    "title": "Concise magic file names",
    "metadata": {
        "review_start_date": "2020-01-08T00:00:00Z",
        "review_end_date": "2020-01-16T00:00:00Z"
    }
}

The consumer of this object may desire to hoist fields from the metadata object to the root level:

struct EvolutionProposal: Codable {
    var id: String
    var title: String
    var reviewStartDate: Date
    var reviewEndDate: Date
}

Today, this would require writing a custom encoding and decoding implementation.

Proposed solution

I propose that we add support for Encoding and Decoding nested JSON keys using dot notation:

struct EvolutionProposal: Codable {
    var id: String
    var title: String
    var reviewStartDate: Date
    var reviewEndDate: Date
    
    enum CodingKeys: String, CodingKey {
        case id
        case title
        case reviewStartDate = "metadata.reviewStartDate"
        case reviewEndDate = "metadata.reviewEndDate"
    }
}

Prior art

NSDictionary.value(forKeyPath:) supports retrieving nested values using dot notation.

Many existing model parsing frameworks support dot notation for decoding nested keys. Some examples include:

  • Mantle, "Model framework for Cocoa and Cocoa Touch"
  • Unbox, "The easy to use Swift JSON decoder"
  • ObjectMapper, "Simple JSON Object mapping written in Swift"

Detailed design

I propose implementing this behavior by introducing new JSONDecoder.NestedKeyDecodingStrategy and JSONEncoder.NestedKeyEncodingStrategy options. These options would function similarly to existing encoding and decoding options like KeyEncodingStrategy, DateEncodingStrategy, NonConformingFloatEncodingStrategy, etc.

open class JSONDecoder {

    /// The values that determine how a type's coding keys are used to decode nested object paths.
    public enum NestedKeyDecodingStrategy {
        // A nested key decoding strategy that doesn't treat key names as nested object paths during decoding.
        case useDefaultFlatKeys
        // A nested key decoding strategy that uses JSON Dot Notation to treat key names as nested object paths during decoding.
        case useDotNotation
        // A nested key decoding strategy that uses a custom mapping to treat key names as nested object paths during decoding.
        case custom((CodingKey) -> [CodingKey])
    }
    
    /// The strategy to use for encoding nested keys. Defaults to `.useDefaultFlatKeys`.
    open var nestedKeyEncodingStrategy: NestedKeyEncodingStrategy = .useDefaultFlatKeys
    
    // ...
    
}

JSONDecoder will use the NestedKeyDecodingStrategy to internally convert the original flat CodingKey into a nested [CodingKey] path. JSONDecoder will follow this path to retrieve the value for the given CodingKey.

Using the useDotNotation option, keys will be transformed using typical JSON / JavaScript dot notation:

  • "id" -> ["id"]
  • "metadata.review_end_date" -> ["metadata", "review_end_date"]
  • "arbitrarily.long.nested.path" -> ["arbitrarily", "long", "nested", "path"]

Passing NestedKeyDecodingStrategy.useDotNotation to our JSONDecoder instance allows the examples outlined above to be decoded using their compiler-synthesized codable implementation:

let decoder = JSONDecoder()
decoder.nestedKeyDecodingStrategy = .useDotNotation // Introduced in this proposal
decoder.keyDecodingStrategy = .convertFromSnakeCase
decoder.dateDecodingStrategy = .iso8601
try decoder.decode(EvolutionProposal.self, from: Data(originalJsonPayload.utf8)) // ✅

The same public API described for JSONDecoder.NestedKeyDecodingStrategy would be used for JSONEncoder.NestedKeyEncodingStrategy.

Alternatives considered

Make this the default behavior

Valid JSON keys may contain dots:

{
    "id": "SE-0274",
    "title": "Concise magic file names",
    "metadata.review_start_date": "2020-01-08T00:00:00Z",
    "metadata.review_end_date": "2020-01-16T00:00:00Z"
}

It's very likely that there are existing Codable models that rely on this behavior, so we must continue supporting it by default.

We could potentially make NestedKeyDecodingStrategy.useDotNotation the default behavior of JSONDecoder by preferring the flat key when present. This (probably) wouldn't break any existing models.

We wouldn't be able to support both nested and flat keys in JSONEncoder, since encoding is a one-to-one mapping (unlike decoding, which can potentially be a many-to-one mapping).

Support indexing into arrays or other advanced operations

This design could potentially support advanced operations like indexing into arrays (metadata.authors[0].email, etc). Objective-C Key-Value Coding paths, for example, has a very complex and sophisticated DSL. The author believes that there isn't enough need or existing precident to warrant a more complex design.

Open Questions

Could this be back-deployed to prior versions of iOS / the language toolchain? Otherwise it would have to be marked with, say, @available(iOS 14.0, *) annotations.

18 Likes

In general I have no objection to this proposal because, as you say, there's a decent amount of precedence for this from similar implementations.

The one objection I have is with the definition of NestedKeyDecodingStrategy:

On the surface this looks fine and has the benefit of being similar to the existing *Strategy types on decoders.

However, the implementation of this as an enum makes it extremely difficult to extend. This shortcoming is also present in every other *Strategy type and is (IMO) one of major flaws with the API.

By declaring this as an enum, we're explicitly not exposing the default implementations of these strategies to consumers. This means that anyone who wants to implement a custom strategy must do so from the ground up.

This is particularly felt with the KeyDecodingStrategy, where anyone wanting to do a "convert from snake case but allow for acronyms" strategy must implement the entire functionality of .convertFromSnakeCase and "recognize acronyms". (In other words, if you want to turn "html_url" into "htmlURL", you're entirely on your own.) You do not have access to the underlying code to turn it into htmlUrl and then post-process it.

The same short-coming exists here. Anyone who wants to build a custom NestedKeyDecodingStrategy but also rely on the the logic of the other decoders is entirely out-of-luck. Their only option is to essentially copy the implementation out of the standard library repository into their own code and go from there.

Instead, I propose that we learn from the mistakes of the existing API and instead define NestedKeyDecodingStrategy as a struct:

public struct NestedKeyDecodingStrategy {
    public static let useDefaultFlatKeys: NestedKeyDecodingStrategy
    public static let useDotNotation: NestedKeyDecodingStrategy
    public static func custom(_ decoder: @escaping (CodingKey) -> [CodingKey]) -> NestedKeyDecodingStrategy

    public func decode(_ codingKey: CodingKey) -> [CodingKey]
}

By making this a struct and exposing a decode(_:) method, you allow anyone implementing a custom strategy to create nested strategies that can re-use that logic, without requiring developers to wholesale copy code out of the standard library.

11 Likes

If we did use an enum we could add our own custom strategies using the 'catch all' custom case like:

extension NestedKeyDecodingStrategy {
    static var myStrategy: NestedKeyDecodingStrategy {
        return .custom { ... }
    }
}

However despite this I'm with Dave here. +1 to use a struct. Enums like this have always felt wrong to me.

My only slight amendment to Dave's suggestion is to drop the static func custom since 'custom's would be defined using init in a static member instead, like useDefaultFlatKeys and useDotNotation.

2 Likes

I'm a reluctant +1 on this. I want the capability, but find it unfortunate that for now this is probably the right way to do it.

I tried to implement this exact thing with a PropertyWrapper in CodableWrappers and found it to not be possible due to requiring a change to the [CodingKey] path. (BTW if someone knows of a workaround that would be awesome :relaxed:).

To be clear, the following is my frustration with the current state of Codable customization not a critique of this specific pitch.

What that, and work building an XML(En/De)coder showed me is that it's really unfortunate custom serialization has been put in the individual en/decoders. This and all the existing customization options in JSON(En/De)coder have nothing to do with JSON, and only exist because it's the only way to do it with the current architecture and tools available.

It also means they're limited to that Type. JSONEncoder and JSONDecoder have ~identical options, but we're forced to set them up separately for both. Beside that, it also means that generic customization isn't available for any other (En/De)coder, which means they have to be added separately, which means they usually aren't, (Not even in Foundation, See: PropertyListDecoder).

All that said, it seems we will need new language tools in order to do it "right" enough to belong in foundation. Since I'm guessing that's still years away and de-nesting serialization is something many would benefit from right now...I'm all for adding it.

2 Likes

What you're pointing to as a bug may be intended as a feature. Exposing the implementation means people can, and would use it in other places in ways for unintended use cases. That means any change to the implementation could break other's code. I can't speak to the author of the other options, but that's generally how ~Language level APIs are generally designed.

Regardless, all the other options are enums so unless we're going to change all of them to structs it's probably better to stay consistent.

2 Likes

Definitely in favor of the general idea. Implementation-wise I agree with Dave an Ian that a struct is probably the better choice.

How do you plan to avoid ambiguity in decoding this JSON using dot notation:

{
    "1": {
        "2": "a",
        "3": "b"
    },
    "1.2": "c"
}
5 Likes

Swift could add a decode method on the existing strategy enums, you don't need a new struct.

However, I've thought about this before and realised ... it's not actually that useful. If you have a convertFromSnakeCase strategy with a hypothetical coding key transform method at your disposal, what are you going to do with it? The function converts "html_url" to "htmlUrl", but does that really help you in your end to goal to convert to "htmlURL"? I don't think so. You are still going to have iterate the string and perform a lot of string manipulation logic. It's arguably easier to start from "html_url", split on underscores, and go from there.

By not adopting the dot-notation key coding strategy and decode in exactly the same way you currently do. This proposal is purely additative, opt-in and changes no current behaviour.

1 Like

I think this proposal is sensible and a nice convenient feature to have in the language. In a production use case, it is typically 'better' to make Codable types that mirror the incoming JSON stream exactly (so you would have a nested Metadata struct in the EvolutionProposal example), but that amount of ceremony often feels like overkill for otherwise straightforward decodes. Supporting nesting with dot notation is a nice-to-have for simple cases and things like scripting contexts too.

With a long term view, we can't add strategies for every single edge case. The overarching solution to addressing needs like this would be to extrapolate the core encoder/decoder logic into a standard library component that is customisable and reusable.

That being said, dot notation coding keys for JSONDecoder feels like something that would be used frequently enough that it is worthy to exist on its own anyway.

2 Likes

Thanks for looking into this, Cal. I have a few thoughts.

First, as mentioned above, I don't think that it is correct that parsing the example struct requires a custom implementation. This would also work:

struct EvolutionProposal: Codable {
    var id: String
    var title: String
    var metadata: Metadata
    struct Metadata: Codable {
        var reviewStartDate: Date
        var reviewEndDate: Date
    }
}

In this case, you wouldn't even need custom keys. The problem is of course that perhaps you don't want to expose this detail as API. I don't think it's actually that difficult to fix that:

struct EvolutionProposal: Codable {
    public var id: String
    public var title: String
    public var reviewStartDate: Date { metadata.reviewStartDate }
    public var reviewEndDate: Date { metadata.reviewEndDate }

    private var metadata: Metadata
    private struct Metadata: Codable {
        var reviewStartDate: Date
        var reviewEndDate: Date
    }
}

So, secondly, this brings us to one of the core ideas of Codable -- representation of your encoded value using Swift types as much as possible. Moving part of the structure into the key name is (sometimes) going in the opposite direction. Using "regular" private/public and struct types is, I think, more intuitive of a solution than requiring a strategy to be set in a different type.

Now, as the author of the key strategy implementation, I'm fully aware that sometimes the strategy approach can be useful. However, there are two main drawbacks:

  1. It applies "globally" across the entire archive. That moves part of the behavior of how encode/decode works from the type itself (where the most knowledge about structure lies) into the encoder/decoder.
  2. It does not apply across different kinds of encoders and decoders. If EvolutionProposal specified the keys with the . syntax then it would effectively require JSONEncoder to encode and decode itself, because part of the data structure is now part of the key name instead.

It's not a black-and-white API boundary. The strategy enums exist along this continuum of control belonging to the Coder vs the type itself.

With that in mind, can we continue to find improvements to reduce the boilerplate of the public/private split above? I think the answer is yes, and especially yes if we consider language improvements along with it that may be enablers. One idea would be a kind of anonymous type for struct Metadata, which is (other than Codable) not very useful. I'm not enough of a language expert to make a concrete proposal here, but perhaps others have an idea.

Another may be adding API to KeyedDecodingContainer (et. al.) that allows decoding with a key path like "x.y.z". Presumably we would need to figure out how to implement it generically, but if we could do that then all decoders would get an implementation of it for free and it would decouple the type's exact format from JSON.

18 Likes

yes-and, don't these also (mostly) apply to every other customization option that already exists in JSON(En/De)coder?

1 Like

The difference in my mind is that the other strategy options on JSONEncoder are more likely to be configuration options for the entire data. For example, it was created from some Java app where dates are specified as milliseconds. Or, the entire API of your REST service specifies that snake case is preferred.

This concept does not apply as cleanly to the idea of a particular type having a nested structure that plays a supporting role of sorts to the main type.

3 Likes

The difference in my mind is that the other strategy options on JSONEncoder are more likely to be configuration options for the entire data. .. This concept does not apply as cleanly to the idea of a particular type having a nested structure that plays a supporting role of sorts to the main type.

I'm thinking of this less as

  • the EvolutionProposal / Codable object has a nested structure.

and more as

  • the EvolutionProposal / Codable object is flat but can map to JSON objects with a nested structure.

It does seem correct for JSONEncoder and JSONDecoder to decide the way in which a type's keys map to and from specific JSON keys.

  • KeyEncodingStrategy.convertToSnakeCase maps ["metadata.reviewEndDate"] to ["metadata.review_end_date"].
  • NestedKeyEncodingStrategy.useDotNotation maps ["metadata.review_end_date"] to ["metadata", "review_end_date"].

All of these JSON payloads below are valid encodings with respect to the EvolutionProposal type itself. Even within the context of this pitch, the EvolutionProposal expresses no preference as to which encoded output is more correct or less correct. The actual output is determined by the JSONEncoder instance used to do the encoding:

  • Encoded with a default JSONEncoder:

    {
        "id": "SE-0274",
        "title": "Concise magic file names",
        "metadata.reviewStartDate": "2020-01-08T00:00:00Z",
        "metadata.reviewEndDate": "2020-01-16T00:00:00Z",
    }
    
    
  • Encoded with .convertToSnakeCase:

    {
        "id": "SE-0274",
        "title": "Concise magic file names",
        "metadata.review_start_date": "2020-01-08T00:00:00Z",
        "metadata.review_end_date": "2020-01-16T00:00:00Z",
    }
    
    
  • Encoded with .convertToSnakeCase and .useDotNotation:

    {
        "id": "SE-0274",
        "title": "Concise magic file names",
        "metadata": {
            "review_start_date": "2020-01-08T00:00:00Z",
            "review_end_date": "2020-01-16T00:00:00Z"
        }
    }
    

I can't really put my finger on it, but the fact that a Codable may encode to a different structure depending to the setting of the Encoder really concerns me.

Other than that I agree with this

1 Like

I suppose in that sense you're right. The problem, or at least the frustration, is there's no other way to do it without a custom init(from decoder: Decoder) or the extra boilerplate of a private struct.

Given the amount of Serialization libraries that have this-or-a-similar feature it seems likely it's something a meaningful number of people that want to do So, pragmatically their use case should be enabled, the question is how.

Given this, or something similar, has an API that will be familiar to existing users and requires no new language features I'm not able to think of a better, (at least short-medium term) solution.

I 100% agree with this analysis:

  • I think there's clear desire / precedent for this design based on similar existing tools.
  • I think this is the correct way to expose this behavior given the current implementation of JSONEncoder / JSONDecoder and the other existing Strategy options.

Yea, the problem is that current coding model has almost perfect separation of concern. A Codable knows nothing about each element or its parent, not without difficulty. It seems to be at odds with this pitch.

1 Like

In some senses yes, but I would posit my same frustrations here are a glaring omission.

For basic Types it's quite elegant. However needing to define things like dateEnodingStrategy in an Individual Encoder is information that IMO belongs to the Type being encoded and such is a major omission from a Codable Type's definition. In the other direction, Individual (En/De)coder altering the CodingKeys is another concern-separation violation.

I almost wonder if the route that should've been taken was that of String.StringInterpolation where you would define your own [en|de]code overloads - though IIRC that proposal came after Codable, so it wasn't thought of then.