CodingKeyPath: Add support for Encoding and Decoding nested objects with dot notation

phoneyDev · 2020-03-28T16:56:45.714Z

AlamoFireObjectWrapper has a feature similar to this. In the normal case when decoding you tell the decoder: decode my object of this Type. In the alternate case you tell the decoder: decode my object of this Type at this Keypath. It's simple and works well. This is one pain point in the current implementation of Decodable. I was recently converting some code from AlamoFireObjectWrapper to Decodable (since the release of Alamofire 5 supports Decodable better) and was forced to write a generic wrapper class to access my nested Type.

(Part of) this functionality can therefore be implemented by adding a keypath to the decoder. The other part of the proposal aims to flatten Types that are nested in the JSON. That's not so important to me. In a declarative world one would do something like that with something like the new enum or adding support for dot notation in the existing CodingKeys enum rather than forcing the developer to write code for this, like now.

One problem I see is that if the Type is modified to de-nest it then if the Type is found in different places in different JSON then there would be a conflict. I often work with different APIs that return either a single instance of a Type or a list of instances of the Type where the list may be paged so that other properties exist at the top level. If the Type has to be changed to indicate that it's nested then things won't work in the not nested case.

drexin · 2021-04-19T18:51:36.986Z

First of all, thanks for the pitch. I am reviving this, because there has been some discussion around this offline.

I am generally in favor of adding more customization options, but they should be well justified. Other people on this thread have already mentioned, that in the particular example case it would be preferable to use a separate type for the metadata. I'm not saying that there aren't any cases where this proposal would add sufficient benefit, but it would be good to have such an example in the proposal.

Another concern I share with others, is that the name CodingKeyPath implies that it shares the typesafe and structural nature of KeyPath, which it doesn't.

Additionally I am not a fan of using strings to encode structural paths. In the protocol it is then being represented as [CodingKey]. So how are we getting from String to [CodingKey]? It seems that the String representation would only be used by the compiler, to generate the CodingKey enums that end up being used to represent the path. I don't think the compiler should interpret string contents in this way.

cal · 2021-04-19T19:51:17.104Z

Thanks for the feedback!

Another concern I share with others, is that the name CodingKeyPath implies that it shares the typesafe and structural nature of KeyPath , which it doesn't.

Open to alternative names for this! Perhaps just "CodingPath" would work here?

Additionally I am not a fan of using strings to encode structural paths. In the protocol it is then being represented as [CodingKey] . So how are we getting from String to [CodingKey] ?

In the specific design outlined in this pitch, the [CodingKey] path is generated by splitting the String at each dot (.). There is a fair bit of prior art for this in other modal parsing frameworks, but I agree that representing the path using [String] would be preferable over using dot notation in a single String.

For example, I think a design like this would be quite ergonomic:

enum CodingPaths: [String], CodingPath {
  case id
  case title
  case reviewStartDate = ["metadata", "review_start_date"]
  case reviewEndDate = ["metadata", "review_end_date"]
}

This is not currently supported (I get a Raw value for enum case must be a literal error). I'm not familiar with why this limitation exists -- it seems like this could be possible in theory.

drexin · 2021-04-19T20:54:20.643Z

cal:

Additionally I am not a fan of using strings to encode structural paths. In the protocol it is then being represented as [CodingKey] . So how are we getting from String to [CodingKey] ?

In the specific design outlined in this pitch, the [CodingKey] path is generated by splitting the String at each dot (.). There is a fair bit of prior art for this in other modal parsing frameworks, but I agree that representing the path using [String] would be preferable over using dot notation in a single String.

For example, I think a design like this would be quite ergonomic:
enum CodingPaths: [String], CodingPath {
  case id
  case title
  case reviewStartDate = ["metadata", "review_start_date"]
  case reviewEndDate = ["metadata", "review_end_date"]
}
This is not currently supported (I get a Raw value for enum case must be a literal error). I'm not familiar with why this limitation exists -- it seems like this could be possible in theory.

I should have been more clear: I don't think that the keys should be represented as strings, only to have them turned into CodingKeys by the compiler. The reason it is okay for the existing CodingKey to use String, is that it is a runtime only override of the field name. The compiler does not look at the string itself and certainly does not interpret it in any way to generate code. If we are going to add this feature, I don't want it to feel like a hack and if there is anything we have to add to make it nice, we need a strong justification for the overall feature first.

millenomi · 2021-04-19T23:44:16.302Z

cal:

This proposal is purely additive, so it has no appreciable effect on source compatibility.

This is prima facie incorrect (you're changing the semantics of the CodingKey string raw value interpretation); and also may have real-life effects, because the pitch isn't just for JSON containers. For example, I might see the Vapor URL decoding setup breaking easily with this change, as URL parameters often have dots for namespacing (e.g. for OAuth-mandated fields).

I recommend perhaps a change in semantics when the coding key value is a different type, perhaps a path type of some sort rather than a dot in a string.

millenomi · 2021-04-19T23:46:34.085Z

cal:

Open to alternative names for this! Perhaps just " CodingPath " would work here?

Encoder calls an array of coding keys a 'coding path': codingPath | Apple Developer Documentation

cal · 2021-04-19T23:50:30.950Z

millenomi:

you're changing the semantics of the CodingKey string raw value interpretation

This pitch doesn't propose any changes to the CodingKey type itself. It would definitely be a breaking change to tweak the semantics of the existing CodingKey type:

cal:

Alternatives considered

Make this the default behavior for CodingKeys

Valid JSON keys may contain dots:
{
    "id": "SE-0274",
    "title": "Concise magic file names",
    "metadata.review_start_date": "2020-01-08T00:00:00Z",
    "metadata.review_end_date": "2020-01-16T00:00:00Z"
}
It's practically guaranteed that there are existing Codable models that rely on this behavior. We can't add dot-notation keypath semantics to the existing CodingKeys type without breaking backwards compatibility for these models.

millenomi:

I recommend perhaps a change in semantics when the coding key value is a different type, perhaps a path type of some sort rather than a dot in a string.

Totally agree -- this pitch proposes a new CodingKeyPath / CodingPath type separate from the existing CodingKey type

tomerd · 2021-08-03T23:50:18.749Z

Hi @cal,

Thanks for making this pitch. It brings to light the fact we can improve the ergonomic of decoding deeply nested, large payloads that are encoded by an external party. We realize that such decoding scenarios are common on both server and client where you have to deal with data coming from external sources, commonly JSON from RESTful HTTP services, and would like to see it solved.

@Tony_Parker and myself discussed this further, and the ergonomic issue we see is that it can be fairly heavy handed to need and define the entire data structure hierarchy when the path to the data you want to extract is well known. For example, given the contrived JSON:

{ "a": { "b": { "c": { "d": { "e": { "f": { "g": { "h": { "i": { "foo": 42, "bar": true }}}}}}}}}}

It could be nice to only need and declare:

struct Payload: Codable  {
  var foo: Int
  var bar: Bool
}

And be able to decode it directly, given that we know the path to the payload is “a.b.c.d.e.f.g.h.i”. For completeness, the achieve this today you need to declare the entire hierarchy:

struct Payload: Codable {
  var a: A
  
  struct A: Codable {
    var b: B
  
    struct B: Codable {
      var c: C
    
      struct C: Codable {
        var d: D
      
        struct D: Codable {
          var e: E
        
          struct E: Codable {        
            var f: F
          
            struct F: Codable {
              var g: G
                      
              struct G: Codable {
                var h: H 
                         
                struct H: Codable {              
                  var i: I
                
                  struct I: Codable  {
                    var foo: Int
                    var bar: Bool
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

and access it via the type-safe accessors:

let payload = JSONDecoder().decode(Payload.self, from: data)
let i = payload.a.b.c.d.e.f.g.h.i

The main advantage of the current API is that it puts emphasis on type-safety. As such, we don’t feel string based key parsing (as proposed) is the way to go, given the following pitfalls with this (some of which have been pointed out on the thread):

It introduces complexity into understanding Codable because there are now two kinds of keys (CodingKeys and CodingKeyPaths).
If and when the decoding requirements expand and the user needs more values, it’s tempting to continue down the string-based path instead of using strong-types, or IOW it subtly encourages users to prefer short term convenience over type-safety.

That said, if we narrow the proposal to focus on the initial axis into the data, i.e. in such cases that the data starts at some well know deeply nested path, then we could probably solve this with a higher level API on the concrete Decoder itself. to use the example above, something like:

let payload = JSONDecoder().decode(Payload.self, from: data, at: "a.b.c.e.f.g.h.i")

Where that last argument is a CodingKeyPath type which is expressible by string literal. That reduces the scope of the change significantly and avoids the pitfalls mentions above. The last argument will do the string parsing before giving JSONDecoder an Array-like type, leaving the complexity outside the decoder itself.

We should further discuss where this higher level API belongs - the concrete Decoders or somewhere else, but we suspect this problem is more common in JSON compared to other Codable formats, so a JSON centric solution could potentially suffice.

cal · 2021-08-03T23:55:24.437Z

This would be a great improvement!

tomerd · 2021-08-04T00:01:15.936Z

cal:

This would be a great improvement!

Thanks @cal. Do you want to make the formal pitch and proposal for such change?

cal · 2021-08-04T00:16:37.019Z

Are you thinking we would add this to Decoder (likely via a protocol extension, so it would be supported on all concrete Decoders?), or specifically just to JSONDecoder?

Last I heard, Foundation (and thus additions to JSONDecoder) are not included in the scope of Swift Evolution. Is that still the case?

I do think Decoder is a reasonable place for this, since it seems broadly useful and should compose nicely on top of existing methods available on the Decoder API. If that direction feels reasonable, I could work on a proposal and implementation for this

tomerd · 2021-08-04T00:50:38.243Z

cal:

Are you thinking we would add this to Decoder (likely via a protocol extension, so it would be supported on all concrete Decoder s?), or specifically just to JSONDecoder ?

Last I heard, Foundation (and thus additions to JSONDecoder ) are not included in the scope of Swift Evolution. Is that still the case?

I do think Decoder is a reasonable place for this, since it seems broadly useful and should compose nicely on top of existing methods available on the Decoder API. If that direction feels reasonable, I could work on a proposal and implementation for this

@Tony_Parker do you have a preference / opinion?

tomerd · 2021-08-04T16:42:27.881Z

cc @drexin @ktoso @weissi @fabianfett

Philippe_Hausler · 2021-08-04T16:46:30.253Z

That almost seems like it should be an extension on TopLevelDecoder only right? Which currently lives in Combine, and I think I can safely say that was something we have wanted to move lower for a while now. Perhaps this is the impetus to do so?

alextud · 2021-08-04T17:10:48.616Z

This will be a good addition. Currently we use same approach as most API call returns data at different root keys based on resource type.

extension JSONDecoder {
    
    fileprivate static let keyPaths: CodingUserInfoKey = CodingUserInfoKey(rawValue: "keyPath")!
    
    open func decode<T>(_ type: T.Type, from data: Data, keyPath: String, separator: Character = ".") throws -> T where T : Decodable {
        self.userInfo[JSONDecoder.keyPaths] = keyPath.split(separator: separator).map({ String($0) })
        return try decode(ProxyModel<T>.self, from: data).object
    }
}

tera · 2021-08-04T17:20:52.187Z

tomerd:

For example, given the contrived JSON:

{ "a": { "b": { "c": { "d": { "e": { "f": { "g": { "h": { "i": { "foo": 42, "bar": true }}}}}}}}}}

It could be nice to only need and declare:
struct Payload: Codable  {
  var foo: Int
  var bar: Bool
}
And be able to decode it directly, given that we know the path to the payload is “a.b.c.d.e.f.g.h.i”.

granted this is a hack and far from nice by any metric, but still doable today!

let object = try JSONSerialization.jsonObject(with: data) as! NSObject
let i = object.value(forKeyPath: "a.b.c.d.e.f.g.h.i")!
let data2 = try JSONSerialization.data(withJSONObject: i)
let a = try JSONDecoder().decode(Payload.self, from: data2)

tomerd · 2021-08-04T18:13:05.548Z

Philippe_Hausler:

That almost seems like it should be an extension on TopLevelDecoder only right? Which currently lives in Combine, and I think I can safely say that was something we have wanted to move lower for a while now. Perhaps this is the impetus to do so?

personally, I would love to see that. Since its useful regardless, maybe a separate pitch to add a TopLevelDecoder/TopLevelEncoder to the Stdlib is in place, then if we decide TopLevelDecoder is the right place for such API, we can relate the second pitch to the TopLevelEncoder/TopLevelDecoder one.

Tony_Parker · 2021-08-11T23:42:50.434Z

Is this possible to implement in a generic context like TopLevelDecoder? I imagined that JSONDecoder would be able to do a far more efficient implementation by descending into the JSON data manually first.

Tony_Parker · 2021-08-11T23:43:25.886Z

If we decided here to add it to JSONDecoder, I will help make that happen for Foundation.

itaiferber · 2021-08-12T01:13:14.766Z

One thought here: if this does get expressed in a generic way at a top-level context, I think it would be nice to spell the method as

static func decode<T: Decodable>(_ type: T.Type, from data: Data, at codingKeyPath: [CodingKey])

to allow key paths to include keys which have a delimiter in them (e.g. "my.key", which would otherwise be parsed as ["my", "key"]), as well as integer keys.

It's then possible to also offer

static func decode<T: Decodable>(_ type: T.Type, from data: Data, at codingKeyPath: String, separator: String = ".")

with a default implementation which splits codingKeyPath on the delimiter, turns each component into a CodingKey, then calls the [CodingKey] variant. JSONDecoder (and others) could also override this implementation to do something even more efficient if relevant.

(The [CodingKey] version can also made significantly easier to call using a general key type, e.g. AnyCodingKey pitched in [Pitch] Allow coding of non-`String`/`Int` keyed `Dictionary` into a `KeyedContainer`)