CodingKeyPath: Add support for Encoding and Decoding nested objects with dot notation

Introduction

Today, encoding and decoding Codable objects using the compiler's synthesized implementation requires that your object graph has a one-to-one mapping to the object graph of the target payload. This decreases the control that authors have over their Codable models.

I propose that we add a new CodingKeyPath type that allows consumers to key into nested objects using dot notation.

Previous related pitch thread: Add support for Encoding and Decoding nested JSON keys

Motivation

Application authors often have little to no control over the structure of the encoded payloads they receive. It is often desirable to rename or reorganize fields of the payload at the time of decoding.

Here is a theoretical JSON payload representing a Swift Evolution proposal (SE-0274):

{
  "id": "SE-0274",
  "title": "Concise magic file names",
  "metadata": {
    "review_start_date": "2020-01-08T00:00:00Z",
    "review_end_date": "2020-01-16T00:00:00Z"
  }
}

The consumer of this payload may prefer to hoist fields from the metadata object to the root level:

struct EvolutionProposal: Codable {
  var id: String
  var title: String
  var reviewStartDate: Date
  var reviewEndDate: Date
}

Today, this would require writing a fair amount of boilerplate. The consumer would need to either write custom encoding and decoding implementation or proxy to Codable subtypes.

Proposed solution

I propose that we add a new CodingKeyPath type that allows consumers to key into nested objects using dot notation.

struct EvolutionProposal: Codable {
  var id: String
  var title: String
  var reviewStartDate: Date
  var reviewEndDate: Date
    
  enum CodingKeyPaths: String, CodingKeyPath {
    case id
    case title
    case reviewStartDate = "metadata.review_start_date"
    case reviewEndDate = "metadata.review_end_date"
  }
}

Prior art

NSDictionary.value(forKeyPath:) supports retrieving nested values using dot notation.

Many existing model parsing frameworks support dot notation for accessing nested keys. Some examples include:

  • Mantle, "Model framework for Cocoa and Cocoa Touch"
  • Unbox, "The easy to use Swift JSON decoder"
  • ObjectMapper, "Simple JSON Object mapping written in Swift"

Detailed design

New Standard Library types

/// A type that can be used as a key path for encoding and decoding.
public protocol CodingKeyPath {
  /// The components of this path. Derived automatically for a `CodingKeyPaths` enum:
  ///
  ///   enum CodingKeyPaths: String, CodingKeyPath {
  ///     /// components = ["rootValue"]
  ///     case rootValue       
  ///
  ///     /// components = ["nestedObject", "value"]
  ///     case nestedValue = "nestedObject.value" 
  ///  }
  ///
  var components: [CodingKey] { get }
}

/// A container for encoding with a `CodingKeyPath` type.
///  - Internally wraps a `KeyedEncodingContainer`. 
///  - Recursively follows a `CodingKeyPath` by encoding a `nestedContainer` for each component.
public struct KeyPathEncodingContainer<K: CodingKeyPath> {
  public mutating func encode<T>(_ value: T, forKeyPath keyPath: K) throws where T: Encodable
  public mutating func encodeIfPresent<T>(_ value: T?, forKeyPath keyPath: K) throws where T: Encodable
}

/// A container for decoding with a `CodingKeyPath` type.
///  - Internally wraps a `KeyedDecodingContainer`. 
///  - Recursively follows a `CodingKeyPath` by decoding a`nestedContainer` for each component.
public struct KeyPathDecodingContainer<K> where K: CodingKeyPath {  
  public func decode<T>(_ type: T.Type, forKeyPath keyPath: K) throws -> T where T: Decodable
  public func decodeIfPresent<T>(_ type: T.Type, forKeyPath keyPath: K) throws -> T? where T: Decodable
}

New Standard Library methods

This proposal doesn't add any new requirements on the Encoder and Decoder protocols, so all existing implementations (JSONEncoder, PlistDecoder, etc.) will receive this behavior automatically.

  • KeyPathEncodingContainer and KeyPathDecodingContainer simply wrap the existing KeyedEncodingContainer and KeyedDecodingContainer types, so they don't require any additional support.
public extension Encoder {
  func keyPathContainer<KeyPath>(keyedBy type: KeyPath.Type) -> KeyPathEncodingContainer<KeyPath> where KeyPath: CodingKeyPath
}

public extension KeyedEncodingContainer {
  mutating func nestedKeyPathContainer<KeyPath>(keyedBy type: KeyPath.Type, forKey key: Key) throws -> KeyPathEncodingContainer<KeyPath> where KeyPath: CodingKeyPath
}

public extension UnkeyedEncodingContainer {
  mutating func nestedKeyPathContainer<KeyPath>(keyedBy type: KeyPath.Type) throws -> KeyPathEncodingContainer<KeyPath> where KeyPath: CodingKeyPath
}
public extension Decoder {
  func keyPathContainer<KeyPath>(keyedBy type: KeyPath.Type) throws -> KeyPathDecodingContainer<KeyPath> where KeyPath: CodingKeyPath
}

public extension KeyedDecodingContainer {
  mutating func nestedKeyPathContainer<KeyPath>(keyedBy type: KeyPath.Type, forKey key: Key) throws -> KeyPathDecodingContainer<KeyPath> where KeyPath: CodingKeyPath
}

public extension UnkeyedDecodingContainer {
  mutating func nestedKeyPathContainer<KeyPath>(keyedBy type: KeyPath.Type) throws -> KeyPathDecodingContainer<KeyPath> where KeyPath: CodingKeyPath
}

Codable Conformance Synthesis

The compiler with synthesize init(from decoder: Decoder) and encode(to encoder: Encoder) implementations for types that provide a CodingKeyPaths enum.

  • It is invalid for a type to provide both a CodingKeys enum and a CodingKeyPaths enum.
  • If a Codable type doesn't provide either a CodingKeys enum or a CodingKeyPaths enum, the compiler will synthesize a CodingKeys enum.
  • The compiler will never automatically synthesize a CodingKeyPaths enum.
struct EvolutionProposal: Codable {
  var id: String
  var title: String
  var reviewStartDate: Date
  var reviewEndDate: Date

  enum CodingKeyPaths: String, CodingKeyPath {
    case id
    case title
    case reviewStartDate = "metadata.reviewStartDate"
    case reviewEndDate = "metadata.reviewEndDate"
  }
    
  /// Synthesized by the compiler:
  init(from decoder: Decoder) throws {
    let container = try decoder.keyPathContainer(keyedBy: CodingKeyPaths.self)
    id = try container.decode(String.self, forKeyPath: .id)
    title = try container.decode(String.self, forKeyPath: .title)
    reviewStartDate = try container.decode(Date.self, forKeyPath: .reviewStartDate)
    reviewEndDate = try container.decode(Date.self, forKeyPath: .reviewEndDate)
  }
    
  /// Synthesized by the compiler:
  func encode(to encoder: Encoder) throws {
    var container = encoder.keyPathContainer(keyedBy: CodingKeyPaths.self)
    try container.encode(id, forKeyPath: .id)
    try container.encode(title, forKeyPath: .title)
    try container.encode(reviewStartDate, forKeyPath: .reviewStartDate)
    try container.encode(reviewEndDate, forKeyPath: .reviewEndDate)
  }
    
}

Source compatibility

This proposal is purely additive, so it has no appreciable effect on source compatibility.

  • Code synthesis behavior and/or source validity may change for Codable models that currently have a subtype named CodingKeyPaths.
  • A quick GitHub search for enum CodingKeyPaths doesn't yield any relevant results, so this seems like a non-issue.

Effect on ABI stability

This proposal is purely additive, so it has no effect on ABI stability.

Effect on API resilience

This proposal is purely additive to the public API of the Standard Library. If this proposal was adopted and implemented, it would not be able to be removed resiliently.

Alternatives considered

Support indexing into arrays or other advanced operations

This design could potentially support advanced operations like indexing into arrays (metadata.authors[0].email, etc). Objective-C Key-Value Coding paths, for example, has a very complex and sophisticated DSL.

  • The author believes that there isn't enough need or existing precident to warrant a more complex design.
  • Indexing into arrays seems useful on the surface, but would be quite limited in practice.
    • For example, you would be able to index into the first element of an array ([0]) but not the last element of the array.
    • Additionally, UnkeyedEncodingContainer and UnkeyedDecodingContainer only support sequential access (no performant support for random access).

Make this the default behavior for CodingKeys

Valid JSON keys may contain dots:

{
    "id": "SE-0274",
    "title": "Concise magic file names",
    "metadata.review_start_date": "2020-01-08T00:00:00Z",
    "metadata.review_end_date": "2020-01-16T00:00:00Z"
}

It's practically guaranteed that there are existing Codable models that rely on this behavior. We can't add dot-notation keypath semantics to the existing CodingKeys type without breaking backwards compatibility for these models.

  • We could make this the default decoding behavior without breaking backwards compatibility by preferring the flat key when an exact match is present.

  • We cannot make this the default encoding behavior without breaking backwards compatibility. Encoding must be a one-to-one mapping (unlike decoding, which can potentially be a many-to-one mapping).

Enable this behavior by setting a flag on the encoder or decoder

A previous version of this proposal added NestedKeyEncodingStrategy and NestedKeyDecodingStrategy configuration flags to Foundation.JSONEncoder and Foundation.JSONDecoder:

let decoder = JSONDecoder()
decoder.nestedKeyDecodingStrategy = .useDotNotation
try decoder.decode(EvolutionProposal.self, from: Data(originalJsonPayload.utf8))

@Tony_Parker (on the Foundation team at Apple) noted two main drawbacks to that approach:

This CodingKeyPaths approach described in this proposal is:

  1. Configured on a per-type basis
  2. Compatible "for free" with all existing Encoder and Decoder implementations.

More Alternatives Considered

Additional Alternatives Considered were added based on feedback from this pitch thread:

2 Likes

I have the concern that this addresses a rather minute problem particular to JSON rather than all possible encodings. More generally even, the type bears the knowledge that there is some representation that for an external reason needs nesting behavior — while the point of Encodable/Decodable is to make the encoding/decoding process agnostic of this representation. If something should care of it — it is the particular encoder/decoder that recursively seeks for a given key.

To clarify: mathematically, "metadata.reviewStartDate" and "reviewStartDate" are equivalent in the sense that they are still distinct (at least in your example — think namespaces); introducing the path component didn't really make any difference for the decoded/encoded struct in terms of uniqueness of the keys.

3 Likes

Looks good, but please do not make the same mistake that CodingKeys did. Each of your CodingKeyPaths represents one path, not multiple paths.

3 Likes

Addendum to my post above:

"One-to-one" merely means "unambiguous in both ways" (or again in maths terms, a bijection); it has nothing to with the actual mapping — be the mapping's domain numbers, strings, keys, key paths etc. Your implementation is still one-to-one, because you're only interested in the leaf nodes. If a type (or for that sake, decoder/encoder) is so concerned with the internal structure of the encoding, it'd be anyway wiser for the actual implementation to introduce a domain-specific function that actually does the mapping, since key paths are only a subset of cases that you can theoretically handle, and if you continue going this way, you'd need to introduce a whole number of coding strategies to cover all possible scenarios (numbers, strings, emojis etc.) :man_shrugging:t2:

It feels to me like the example you presented would be simpler if the nesting is explicit:

struct EvolutionProposal: Codable {
  let id: String
  let title: String

  struct Metadata: Codable {
    let reviewStart: Date
    let reviewEnd: Date
  }
  let metadata: Metadata
}

If it really bothers you to type proposal.metadata.reviewStart you could always use @dynamicMemberLookup to simplify the access:

@dynamicMemberLookup
struct EvolutionProposal: Codable {
   <fields>
  subscript<T>(dynamicMember keyPath: KeyPath<Metadata, T>) -> T {
    return metadata[keyPath: keyPath]
  }
}

This would allow you to do proposal.reviewStart in your own code.

I believe you could even construct a CodableWithMetata protocol that provides a default implementation of subscript<T>(...) for types that have a metadata field, so all you would need is @dynamicMemberLookup.

3 Likes

I’m not sure I have ever come across a situation where it was annoying enough to need to represent the JSON structure explicitly that I would have used this feature to avoid it; conversely, I have definitely come across plenty of situations where I feel it is beneficial to write the little extra code to explicitly represent each object in the JSON structure.

Nevertheless, I do have one piece of constructive criticism on the pitch as-written. Even as you are avoiding conflict with the need to parse documents where a dot (.) is found as part of a name by using a new type instead of reusing CodingKeys you are introducing an incompatibility of your feature with any such names. Did you consider adopting a key path syntax like JSON Pointer that solves for these ambiguities already?

Another alternative, in case anyone feels this doesn't apply well to anything outside the realm of JSON: no new key types are added, but add a property to JSONDecoder that determines how it should handle keys with . in them. Something like keyDecodingStrategy but orthogonal to the enums that already exist there (perhaps keyDecodingStragety should have been an options set?) .

Anyway, this property could be used to determine whether it assumes "foo.bar" refers to a single key or two separate keys, where bar is a nested key. I suspect this would satisfy most people's needs—it would certainly satisfy mine.

This is something I've wanted for a long time. Numerous small nested structs can turn into a code smell, and I like to avoid implementing the synthesized init at all costs, especially for common operations like lifting nested keys to the top level.

1 Like

Do we need a new Container type, or could the CodingKeyPath type just be an argument to a new function on KeyedCodingContainer?

KeyedEncodingContainer is generic on a specific CodingKey type, so models that use CodingKeyPaths (instead of CodingKeys) wouldn't be able to create one:

protocol Encoder {
  func container<Key: CodingKey>(keyedBy type: Key.Type) -> KeyedEncodingContainer<Key>
}

I think it's best to stick to existing precedent when writing Evolution Proposals. A new CodingKeyPath system should follow the same semantics as the existing CodingKey system.

I also personally agree with the original rational for choosing the name CodingKeys:

The CodingKeys type acts more as a namespace for keys, rather than an enumeration itself.

It would be exceedingly rare to need to type var key = CodingKeys.myProperty. Instead, the naming was chosen to be read in context: encoder.container(keyedBy: CodingKeys.self) should read as “a container keyed by my coding keys” (“keys” here, being a plural word, is more intuitive than “key”).

Note that because the keyed container is generic on the key namespace, it’s never necessary to refer to the full type again: container.encode(myProperty, forKey: .myProperty), not container.encode(myProperty, forKey: CodingKeys.myProperty). The type is only referred to explicitly once, and the individual cases afterwards are “self-evident”.

I think we are both wrong to have called it a "rationale". That word would imply that there was specific intent before the name was chosen. Instead, the name is just wrong and now retroactive kool-aid has to be poured.

Could this be implemented as a new requirement of the CodingKey protocol?

 public protocol CodingKey: CustomStringConvertible,
                            CustomDebugStringConvertible {
 
   // Existing requirements.
   var stringValue: String { get }
   init?(stringValue: String)
 
   // Existing requirements.
   var intValue: Int? { get }
   init?(intValue: Int)
 
+  // New requirement.
+  var isNested: Bool
 }
 
+extension CodingKey {
+
+  // Default implementation.
+  public var isNested: Bool { false }
+}

The example from your proposal would become:

 struct EvolutionProposal: Codable {
   var id: String
   var title: String
   var reviewStartDate: Date
   var reviewEndDate: Date
 
-  enum CodingKeyPaths: String, CodingKeyPath {
+  enum CodingKeys: String, CodingKey {
     case id
     case title
     case reviewStartDate = "metadata.review_start_date"
     case reviewEndDate = "metadata.review_end_date"
 
+    // Opt into nested objects with dot notation.
+    var isNested: Bool { true }
   }
 }

As the person ultimately responsible for this API naming, and who responded to you in Radar (quoted above), I have to respectfully dispute this — the names for all of the API components were all intentionally chosen after much debate.

To expand on the quotes above and in the linked blog post: we deviated from existing naming guidelines because CodingKeys is not a traditional enum. The recommendation for singular names makes sense within the context as described — traditional enums are inhabited by a single value at a time, so a singular name is preferred:

  • A CompassPoint has only one direction at a time: .north, .south, .east, or .west
  • A Result has only one value at a time: .success or .failure
  • A DispatchQoS has only one QoSClass at a time: .background, .utility, .userInitiated, ... etc.
  • A UIButton has only one ButtonType at a time: .system, .plain, .custom, ... etc.

Importantly, the singularity of these type names makes sense because these values are typically stored properties which can have only one value at a time.

In contrast, a type with CodingKeys has all of the values enumerated in the CodingKeys type: an .id and a .title and a .reviewStartDate and a .reviewEndDate, in the form of properties on the type itself. This is how I refer to this type in speech, too: "in order to override some of the default behavior for encoding and decoding your values, you can manually provide a coding keys enum to map your properties differently".

Moreover, CodingKeys enums are not meant to be stored anywhere; their values are referred to by name at call sites, so the singularity requirement of traditional enums is largely irrelevant. This is what I tried to describe by stating

The CodingKeys type acts more as a namespace for keys than an enumeration itself.

but I should have been clearer and described this more in depth.

To that end, CodingKeyPaths is consistent and would be the preferred naming for this new type: it is the list of coding key paths which define how your type encodes and decodes.

6 Likes

To weigh in on the proposal itself: I think this is a solid approach for a quality-of-life improvement for those who have to work with serialization formats outside of their control, for which a lot of nesting makes nested types somewhat impractical. I did initially have two questions, which were answered by reviewing the implementation itself:

  1. What is the underlying type of CodingKeyPath.components given that components is a type-erased [CodingKey]? These are given the concrete _CodingKeyPathComponent CodingKey type, which can take on any String value
  2. Given that getting nested components requires concrete key types, how are you planning on fetching nested containers using type-erased [CodingKey] lists? The coding keys become concrete _CodingKeyPathComponent keys which can be used to decode

It would be nice for this to be spelled out a little bit more clearly in the pitch itself.

Overall, I prefer this approach to the Encoder/Decoder-specific approach: this design gives every Encoder and Decoder a free implementation, which is nice.


That being said, I will say that I am somewhat apprehensive about this on the whole:

  1. For many developers, CodingKeys and the implicit behavior around them can already at times be surprising and confusing, and I am somewhat worried that further expanding implicit behaviors with an additional approach can be even more confusing, especially since it will actually still be possible to define

    enum CodingKeys: String, CodingKey {
        case nested = "my.nested.key"
    }
    

    but expect it to behave like a CodingKeyPath.

    To that end, too: what happens if I define

    enum CodingKeys: String, CodingKeyPath { ... }
    

    or

    enum CodingKeyPaths: String, CodingKey { ... }
    

    by accident? Is there a way to simplify this current approach to make it more difficult (or ideally, impossible) to get wrong?

  2. I'm slightly concerned about the use of "Key Path" in the name here, because of potential confusing with KeyPaths, which are similar. (Indeed, we've had a few posts on the forums already in the past asking why we don't just use KeyPaths, the answer to which is that key paths don't map 1-to-1 with CodingKeys [you can have CodingKeys which don't map to any property, for example, but not so with KeyPaths].)

    This is actually the reason why Encoder and Decoder have .codingPath rather than .codingKeyPath — to try to avoid some of the confusion of similar naming in similar-but-not-quite-aligning contexts.

These two concerns aren't a problem with your specific pitch, but are two areas that I think would need to have really compelling answers to make a change like this easier to swallow. Given that we already have solutions to this problem overall today (ideally, use nested types) and that it's relatively rare for those solutions to be sufficiently inconvenient to warrant this, I would personally want a solution in this area to significantly outweigh any potential confusion in order to be obviously appealing.

3 Likes

This is really valuable feedback. Do you think the name CodingPaths (instead of CodingKeyPaths) would make more sense for this feature?


For a type to receive a synthesized Codable implementation, the compiler requires exactly either enum CodingKeys: CodingKey or enum CodingKeyPaths: CodingKeyPath.

If you mix up the protocols, the compiler emits a note and won't attempt to synthesize your Codable conformance (existing behavior from the 5.1 compiler):

note: cannot automatically synthesize 'Encodable' because 'CodingKeys' does not conform to CodingKey
    enum CodingKeys: String, CodingKeyPath {
         ^
note: cannot automatically synthesize 'Encodable' because 'CodingKeyPaths' does not conform to CodingKeyPath
    enum CodingKeyPaths: String, CodingKey {
         ^


This one is tricky, since the code as-written is perfectly valid and still useful.

If we're especially concerned about users getting this wrong, one solution could be to introduce an educational warning:

:warning: CodingKey values are treated as exact keys, not nested paths.

  • [FIXIT] To treat it as the nested path ["my", "nested", "key"], use CodingKeyPaths
  • [FIXIT] To silence this warning and treat it as the exact key "my.nested.key", use a raw string literal: #"my.nested.key"#

This is a design I considered, but it has two major drawbacks:

  1. A "key" and a "path" have fundamentally different encoding and decoding semantics, so I think they should be represented by different types.

  2. Existing Encoder and Decoder implementations would have to be updated to check for and respect this flag.
    • This includes Foundation.JSONEncoder, and Foundation.PlistEncoder, but also all custom third-party encoders and decoders.
    • The compiler wouldn't be able to enforce that isNested is handled correctly.
    • The CodingKeyPath implementation in this proposal exists entirely within the Standard Library by composing existing behavior (no changes necessary to existing encoders and decoders).

This would probably be a good addition to the proposal's "Alternatives Considered" section.

1 Like

As a standalone proposal targeted at a relatively narrow problem, I agree. But the fact is that there is a broad set of challenges we face when working with serialization formats outside of our control. I am reluctant to move forward with a narrow proposal without seeing how it fits into a broader solution that addresses the full problem space. IMO, a manifesto-style document is warranted before we move forward with enhancements to Codable.

Rust's serde has been mentioned in the past. I would very much like to see a similar extensible attribute-based approach for specifying coding strategies. Specifically one that does not affect the type's storage the way property wrappers do, and instead only affects synthesis of its Codable conformance.

CodingKeyPaths may still have a place in such a world, but maybe they don't. Personally, I would much rather configure the coding key (or key path) alongside the property itself rather than have the indirection that CodingKeys requires. My Sourcery template supports this and it's very nice. When you're able to specify coding behavior this way there is no need for CodingKeys or CodingKeyPaths.

If we move forward with narrow solutions and no vision for where the overall design is headed, we may end up with too much inertia to end up with a more complete, coherent and consistent design.

7 Likes

Could you share some specific examples? What other problems in this space are you concerned about?

By far the largest and most complex topic is dealing with encodings of enums with associated values. There are many different ways to encode sum types and no standard format. This problem is exacerbated by the fact that most languages used in cloud development don't even support sum types so and many cloud developers are not familiar with them. So you can end up with an ad-hoc mess of encodings of what is essentialy a sum type even in a single API.

Some other miscellaneous things that I have seen:

  • providing a default value used when a key is missing or value is null
  • converting invalid values to nil or a default value
  • local (properly-specific) encoding strategies (most frequently needed for dates)
  • properties that are ignored for the purpose of encoding / decoding
  • "empty" values that are omitted when encoding (usually array or dictionary)
  • "inline" properties whose coding keys appear at the same level in the encoded format as peer properties - kind of the opposite of coding key paths

These are some examples I have personally encountered. I don't know serde well, but I imagine it supports use cases I haven't mentioned here.

3 Likes

I strongly agree with this. Rather than adding piecemeal enhancements like this pitch, there should be a fundamental reevaluation of Codable's capabilities now that we're 3 years on from its introduction.

5 Likes
Terms of Service

Privacy Policy

Cookie Policy