KeyPaths and Codable

Evolution Discussion
1

Swift 4's key paths provide a great way for referencing a type's properties in a Swifty way. Since Swift 4's release, many ORMs have adopted key paths in lieu of strings for doing query building. Below are a couple examples from real ORMs built on Swift 4.

Fluent (SQL / NoSQL)

Planet.query(on: conn).filter(\.type == .smallRocky).all()

Perfect CRUD (SQL)

personTable.order(by: \.lastName, \.firstName)

Kuery (Core Data)

Query(Person.self).filter(\Person.age > 20)

Advantages

KeyPaths offer a couple key advantages over Strings:

  • Type-safe: The compiler helps guarantee you've spelled the key path correctly.
  • Auto-completable: If the force is strong with you, Xcode's autocomplete can help you fill in key paths automatically.
  • Strong type info: Key paths know the type of their value allowing you to use leading dot syntax.

Comparing Fluent version 2 (pre Swift 4) to 3 helps highlight this:

// Fluent 2
Planet.makeQuery().filter("type" == PlanetType.smallRocky).all()
// Fluent 3
Planet.query(on: conn).filter(\.type == .smallRocky)

The Problem

The problem with using key paths in this manner is that the relation between key paths and codable is not entirely clear. Let's take the Fluent query for example:

Planet.query(on: conn).filter(\.type == .smallRocky)

When used with a SQL database, this query should result in something along the lines of:

SELECT * FROM "planets" WHERE "type" = ? ["smallRocky"]

While this seems pretty straight forward, the conversion from \.type to "type" is actually quite complex.

Take the Planet model for example:

struct Planet: Codable {
    var name: String
    var type: PlanetType
}

Here it's pretty clear that \.type should yield the string "type". But what happens if we were to define a custom CodingKeys enum for our type?

struct Planet: Codable {
    enum CodingKeys: String, CodingKey {
        case name = "name"
        case type = "planet_type"
    }
    var name: String
    var type: PlanetType
}

Given this Planet struct, the aforementioned SQL query should now be:

SELECT * FROM "planets" WHERE "planet_type" = ? ["smallRocky"]

This is an important distinction because it means that: we need a way to relate a key path into a given Codable type to its coding keys.

Potential API

Ideally, we could use a key path to query a codable model for the associated coding key. Spelling aside, the API could look something like this.

struct Planet: Codable {
    enum CodingKeys: String, CodingKey {
        case name = "name"
        case type = "planet_type"
    }
    var name: String
    var type: PlanetType
}

let key: CodingKey? = Planet.codingKey(for: \.type)
print(key?.stringValue) // Optional("planet_type")

Current Solution

The Swift ORMs I mentioned earlier have some unique (some might call hacky) ways for handling this. Fluent and Perfect CRUD use specialized decoders + some fancy tricks to determine which [CodingKey] path is related to a given KeyPath. Learn more here.

The CoreData based Kuery ORM is able to use the Objective-C bridged _kvcKeyPathString value to get the string value for a key path. As far as I know, this is not affected by codable.

This issue has been touched on a bit before (see SR-5220) but I have yet to see anyone discuss key paths in relation to codable specifically.

Discussion

I would love to get the Core Team's opinion on whether this usage of key paths is something that Swift might support more formally in the future or if it is somewhat of an unintended side effect.

20 Likes
2

This has been brought up a few times in the past, and I’d be happy to explore the relationships between KeyPaths and CodingKeys further. I don’t myself have any good answers at the moment, but the main difficulties that I see in this area are primarily related to the fact that CodingKeys do not need to map 1-to-1 to properties, unlike KeyPaths.

If you look primarily at synthesized Codable implementations one might assume that CodingKeys map well to properties, but keep in mind that:

  1. A given property need not have an associated CodingKey — for Encodable types, any property may be left out of the CodingKeys enum if one does not wish it to be encoded; similarly, for Decodable types, properties with default values may be left out of the CodingKeys enum if one does not wish for them to be decoded
  2. The further one diverges from the synthesized implementation, the more this mapping falls apart. If you implement encode(to:) and init(from:) directly, there is no requirement that CodingKeys map in any way to properties — CodingKeys can be completely made up, have different names, or simply not exist. Types may choose to encode data in nested containers under multiple different CodingKey types, which totally breaks any sort of mapping
  3. A type need not even define its own CodingKeys enum or encode with any keys — types can encode into unkeyed and single-value containers just fine

Given this, how would we reconcile these two different concepts? Would we only support types which do offer a 1-to-1 mapping? (Note that there isn’t a way to tell dynamically whether this mapping exists, and if a type overrides encode(to:) or init(from:), there’s little we can statically tell about how a CodingKey type is used, too.)

[FWIW, I wouldn’t necessarily call this an unintended side-effect — I think these two features have a lot of overlap which may cause them to appear superficially related in many ways, but in fact, their goals can be quite distinct, in a potentially irreconcilable way.]

1 Like
4

You bring up a lot of good points.

If a property does not have a match in the CodingKeys, then attempting to resolve the key path to a coding key should fail (return nil or otherwise). That's expected since the key will not appear in the encoded data.

In the case of custom encode / decode methods, the model could potentially be required to define manually how key paths relate to coding keys.

Another approach to the problem of using key paths in ORMs would be allowing developers to get the string representation of the key path (what _kvcKeyPathString does).

Given the complexity around relating properties to encoded data, I think it would be a fair tradeoff to say that types with non-standard coding key to key path naming conventions would need to describe these relations manually. (i.e., planet.type encoding to key "planet_type" instead of "type" would need to declare this to the ORM). In that case, ORMs could utilize the ability to "reflect" the key path's string name as a reasonable default as long as a method is provided for the type to define its own conversions.

5

I recently discovered GitHub - Kitura/TypeDecoder: A Swift library to allow the runtime inspection of Swift language native and complex types. which also attempts to solve this problem.

/cc @Kyle_Jessup @IanPartridge

6

In the long term, I think this would be interesting to explore. In the fullness of time, I'd like to see a more "protocol-oriented" and extensible approach to key paths, which could allow for CodingKeyPath to be a refinement of KeyPath with the added requirements of coding, such as being able to map to a key string/integer value, and be usable as part of initialization from a coder.

11 Likes
7

Do you think something similar to _kvcKeyPathString could be considered in the shorter term? i.e., Swift 5.0 or 5.1?

Basically something like this:

let key = \Planet.type
print(key.pathString) // Optional("type")

Or perhaps the path would be an array, to account for nesting:

let key = \Planet.type.description
print(key.pathStrings) // Optional(["type", "description"])

I'd be happy to submit a proposal for this if so.

That would be really great! I'm also happy to help work on proposals for this system as a longer-term solution.

1 Like
8

Extending reflection support beyond what we have now is probably out of scope for Swift 5.

9

Since a better solution to this problem is out of scope until at least Swift 6, I've worked a bit to improve Vapor's Codable reflection algorithm and I will be moving it to a separate package. Anyone needing the ability to reflect types or convert key paths to strings should take a look at this and consider contributing!

https://github.com/vapor-community/codable-kit/pull/1

7 Likes
10

Well, I just want to weigh in that this would be a formidable feature.

Is there anything new in sight to improve reflection support?

1 Like
11

A bit late to the party but I've found that adding a new protocol conformance to my CodingKeys can help:

public protocol InitializableByKeyPath {
    associatedtype Root
    init<Value>(keyPath: KeyPath<Root, Value>) throws
}

Passing the object's type through to the KeyedDecodingContainer and KeyedEncodingContainer eventually allowed me to write :

extension Planet: Codable {
    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        name = try container.decode(forKeyPath: \.name)
        type = try container.decode(forKeyPath: \.type)
    }

    func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)
        try container.encode(self, forKeyPath: \.name)
        try container.encode(self, forKeyPath: \.type)
    }
}

I did however find it very hard to go from CodingKey to KeyPath due to the lack of type information available, but it is possible when encoding (maybe).

I do think there is a very common case where there is a 1-to-1 mapping: that of compiler generated CodingKeys. Supporting InitializableByKeyPath (and CaseIterable) in this specific case, along with making the generated CodingKeys available would be low cost and allow for much cleaner custom init(from:) and encode(to:) implementations.

1 Like