Introspection of KeyPaths

@Pampel You might want to look at KeyPath-to-String conversion performed by GitHub - vapor/core: 🌎 Utility package containing tools for byte manipulation, Codable, OS APIs, and debugging.. See for example https://github.com/vapor/core/blob/master/Tests/CoreTests/ReflectableTests.swift

Now, personally, as the author of GRDB (an SQLite database library that quite a few users love to use), I have never wanted to provide KeyPath-to-column automatic conversion.

The reason is that I think the intimate details of the relationship between a record type and the database should remain private. Column names are such details. The fact that the synthesized CodingKeys are private as well gives a good precedent.

By not relying on key paths, you can encapsulate your record exactly how you need it.

For example, the record below hides its latitude/longitude. There is no available key path which can talk to those database columns.

struct Place: Codable {
    var id: UUID
    var title: String
    private var latitude: CLLocationDegrees
    private var longitude: CLLocationDegrees
    var coordinate: CLLocationCoordinate2D {
        get { ... }
        set { ... }
    }
}

This ability to cleanly distinguish the private inner details and the internal/public facet of your records, the ability to refactor and migrate your database with minimal-to-zero impact on clients of those types, the ability to have complex records behave just the same as simple trivial ones, those are advantages that would be instantly ruined if key paths were publicly fostered as column proxies.

When such guts are unfortunately exposed, and when you realize that you really need to hide those guts because they are impractical to work with, you have to build a second layer of models, that wrap the first ones. Not only is this second layer a chore to build, but it is likely that not all of your records need one. You end up with an inconsistent database facade, with a mix of low-level and high-level types, without any clear reason why this mess has started. As a matter of fact, it's pretty clear: that's because of the KeyPath-to-string "convenience" conversion :sweat_smile:

7 Likes

Now of course I wonder if Fluent users would confirm this prediction. I would enjoy a reality check :-)

1 Like

This is certainly possible right now with KeyPath’s current internals. It would require some more thought as to what the path component type looks like from an API point of view, but things like component type and name are all there (you’ll have to piece together offsets, but doable).

1 Like

IIRC, the only time I miss such capability is when I have to works with Obj-C API that must take a string Key Path and don't have KeyPath based equivalent.

1 Like

I totally agree with you in principle, although I have actually worked on projects that used ORMs with fluent configuration that do exactly this and found them an absolute pleasure to work with - there are plenty of areas where code driving a schema and queries against it is totally fine, and in those cases, this might be a good fit. Personally, I'd be cautious about using it for a major client project, but I'd love to be able to use it to spike out PoCs for personal stuff.

Using KeyPaths in the configuration of database adjacent code can really help, if not to generate the schema, just to validate that your code is compatible with it, e.g..

struct PersonMappingConfiguration {
    func map() {
        mapper.hasMany(\.pets)
        mapper.makeUnique(\.nickName)
        mapper.useColumnName("firstName", forProperty: \.name).makeReadonly()
    }
}

There's all sorts of opportunities in there to use the type system to both validate the schema, isolate the 'front end' of the type versus it's db representation, and prevent clients of your api doing bad things. Again, not always the right thing, but a great tool for where you want something similar, and one that is proven to work.

But, this isn't just about SQL.

I've used similar features for building and validation configuration, generating documents, improving testing and debugging, providing validation of user input and other things I can't remember. It's not a tool I use often, but when I need something similar, it's the perfect thing for the job.

3 Likes

Here's a use case for KeyPath introspection with CRDT: "Query into dynamic data using static key paths"

1 Like

Thanks for your answer, @Pampel. Yes, Fluent users are generally delighted. Maybe the trouble I envision does not bother them. Or maybe many servers mainly perform CRUD operations, and don't use the database models much, avoiding the need to hide database details.

1 Like

For sure, there are plenty of circumstances where this use case won't be appropriate. Options are good though, we shouldn't be too judgemental about what people might use this for.

You are right. Now, it's also useful to freely explore the consequences of some practices, emit hypothesis, confront them to oneself's past experience, and experience of others. To this end, those hypothesis have to be expressed. I don't think it was "judgemental" to express that IMHO, key path-to-string conversion can create trouble, while its absence fosters more robust practices.

I even provided a link to an implementation of path-to-string conversion, see how I don't prevent anyone from doing anything :wink:

1 Like

Perhaps 'judgemental' was a little strong!

That said, code based off KeyPath introspection could cause the trouble you mentioned, but it can also rid code of string typing, so it also fosters more robust practices.

Yes. In order to avoid string typing, GRDB fosters relying on the CodingKeys generated by Codable synthesis. They have a built-in stringValue property, and don't require much fuss.

I've added some more examples to the body of this discussion.

Thanks! You may be interested in SQL Interpolation and Record Protocols, where Swift string interpolation is put to good use.

extension Player {
    static func maximumScore() -> SQLRequest<Int> {
        "SELECT MAX(\(CodingKeys.score)) FROM \(self)"
    }
}

let score = try Player.maximumScore().fetchOne(db) // Int?

My favorite potential use-case for key path introspection is serialization of key paths as JSON References. Only works given certain structural assumptions but in the context of a well-known schema (like, oh, I don’t know, OpenAPI, as a random example with no personal significance) it could be really nice.

1 Like

I did an implementation:

2 Likes

While building a CRDT (see, I considered writing a similar proposal. During resarch I came up with the following questions.

  1. Computed members.
    Should one be able to introspect a key path and get information if the introspected member is computed?
    Should the compiler give us a way to reference key paths, which must not contain computed members

  2. Subscripts
    While members could be represented a strings, how should we represent subscript?
    As far as I know, a subscript can be called with any type.
    Is a subscript similar to a computed member or not.

  3. Optionals
    Should one be able to get information if a member is optional.

I don’t see why that’s important am I missing something? Couldn’t we just write:

struct Foo { var bar: Int { 5 } }

let name = \Foo.bar.nameComponents.last!

print(name) // bar

Note: I don’t know how the proposed syntax API would be, so used a nameComponents array as it seems kind of convenient

I think optionals should be treated just like any other type. If we started making exceptions it would be hard to maintain.

I think that the String “[0]” would be just fine for a subscript, although it’d be interesting to explore other ways.

The trouble is that not every subscript takes just an Int. Subscripts can take multiple parameters, they can have argument labels, and the parameter types can be anything, possibly not conforming to CustomStringConvertible.

4 Likes

The keyPath to string conversion in core isn't extremely helpful, as you either have to provide the mapping yourself (which defeats the purpose), or depend on the object in question implementing Decodable as well. It's kind of a hack

This is a use case for which I have been searching around for a way to convert KeyPaths to their path string components.

There are two main features described below:
1: Piecemeal Construction:
With a typical initializer (T.init), all parameters must be known at the call site. In a situation where the parameters are determined through a series of steps, the set of parameters must be passed through these steps in some user defined structure (P) that is specific to the type being initialized. At the end, the parameter set is unpacked and supplied, again via some user specified mapping (M), to the initializer.

P | foo | bar | baz | M | T.init

In my case, I want to generalize that parameter-set structure such that it can be used for any type.
Furthermore, I want to rely on the existing Codable implementation (especially the synthesized implementations) to facilitate the mapping (M) between the parameter set (P) and the object.

2: Transformation between similar types
Given two distinct types A and B, transform A into B provided that the members of B are a subset of the members of A.

Required Infrastructure

import AnyCodable // https://github.com/Flight-School/AnyCodable.git

struct ConstructionModel<Model> {
    var data: [String: AnyCodable] = [:]

    subscript<Value>(_ keyPath: KeyPath<Model, Value>) -> Value? {
        get { data[String(describing: keyPath)]?.value as? Value }
        set { data[String(describing: keyPath)] = AnyCodable(newValue) }
    }

    func get<Value>(_ key: String, as: Value.Type) -> Value? {
        data[key]?.value as? Value
    }
}

Example: Piecemeal Construction

struct BigThing: Codable {
    var name: String
    var age: Int
    var quest: String
}

var model = ConstructionModel<BigThing>()
// Elsewhere...
model[\.name] = "Arthur"
// Elsewhere...
model[\.age] = 42
// Elsewhere...
model[\.quest] = "The Holy Grail"

Assuming the ability to recover string path names from a KeyPath, the following would then be possible:

try BigThing(from: CustomDecoder(model: model))

The CustomDecoder used would simply leverage the String + Type provided for each field during decoding and use those parameters to look up the appropriate value with ConstructionModel.get(_:as:).

Example: Transform between similar types

struct ThingOne: Codable {
    var name: String
}

struct ThingTwo: Codable {
    var name: String
}

let transformer = Transformer()
ThingOne(name: "One").encode(to: transformer)
try ThingTwo(from: transformer)

Here the Transformer conforms to both Encoder and Decoder such that it consumes the input object into an internal ConstructionModel (as shown above) and then references that ConstructionModel to create the specified output (if possible).

Compound Use Case
Firstly, I realize that the type transformation feature is possible using only Codable. However, I mention it here as context for a bit more detail on another part of my use case:

  • Given two types A and B, map A.x to B.y (generically) provided that A.x and B.y are of the same type.

In practice the described function signature looks something like this:

func map<A, B>(_ lhs: KeyPath<A, V>, to rhs: KeyPath<B, V>)

The important part for me here is the compile time guarantees of the KeyPaths. I know that I am mapping two fields that could plausibly be mapped to one another.

Final Notes
Anyway, there's a bunch of other code I have built up around what I've described here. The purpose for that code being validation and construction using a defined "shape".
That is to say...

  • Given a definition of a "shape" S, attempt to transform an instance of type T into S provided that T passes a set of validation constraints defined by S. Also, mapping any important pieces of T onto S.

And the opposite direction:

  • Given an instance of a "shape" S, attempt to construct an instance of type T such that T can be again transformed into S using the aforementioned "shape" definition.
1 Like