Codable Improvements and Refinements

codable

(David Hart) #19

I would love to start discussing the first step of this plan, Custom Attributes, but the big unknown for me is what reflection API those attributes would be available on. Mirror always felt a bit hacky to me so I think that the real initial step is to design new reflection APIs to be able to fit attributes on.

Thoughts?


(Matthew Johnson) #20

I agree that a new reflection API should come before attributes are exposed via reflection. But I'm not really interested in reflection and would hate to see a design of a new reflection API hold up work on custom attributes and macros. IMO that's an orthogonal use of attributes and shouldn't be considered a prerequisite. On the other hand, attributes aren't that useful without a way to code against them so it does seem like something else should proceed in parallel.


(David Hart) #21

Wouldn't the initial use of attributes be runtime access to them? That seems like the logical, easy, first step.


(Matthew Johnson) #22

I’m not the right person to answer that question, but the need to redesign reflection seems to be a potential barrier to that being “the logical, easy first step”. Personally, I would prefer to see a culture develop in Swift that uses attributes more commonly at compile time (via macros and other static meta programming capabilities) than one that uses them at runtime via reflection.


(David Hart) #23

I see what you mean, but a macro system to support compile-time synthesis seems to be a much higher hurdle than a redesign of the reflection APIs.

Me too: that seems more in line with the language as a whole. But if the easiest first step is to get runtime access to attributes before the macro-system to do compile-time synthesis, I wouldn't mind doing it first.


(Jordan Rose) #24

We could also just make the default CodingKeys fileprivate instead of private.

(signed, a person who was never happy with SE-0025 in the first place, so it's not surprising he would suggest this)


(Gwendal Roué) #25

Yes, please! This is very very sensible, because it does not betray too much the original intent of hiding the coding keys. Small action, but huge efffect!

Edit: it looks like I missed some subtleties. See the next message.


(Itai Ferber) #26

Keep in mind that this does not play nicely with inheritance and visibility from subclasses (and especially, accidental inheritance of a superclass's CodingKeys type).

@gwendal.roue (Reminder about the conversation in Extensions to private nested types (illustrated with CodingKeys)) :slight_smile:


(Matthew Johnson) #27

Non-final classes are the root of all evil! :wink:


(Trevor Elkins) #28

IMO we should move away from CodingKey, not add more features to it. I always saw it as a short-term shim until something else came along, like annotations. I can't think of a single thing CodingKey gains us over an annotation.

Most of the time we only want to tweak one or two properties, but then have to write a huge CodingKey enum or implement the init(from:) initializer. I really think the API would be more enjoyable if we flipped that experience around.


(Matthew Johnson) #29

+1. This is how my Sourcery template works. You just specify a custom coding key using an annotation if necessary. This approach is very declarative and places everything you need to know about a value in one place.


(Gwendal Roué) #30

-1. I mean: not too fast. There is more stuff in a coding key that you can put in an annotation.

Here are typical database requests I can derive from reusable coding keys:

extension Round {
    static func current() -> QueryInterfaceRequest<Round> {
        return Round
            .filter(CodingKeys.finishDate == nil)
            .order(CodingKeys.startDate.desc)
            .limit(1)
    }
}

extension Player {
    static func maxScore() -> QueryInterfaceRequest<Int> {
        return Player.select(max(CodingKeys.score), as: Int.self)
    }
}

It is pretty satisfying to me that absolutely no hack (== nasty language tricks out of reach of beginners) is required for this code to be able to work.


(Trevor Elkins) #31

Doesn't that require having the user maintain their own CodingKeys enum in order to reference the keys? AFAIK you cannot reference a synthesized CodingKey value on a type, though I could be wrong about that. The API looks really slick but to me has a tradeoff of implementing boilerplate code.

Depending on how some other proposals turn out, it's possible a comparable API could be implemented with those features (like what Chris Lattner dreams about :smiley:).

edit: sorry I see you did mention the synthesis issue with an earlier post.


(Nathan Harris) #32

Not to mention the same style of API is possible with KeyPaths, much like one is capable with Vapor's Fluent API.


(Gwendal Roué) #33

Yes. As you rightly inferred from previous posts, this requires:

  • an extension to the CodingKeys enum (and thus, currently, an explicit definition of the enum)
  • database requests to be defined in the same file as the record type because CodingKeys is better defined private.

The explicit definition of the CodingKeys is somewhat fine because the compiler warns when keys are missing or contain typos that prevent the synthesis of encode(to:) or init(from:) (and that's pretty cool - even if there is a danger with optional properties because compiler accepts missing keys for them).

Complete code may look like:

Player.swift
// The struct
struct Player: Codable {
    var name: String
    var score: Int
}

// Database support:
// - can decode database rows
// - can generate SQL requests
// - can write in the database
// - can use CodingKeys as database columns
extension Player: FetchableRecord, TableRecord, PersistableRecord {
    private enum CodingKeys: String, CodingKey, ColumnExpression {
        case name, score
    }
}

// Database helpers
extension Player {
    /// Request for the best player. Usage:
    ///
    ///    // Player?
    ///    let bestPlayer = try database.read(Player.best().fetchOne)
    static func best() -> QueryInterfaceRequest<Player> {
        // Generates SELECT * FROM player ORDER BY score DESC LIMIT 1
        return Player.order(CodingKeys.score.desc).limit(1)
    }

    ...
}

(Daniel Höpfl) #34

I’m a huge fan of SE-0030 for this kind of language extension.

But I think SE-0030 needs to be extended:

  • Allow arguments to behaviors (shouldn’t be a problem, init is already in SE-0030):

    var [CodableKey("last_seen")] lastSeen: Date
    
  • Allow having more than one behavior (needs some evaluation, might result in some kind of multi-inheritance):

    var [CodableKey("last_seen"),
         CodableDateFormat(.iso8601),
         observable] lastSeen: Date
    
  • Some kind of runtime reflection will be useful:

    var format = .deferredToDate
    if let [CodableDateFormat] v = x.lastSeen {
       format = v.format
    }
    

If SE-0030 (and above extensions) were already in place, the improvements could be solved as Library-only change without being limited to Codable improvements.


(Matt Diephouse) #35

If I could change any one thing about Codable, I'd generalize the compiler magic that makes it possible.

Focusing on Decodable for a minute, if you could take a type, get a list of KeyPaths, get the name of each, and construct an instance of the type by providing values for each key path, then:

  1. Deodable could probably be defined in the stdlib instead of the compiler. The rest of what it does—decoding values from formats—is already in the stdlib.

  2. The community could experiment with wrappers inside init(from: Decoder) that help with the pain points experienced here. It'd be possible to add a way to get what's currently the default synthesized init and override the defaults of some properties. I think this experimentation could lead to better ideas for the stdlib.

  3. People could explore other solutions for problems that Codable isn't a great fit for. Maybe decoding JSON APIs would be better served by an API that was less focused on roundtripping values, e.g.

Encodable could have a similar treatment.


One small change I think would help would be to add return type inference to the decode methods:

func decode<T>(
  _ type: T.Type = T.self, // Add this default
  forKey key: KeyedDecodingContainer<K>.Key
) throws -> T where T : Decodable

I understand the limitations and downsides of return type inference generally, but 99% of the time when I write a decodable implementation, I'm either assigning to a property or passing a value to an init. A default would remove a lot of tedium.


(Chris Lattner) #36

+1, but please start a new thread so others who are interested in the topic have a chance to notice it, thanks!


(Chris Lattner) #37

Just to respond to a few things upthread:

  1. You're right, there would have to be a step three, actually applying these by defining attributes to customize codable, equatable, hashable, etc...

  2. It isn't clear to me what the right design and scope for user defined attributes are. I'm not a strong advocate for unconditionally reflecting attributes into the runtime reflection metadata, but that is done in other systems and seems to work for them. Even if that were the model, we wouldn't be blocked by providing an actual new runtime reflection API (though we really need one :-)

  3. The default implementation could either be dynamic or static, and if it is static, we have the choice of baking it into the compiler (like today, but listening to new attributes) or making it a macro system that "statically reflects" over the attributes and properties of a type (e.g. have a #for loop that can iterate over lists of declaration properties, which stamps out code). I'm a little nervous about an actual dynamic reflection based system, out of concern that it would produce brutally slow code for important things like equatable/hashable but perhaps that's premature optimization. It would definitely be easier to design this than a static reflection based system.

  4. each of these points have lots of interesting tradeoffs, and it would be great to pull in other folks who are interested in these sorts of topics by starting threads dedicated to them, x-referencing this thread for context.


(David Hart) #38

We could start with a dynamic solution (ease of use for developers), and wait until we have the macro system to use it statically and only then rewrite Hashable, Equatable, and Codable using them.