Codable Improvements and Refinements

[hartbit (David Hart)]

@anandabits

IMO that's an orthogonal use of attributes and shouldn't be considered a prerequisite. On the other hand, attributes aren't that useful without a way to code against them so it does seem like something else should proceed in parallel.

Wouldn't the initial use of attributes be runtime access to them? That seems like the logical, easy, first step.

[anandabits (Matthew Johnson)]

@hartbit

Wouldn't the initial use of attributes be runtime access to them? That seems like the logical, easy, first step.

I’m not the right person to answer that question, but the need to redesign reflection seems to be a potential barrier to that being “the logical, easy first step”. Personally, I would prefer to see a culture develop in Swift that uses attributes more commonly at compile time (via macros and other static meta programming capabilities) than one that uses them at runtime via reflection.

[hartbit (David Hart)]

@anandabits

I’m not the right person to answer that question, but the need to redesign reflection seems to be a potential barrier to that being “the logical, easy first step”.

I see what you mean, but a macro system to support compile-time synthesis seems to be a much higher hurdle than a redesign of the reflection APIs.

@anandabits

Personally, I would prefer to see a culture develop in Swift that uses attributes more commonly at compile time

Me too: that seems more in line with the language as a whole. But if the easiest first step is to get runtime access to attributes before the macro-system to do compile-time synthesis, I wouldn't mind doing it first.

[jrose (Jordan Rose)]

@gwendal.roue

I, too, wish that one could extend a type's CodingKeys enum. Even the private synthesized one.

We could also just make the default CodingKeys fileprivate instead of private.

(signed, a person who was never happy with SE-0025 in the first place, so it's not surprising he would suggest this)

[gwendal.roue (Gwendal Roué)]

@jrose

We could also just make the default CodingKeys fileprivate instead of private .

Yes, please! This is very very sensible, because it does not betray too much the original intent of hiding the coding keys. Small action, but huge efffect!

Edit: it looks like I missed some subtleties. See the next message.

[itaiferber]

Keep in mind that this does not play nicely with inheritance and visibility from subclasses (and especially, accidental inheritance of a superclass's CodingKeys type).

@gwendal.roue (Reminder about the conversation in Extensions to private nested types (illustrated with CodingKeys))

[anandabits (Matthew Johnson)]

@itaiferber

Keep in mind that this does not play nicely with inheritance and visibility from subclasses

Non-final classes are the root of all evil!

[rovertsnikle (Trevor Elkins)]

IMO we should move away from CodingKey, not add more features to it. I always saw it as a short-term shim until something else came along, like annotations. I can't think of a single thing CodingKey gains us over an annotation.

Most of the time we only want to tweak one or two properties, but then have to write a huge CodingKey enum or implement the init(from:) initializer. I really think the API would be more enjoyable if we flipped that experience around.

[anandabits (Matthew Johnson)]

@rovertsnikle

IMO we should move away from CodingKey , not add more features to it. I always saw it as a short-term shim until something else came along, like annotations. I can't think of a single thing CodingKey gains us over an annotation.

+1. This is how my Sourcery template works. You just specify a custom coding key using an annotation if necessary. This approach is very declarative and places everything you need to know about a value in one place.

[gwendal.roue (Gwendal Roué)]

@rovertsnikle

IMO we should move away from CodingKey , not add more features to it. I always saw it as a short-term shim until something else came along, like annotations. I can't think of a single thing CodingKey gains us over an annotation.

-1. I mean: not too fast. There is more stuff in a coding key that you can put in an annotation.

Here are typical database requests I can derive from reusable coding keys:

extension Round {
    static func current() -> QueryInterfaceRequest<Round> {
        return Round
            .filter(CodingKeys.finishDate == nil)
            .order(CodingKeys.startDate.desc)
            .limit(1)
    }
}

extension Player {
    static func maxScore() -> QueryInterfaceRequest<Int> {
        return Player.select(max(CodingKeys.score), as: Int.self)
    }
}

It is pretty satisfying to me that absolutely no hack (== nasty language tricks out of reach of beginners) is required for this code to be able to work.

[rovertsnikle (Trevor Elkins)]

Doesn't that require having the user maintain their own CodingKeys enum in order to reference the keys? AFAIK you cannot reference a synthesized CodingKey value on a type, though I could be wrong about that. The API looks really slick but to me has a tradeoff of implementing boilerplate code.

Depending on how some other proposals turn out, it's possible a comparable API could be implemented with those features (like what Chris Lattner dreams about ).

edit: sorry I see you did mention the synthesis issue with an earlier post.

[Mordil (Nathan Harris)]

Not to mention the same style of API is possible with KeyPaths, much like one is capable with Vapor's Fluent API.

[gwendal.roue (Gwendal Roué)]

@rovertsnikle

Doesn't that require having the user maintain their own CodingKeys enum in order to reference the keys?

Yes. As you rightly inferred from previous posts, this requires:

• an extension to the CodingKeys enum (and thus, currently, an explicit definition of the enum)
• database requests to be defined in the same file as the record type because CodingKeys is better defined private.

The explicit definition of the CodingKeys is somewhat fine because the compiler warns when keys are missing or contain typos that prevent the synthesis of encode(to:) or init(from:) (and that's pretty cool - even if there is a danger with optional properties because compiler accepts missing keys for them).

Complete code may look like:

Player.swift

// The struct
struct Player: Codable {
    var name: String
    var score: Int
}

// Database support:
// - can decode database rows
// - can generate SQL requests
// - can write in the database
// - can use CodingKeys as database columns
extension Player: FetchableRecord, TableRecord, PersistableRecord {
    private enum CodingKeys: String, CodingKey, ColumnExpression {
        case name, score
    }
}

// Database helpers
extension Player {
    /// Request for the best player. Usage:
    ///
    ///    // Player?
    ///    let bestPlayer = try database.read(Player.best().fetchOne)
    static func best() -> QueryInterfaceRequest<Player> {
        // Generates SELECT * FROM player ORDER BY score DESC LIMIT 1
        return Player.order(CodingKeys.score.desc).limit(1)
    }

    ...
}

[dhoepfl (Daniel Höpfl)]

@itaiferber

Compiler-heavy solution: some form of property-level annotations that indicate how to synthesize init(from:) and encode(to:) by adjusting how the property is assigned to during encoding/decoding. This could take the form of something similar to SE-0030 Property Behaviors (which was discussed but deferred), or actual user-level annotations (e.g. @codableAdaptor(...) var myProperty: Foo ). This subject has been discussed at length, but I'd like to note that it's unlikely we'd want to implement a feature like this just for Codable, so this direction would likely require revisiting this topic and re-generalizing it

I’m a huge fan of SE-0030 for this kind of language extension.

But I think SE-0030 needs to be extended:

• Allow arguments to behaviors (shouldn’t be a problem, init is already in SE-0030):

  var [CodableKey("last_seen")] lastSeen: Date
  

• Allow having more than one behavior (needs some evaluation, might result in some kind of multi-inheritance):

  var [CodableKey("last_seen"),
       CodableDateFormat(.iso8601),
       observable] lastSeen: Date
  

• Some kind of runtime reflection will be useful:

  var format = .deferredToDate
  if let [CodableDateFormat] v = x.lastSeen {
     format = v.format
  }
  

If SE-0030 (and above extensions) were already in place, the improvements could be solved as Library-only change without being limited to Codable improvements.

[mdiep (Matt Diephouse)]

If I could change any one thing about Codable, I'd generalize the compiler magic that makes it possible.

Focusing on Decodable for a minute, if you could take a type, get a list of KeyPaths, get the name of each, and construct an instance of the type by providing values for each key path, then:

1. Deodable could probably be defined in the stdlib instead of the compiler. The rest of what it does—decoding values from formats—is already in the stdlib.

2. The community could experiment with wrappers inside init(from: Decoder) that help with the pain points experienced here. It'd be possible to add a way to get what's currently the default synthesized init and override the defaults of some properties. I think this experimentation could lead to better ideas for the stdlib.

3. People could explore other solutions for problems that Codable isn't a great fit for. Maybe decoding JSON APIs would be better served by an API that was less focused on roundtripping values, e.g.

Encodable could have a similar treatment.

---

One small change I think would help would be to add return type inference to the decode methods:

func decode<T>(
  _ type: T.Type = T.self, // Add this default
  forKey key: KeyedDecodingContainer<K>.Key
) throws -> T where T : Decodable

I understand the limitations and downsides of return type inference generally, but 99% of the time when I write a decodable implementation, I'm either assigning to a property or passing a value to an init. A default would remove a lot of tedium.

[Chris_Lattner3 (Chris Lattner)]

@hartbit

I would love to start discussing the first step of this plan, Custom Attributes, but the big unknown for me is what reflection API those attributes would be available on. Mirror always felt a bit hacky to me so I think that the real initial step is to design new reflection APIs to be able to fit attributes on.

Thoughts?

+1, but please start a new thread so others who are interested in the topic have a chance to notice it, thanks!

[Chris_Lattner3 (Chris Lattner)]

Just to respond to a few things upthread:

1. You're right, there would have to be a step three, actually applying these by defining attributes to customize codable, equatable, hashable, etc...

2. It isn't clear to me what the right design and scope for user defined attributes are. I'm not a strong advocate for unconditionally reflecting attributes into the runtime reflection metadata, but that is done in other systems and seems to work for them. Even if that were the model, we wouldn't be blocked by providing an actual new runtime reflection API (though we really need one :-)

3. The default implementation could either be dynamic or static, and if it is static, we have the choice of baking it into the compiler (like today, but listening to new attributes) or making it a macro system that "statically reflects" over the attributes and properties of a type (e.g. have a #for loop that can iterate over lists of declaration properties, which stamps out code). I'm a little nervous about an actual dynamic reflection based system, out of concern that it would produce brutally slow code for important things like equatable/hashable but perhaps that's premature optimization. It would definitely be easier to design this than a static reflection based system.

4. each of these points have lots of interesting tradeoffs, and it would be great to pull in other folks who are interested in these sorts of topics by starting threads dedicated to them, x-referencing this thread for context.

[hartbit (David Hart)]

@Chris_Lattner3

The default implementation could either be dynamic or static, and if it is static, we have the choice of baking it into the compiler (like today, but listening to new attributes) or making it a macro system that "statically reflects" over the attributes and properties of a type (e.g. have a #for loop that can iterate over lists of declaration properties, which stamps out code). I'm a little nervous about an actual dynamic reflection based system, out of concern that it would produce brutally slow code for important things like equatable/hashable but perhaps that's premature optimization. It would definitely be easier to design this than a static reflection based system.

We could start with a dynamic solution (ease of use for developers), and wait until we have the macro system to use it statically and only then rewrite Hashable, Equatable, and Codable using them.

[zwaldowski (Zachary Waldowski)]

It's unfortunate how quickly we jumped into the dream syntax we'd like to see in the far-off space future, because I'm really interested in making refinements to what we already have.

Two notes around the desire for something annotation-heavy or reflection-heavy:

1. Regardless of whether it brings you joy or not, the type-safe approach of Codable today is highly desirable for making incorrect programs clearly look incorrect and usually not compile. I would find moving to something unnecessarily based in dynamism and typo-able annotations to be a severe regression. Even if the hypothetical macros were to lean heavily on constexpr, it's still be switching away from something that is verified at compile-time to something that isn't.
2. The discoverability and maintenance stories for "I just want to customize one behavior!" is really under-defined (nb: my one is always different from your one). Having "synthesize everything" vs. "implement everything yourself" as the two options is extremely clear and easy to teach. The current behavior inarguably fulfills the goal of having the simplest case be the easiest and the most complex cases be possible. I have no idea how someone would get familiar with annotation customization points other than just dumping a big list of syntax on them and hoping they figure it out.

Separately, I don't understand the overriding need to kill CodingKeys. The benefit of some yet-unscoped syntax involving magic and key-paths is unclear, except for it involving a construct that is newer to the language. The language already has a feature for defining enumerated lists of things. Similarly, we don't need bespoke syntax for defining an error domain just to write 20 characters less. Moreover, the keys pulled out of an archive are often orthogonal to the properties of a type.

[nuclearace (Erik Little)]

@zwaldowski

It's unfortunate how quickly we jumped into the dream syntax we'd like to see in the far-off space future, because I'm really interested in making refinements to what we already have.

IMO, I think it's important to discuss what a "perfect" or "natural" solution to this would be. Most of the alternatives I've seen feel like a bolt on, and still filled with annoying boilerplate.

The end goal of this compiler-heavy solution is to move as much of the special built-in synthesis out of the compiler and into the standard library. I would much rather spend time building a new system that simplifies the existing solution, rather than keep bolting onto the existing one.

@zwaldowski

Regardless of whether it brings you joy or not, the type-safe approach of Codable today is highly desirable for making incorrect programs clearly look incorrect and usually not compile. I would find moving to something unnecessarily based in dynamism and typo-able annotations

These would still be done at compile time, afaiu. The annotations would be available inside of the macro, so that it could do manipulation on them, and generate the default implementation. You're not losing any type-safety with this solution.

@zwaldowski

I have no idea how someone would get familiar with annotation customization points other than just dumping a big list of syntax on them and hoping they figure it out.

We're not going to change the fact that users should be consulting the documentation for Codable. A sufficiently comprehensive guide to Codable should provide all the needed documentation to use an annotation based solution.