"New Codable" prototype available for feedback

Evolution Discussion
,
61

Don't necessarily count on that sticking around. It's cute, but definitely out of the norm for API styles. It was inspired by both Serde (which uses strings instead of identifiers in the macros anyway) and MetaCodable. The latter gets even more clever by using "Full Width Hyphen-Minus" for the kebab cases. I didn't go that far. :sweat_smile:.

Addressed this a bit above, but allowing custom case algorithms would necessitate a completely different design for how these work. The case conversion happens entirely in the macro and establishes a single (default) encoded string that matches to the given property. Custom code can't run in the macro, so we'd have to do runtime string conversion instead. Maybe I'm being naive, and it'd be fine, but it feels like that could open some subtle behavior differences, and would definitely have some impact on performance.

It also exacerbates @taylorswift's point, I think. A custom algorithm could do who-knows-what with the string. It could lead to even worse confusion for a reader than something that is explicitly defined and locked in at compile time—something you could inspect directly by looking at the macro expansion.

I think there's always opportunity to add more algorithms to the ones the macro supports though, if there's strong support for it.

1 Like
62

If it's not possible to have this:

struct BlogPost {
    @CodingDefault([])
    var tags: [String]
}

expressed more cleanly as:

struct BlogPost {
    var tags: [String] = []
}

(which in itself raises a question if the tool being used – macros – is an appropriate choice to make the most ergonomic API) - I'd consider at least providing a decoding API that allows providing default "template" object to fill the absent fields:

decoder.decode(default: value, from: data) // type is derived from default value

which has an added benefit of an ability to provide a different default value per use case compared to hardcoding that choice in the BlogPost type.

As for:

struct BlogPost {
    @CodingFormat(.iso8601)
    let publishDate: Date
}

I wonder if we should support use cases when date formatting strategy is different for different fields (of the same structure or for different fields belonging to different types of the value tree being encoded / decoded). If not - then date formatting strategy could be supplied as a parameter of the encode/decode call, like it's now done in JsonEncoder / JSONDecoder. Ditto for floating point strategy, data encoding strategy, non conforming floating point numbers encoding strategy, pretty printing / sorting / slash formatting / JSON5 form / etc.

2 Likes
63

Not universally, no. For vars, yes this could work, but lets initialized like this cannot be changed even by an initializer. I'd like to avoid, where possible, a design that forces coding styles like vars everywhere.

Additionally, this syntax encourages a pattern that's notoriously difficult for macros. e.g.

    var x = <arbitraryExpression>

The macro doesn't get to operate on type-checked code—just the AST. With this kind of code, it's unfortunately impossible for the macro to universally determine the type of the expression. We could probably raise an error in the macro if we see this pattern and require an explicit type declaration, but this combined with the previous reason, I currently still prefer to sit on the CodingDefault() side of the fence here.

I'm pretty committed to macros as the tool of choice for code generation in this project. They have some unfortunate limitations, but they're the best tool that we have. Compiler-based code synthesis is really not a sustainable practice.

Furthermore, these macros aren't really "API" per se. Nobody's forced to them. You can also invent your own if you prefer. Or you can code manually to the API—you're just going to end up with a less straightforward time in your manual implementation compared to traditional Codable. This is due, of course, to the performance-focused API design we have as one of the main goals of this project.

We can certainly consider an API like what you propose, but it would primarily be a convenience for the (hopefully few) of those doing manual implementations. The macros implement the same behavior, just inline. It's ultimately a pretty trivial extension anyone could add:

extension CommonDecoder {
  func decode<T: CommonDecodable>(default: @autoclosure () -> T) throws(CodingError.Decoding) -> T {
    return try self.decode(T?.self) ?? default()
}

I'm not quite sure I'm following you in the first part. This macro does let you specify different strategies for different fields by giving each Date field its own @CodingFormat.

Applying formats recursively through the value tree in a generic fashion is something we've discussed elsewhere. It's only possible with dynamic type casting—a bane to both performance and Embedded Swift compatibility. Traditional Codable and JSONDecoder specifically locked itself into these traps and we're trying to avoid that this go around.

64

Only for fields with default values. And even then, I don't think it's quite as significant a drawback as you make it sound.

That's not quite what I meant. The key distinction is that the defaultValue approach I'm describing only fills in missing fields. In your version, a single missing field causes the decoder to discard the entire value and fall back to the default instance.

My point is that this is actually an anti-feature. If I have five different Date fields — whether in the same struct or spread across multiple related types — I will almost certainly want them all to use the same date strategy (and similarly for floating-point formatting, data encoding, etc.).

Repeating the same @CodingFormat(...) annotation on every field quickly becomes noisy and error-prone. These kinds of options are better expressed globally rather than per field. In fact, I'd argue that using different coding strategies for two fields of the same type should generally be discouraged, if not outright prohibited.

I think that risks painting yourself into a different corner. Personally, I'd start from what "users" actually want, then work backward from there: either improve the tooling, extend the macro system if it's insufficient, or move functionality into the compiler when the abstraction fundamentally requires it.

For example, a compiler-based implementation could potentially achieve O(1) code-size complexity while preserving runtime performance comparable to a macro-generated or manually written implementation.

65

in GitHub - andre-tirta/auto-key: A Swift macro that automatically generates CodingKeys enums for Codable structs, classes and actors. · GitHub

you can use KeyAlias macro such as:

@AutoKey
struct Example: Codable {
let imageTiff: String

@KeyAlias("image_TIFF")
let imageTIFF: String

}

does this answer your question?

66

the "advantage" of having codingkey macro like this is: you can map it to entirely another field, if you ever need it.

Such as:

struct SomeStruct: Codable {
  @CodingKey("anotherString") let someString: String
}

that was the consideration I use when creating a simple package AutoKey: GitHub - andre-tirta/auto-key: A Swift macro that automatically generates CodingKeys enums for Codable structs, classes and actors. · GitHub

67

I'm missing the broader vision here then. Could you give a more complete example?

I will point at Rust Serde a lot—not from an angle that new Codable has to be identical to it, but as evidence its designs do work for real projects. It's very clear that it's NOT perfect, and I want to take every opportunity we have to learn from it. But it is a successful design that achieves similar goals that we have here.

Core Serde requires these kinds of annotations on each field. I have indeed seen complaints about this requirement, yes, but I've never heard it called an "anti-feature". That said, other developers have stepped up to try to improve the core macro design, in this case with features like apply in serde_with - Rust. I can look into the possibility of mimicking the same effects with Swift macros attached to the struct. We don't have quite as much freedom as Rust macros though—I don't think we can take attached macro syntax as a direct parameter to another macro, e.g.

@CodableApply(type: Date.self, @CodingFormat(.iso8601))

We also currently don't have recursive application of macros, but we can probably work around this with something a bit more ad hoc like:

@CodableApply(type: Date.self, codingFormat: .iso8601))

This sounds great, but it also seems unrealistic. Based on your previous comments, am I correct in assuming that the example you're comparing this project to is something like cpp-reflect?

From what I understand from projects like this, compile-time-reflection-based solutions are far from a "O(1) code-size" solution. Every template instantiation for every type involved expands into code that handles each field individually. This is actually quite close to the same amount of codegen produced by macro-based solutions as it scales with the number of types and number of fields per type. Typically it's runtime-reflection-based solutions that are able to achieve O(1) code size, but at the cost of CPU throughput.

68

It's merely an alternative to your @CodingDefault... By way of example:

struct BlogPost {
    var required: Int
    var items: [String] = []
}

decode(template: BlogPost(required: 0), from: data) // { "required" : 1 }
// -> BlogPost(required: 1, items: [])

decode(BlogPost.self, from: data) // { "required" : 1, "items": ["hello"] }
// -> BlogPost(required: 1, items: ["hello"])

decode(BlogPost.self, from: data) // { "items": ["hello"] }
// ❌ error, missing field

With an extra flexibility of a possibility to have different "default" values:

// { "required" : 1 }
decode(template: BlogPost(required: 0, items: ["world"]), from: data)
// -> BlogPost(required: 1, items: ["world"])
Just realized I already made similar points upthread – hiding to reduce noise

I don't quite understand, even if you do everything else with macros what stops you from choosing things like date coding strategy, etc more "globally" (e.g. similar to how JSONEncoder / decoder is doing that? Say JSON5 option – would be very awkward to have it varying per field...

I think both the current Codable and Mirror APIs required significant support from both the compiler and standard library. Do you mean that even if such an approach was possible in the early days of Swift, it is no longer realistic going forward?

Imagine a better version of the Mirror API:

  • Fast
  • Not read-only, but read-write
  • Either (A) available without opting in, or (B) only available with opt-in (e.g., via some Mirrorable protocol)

The standard library's Codable implementation could be built on top of that. Compared to a macro-based or hand-tuned manual implementation, that built-in implementation would be almost as fast. It might not even require an additional marker protocol like the Codable we currently have to use.

In terms of out-of-the-box behavior, it would not require additional generated code, so it would be O(1) space-wise. (With option (B), the types in question would need to be marked Mirrorable to be codable and Mirrorable itself would impose some O(n) space overhead in the form of extra metadata per field — similar to what the current Mirror API requires.)

The litmus test question: if we had that ideal Mirror API with the above characteristics today, would we implement new codable infrastructure based on it, or would we prefer a different approach (like macros)? And if it's the former, perhaps we should implement the improved Mirror API first?

69

Apologies for the tangent, but does anyone else wish macros and other attributes could follow simple declarations instead of precede them?

i.e. instead of this:

@KeyAlias("image_TIFF")
let imageTIFF: String

this:

let imageTIFF: String @KeyAlias("image_TIFF")
70

Ah I see! Thanks for elaborating. This is actually quite interesting.

This "template" idea feels like it fills a very similar role to "patching". Given a base object, we update it with any present fields decoded from an archive. Fields that are absent from the JSON are left on the base object unchanged.

In short, yes, I think we could include this kind of functionality. But there are some interesting implications here we must consider. How could we make this work recursively? (Or should we?)

For example:

struct Foo {
  var x: Int
  var bar: Bar
}

struct Bar {
  var y: Int
  var z: Int
}

var template = Foo(x: 1, bar: .init(y: 2, z, 3))

//  { "bar": { "z": 42 } }
let result = try decode(template: template, …)
// 1. Throws an error: "Type 'Bar' missing field 'y'"
// 2. result == Foo(x: 1, bar: .init(y: 2, z, 42)

Looking at prior art in this area, it seems like recursive "patching" is something to be enabled on a per-field basis. (But as discussed for other attributes, could be something that's opted in to ALL fields at once.)

Jackson

The core functionality for "patching" in the Jackson library is ObjectReader.readerForUpdating(…):

Foo template = new Foo(1, new Bar(2, 3));
Foo result = mapper.readerForUpdating(template).readValue(data);

In order for Foo's Bar property to itself be internally patchable, Jackson provides a @JsonMerge attribute:

class Foo {
  int x;
  @JsonMerge
  Bar bar;
}

I believe this suppresses any missing key errors when the nested Bar is decoded and the template value updated directly with any values that are present.

Disclaimer: mimicking Jackson's exact techniques might be difficult, because I believe its entire (de)serialization library is built on runtime reflection, which we have to avoid in Swift.

serde + struct-patch crate

While serde doesn't provide any built-in patching capabilities, in true Rust fashion, there's of course a crate for this.

The struct-patch crate takes a fundamentally different approach than Jackson's. It introduces a Patch trait that gets applied to the base type. It implies "Instances of type Foo can be patched by using its generated patch type FooPatch" where FooPatch by default has all properties of Foo, just optional. You would explicitly decode a FooPatch from JSON first and then .apply() the patch to the template Foo.

To enable recursively patchable values, you annotate the relevant field with #[patch(nesting)] after making sure the nested type itself implements Patch. This changes the base type's derived Patch implementation to use the nested type's own Patch type instead of its original type.

This design is more modular and potentially useful even outside of serde, but it does require a more manual process, as well as more types and a bit more memory usage due to the transient XYZPatch values.

For Swift / NewCodable

My initial impression is that the best choice for Swift here would be to follow a path similar to Rust's. The other option is to tightly integrate it into the XYZDecodable protocols. Instead of

public protocol JSONDecodable: ~Copyable {
  static func decode<D: JSONDecoderProtocol & ~Escapable>(from decoder: inout D) throws(CodingError.Decoding) -> Self
}

… we'd need to have:

public protocol JSONDecodable: ~Copyable {
  static func decode<D: JSONDecoderProtocol & ~Escapable>(from decoder: inout D, defaultValue: Self? = nil) throws(CodingError.Decoding) -> Self
}

And then the macro (or compile-time reflection driven functions, if we had them) would expand to, roughly:

static func decode<D: JSONDecoderProtocol & ~Escapable>(from decoder: inout D, defaultValue: Self? = nil) throws(CodingError.Decoding) -> Self {
  try decoder.decodeStruct { ...
    var prop1: Prop1? = defaultValue?.prop1 ?? nil
    var prop2: Prop2? = defaultValue?.prop2 ?? nil
    // etc.
    // normal field visitor loop, nil property checks, and Self initializer
  }
}

This is certainly not awful and we could do it. I've received many requests for patching support and this could be a nice solution. I do like that it doesn't force us to create a DOM representation or instantiate a transient XYZPatch value. But I think it's worth having a broader discussion about whether a Patchable protocol could be generally useful.

71

Ah, I see what you mean, and "patching" could be interesting in its own right, allowing algebraic operations on values like:

newValue = oldValue + deltaJSON
deltaJSON = newValue - oldValue

That said, my suggestion was more modest: while the current Codable allows optional fields to be absent when decoding, the new Codable could support missing "defaulted" fields (fields with default values). I see a few ways to make this happen:

  1. A plain var field = defaultValue
  • Concise
  • DRY
  • May require compiler support (not sure)
  • Default value is baked into the type (a drawback?)
  1. Something like your @CodingDefault
  • More verbose / noisier
  • Not DRY — in many cases users would supply both a default value and @CodingDefault (the two could even differ!?)
  • Default (@CodingDefault) value is baked into the type (a drawback?)
  1. The template value approach above
  • DRY
  • Somewhat wasteful in terms of CPU usage and extra code size, especially if value creation is expensive: the template might not be used at all if the JSON has every field, and any unused fields are pure overhead
  • Allows supplying custom "defaulted" values for missing fields at the decode site (a feature?)
  • That said, it's best not to apply template values to missing optional fields: the JSON should have the upper hand on what values to use, and a missing optional field (or, equivalently, an explicit null) may well be intentional
1 Like
72

I will reconsider making the macro parsing out the = defaultValue for var fields. I agree it's a more natural, DRY approach. If I do, @CodingDefault will probably remain as an out for lets.

I may have to go back and investigate why the original Codable synthesis skipped all vars initialized like this.

I still view this as a limited subset of the "patch" idea and something that has use cases beyond the compile-time specified defaults that 1 & 2 provide.

I think enabling the feature at the top-level still requires the same kind of protocol API changes that the full-fledged patch feature would, so why not do the whole thing if it doesn't add too much more complexity?

Thanks for the seed of the idea. I'll file an Issue for this and will consider it further.

73

I have a project where I need to use String?? and conditionally set it to .some(.none) (to make it encode the null value instead of omitting it) because there is a difference in behavior on the other end.

74

You are right, null and absent value are strictly speaking not equivalent. Your example even works for the encoding out of the box with no custom encoding:

struct S: Codable { var a: Int?? }
print(String(data: try! JSONEncoder().encode(S(a: nil)), encoding: .utf8)!)             
// {}
print(String(data: try! JSONEncoder().encode(S(a: .some(nil))), encoding: .utf8)!)      
// {"a":null}

It doesn't work for the decoding out of the box without custom decoding:

print(try! JSONDecoder().decode(S.self, from: ###"{"a":null}"###.data(using: .utf8)!))  
// S(a: nil)
print(try! JSONDecoder().decode(S.self, from: ###"{}"###.data(using: .utf8)!))          
// S(a: nil)

(which could be considered a bug).

There's a concern that explicit null in JSON may not be 100% interoperable (for example, when passed through a backend), but many things fall into this category – non-conforming floating-point numbers, numbers larger than 2^53, date/time formats, and so on.

1 Like
75

I gave this a try and built a small custom reflection-based JSON encoder/decoder (<2K lines of code) to better understand what's involved. It was a highly educational exercise! I was able to implement several features I've always wanted from Codable:

  • ~ 3x faster than JSONEncoder/Decoder.
  • Optional distinction between encoding nil optionals as null versus omitting the field.
  • Optional distinction between null and missing fields during decoding.
  • Stable encoded field ordering that matches declaration order (while still allowing optional alphabetical sorting).
  • Zero generated code.
  • Small memory footprint (roughly 200 bytes of metadata per field for each type involved in encoding/decoding).
  • Greater flexibility around missing fields; combined with a template-value-based approach, this could potentially enable the algebraic "delta" calculations mentioned above.
  • Tuple support!

Some Swift features are particularly challenging to support with this approach and are not implemented yet (for example, enums with associated values). It's also probably not suitable for Embedded Swift, as it relies heavily on reflection and existentials.

The current implementation is roughly 3x faster than JSONEncoder/Decoder in my benchmarks, for both encoding and decoding. I'm curious what kind of performance others have achieved with macro-based implementations or other approaches.

4 Likes
76

@tera Feel free to share a link to your source and benchmarks, if you like.

FWIW, the new-codable benchmarks are showing, for me, around a 10x throughput improvement (depending greatly on the particular characteristics of the data being encoded/decoded). I'd love to compare apples to apples more directly though.

1 Like