[Pre-Pitch] Codable Customization using PropertyWrappers

Evolution Pitches
21

I think this is an excellent start on a list of requirements for improving our ability to customize the synthesized Codable implementations. I think user-defined attributes is the right approach to improving Codable synthesis and customization. However, I don't think property wrappers is the right way to go about that.

Property wrappers are only available on properties. They are not available on types, enum cases or associated values. I wrote a large Sourcery template that synthesizes Codable implementations for my team. The encodings we have to support require the ability to annotate all of the above mentioned declarations. Property wrappers simply don't provide a general enough foundation to build a robust facility for customizing Codable synthesis.

Any facility that makes it into the language should be able to support a broad range of customized encodings. Instead of using property wrappers, we should have general user-defined attributes and contracts that allow them to participate in Codable synthesis (perhaps similar in nature to the property wrappers contract, but oriented specifically at encodings). Rust's serde uses an approach along these lines. @Chris_Lattner3 has previously described something similar for Swift.

8 Likes
22

True for simple case.

Not so if you add a coding key. EX: @Wrapper(key: "test")

23

Yes, property wrappers can add overhead. How much overhead depends on the wrapper. But generally the overhead is probably going to be negligible for simple wrappers.

24

I don't believe anything I've done for CodableWrappers will have a meaningful performance impact. I'm intending to do some performance tests at some point, but it's possible it would be more performant than customizing JSON(En/De)coder since it doesn't require the extra options checks.

25

If/when we gain the ability for users to customize code generation that's definitely the way to go. Maybe it's better to keep it as a 3rd party library till then.

Would there be problems with adding the options as Property Wrappers now, and then changing it internally once that capability is added? If the syntax was decided it would be source stable and whether it used a Property Wrapper or some future feature could be an implementation detail. Since Property Wrapper generates code I'm guessing there wouldn't be any ABI issues.

The extendability would probably have to be dropped, but that's still a win in my book.

26

Yes. Property wrappers are intrusive to the model. This affects layout, access control and runtime behavior. I don't think we can adopt this model and later keep the same syntax but completely change the semantics.

I think it's fine if people want to use libraries of property wrappers that allow customization of Codable in their own code. But I don't think it's something we should accept into the standard library or something that Apple should add to Foundation.

2 Likes
27

Would there be problems with adding the options as Property Wrappers now, and then changing it internally once that capability is added?

A perfect example of why this wouldn't work is the mutability/immutability discussion above. With a pure attribute-based approach, the property's mutability is directly determined by the declaration of let vs var... Whereas the property wrapper approach can't help but interfere in the native Swift grammar for property mutability because of the inherent indirection at the heart of its implementation.

2 Likes
28

Thinking it through more, there would likely have to be changes to the mutability issues as @karim reminded me to even remain properly source stable.

Regarding any layout/runtime changes, I think I'm beginning to grasp the issue. My understanding of @propertyWrapper is that once compiled it's just a Property of the Type with no special behavior/layout.

So once a simple @propertyWrapper struct Wrapper (with none of the protocol/generic business that's in my library) is compiled it's backwards deployable and valid unless/until the ABI changes. If later an attribute/code-generation-annotation was later used to replace it code compiled by either version would still be valid.

However since a @propertyWrapper also exposes a Type, it can be used directly and/or used elsewhere so removing it would break things. So to use this model that Type information would have to somehow be removed or not exposed, which AFAIK would require new language features :disappointed:

I guess it will probably have to wait till those features are added or rejected.

29

This recent thread reveals that Codable, property wrappers, and optionals don't always play well together.

It looks like it's not possible to have wrappers of optional properties support missing keys, as regular properties do.

Demonstration 
import Foundation

// A basic property wrapper
@propertyWrapper struct Wrapped<T> {
    var wrappedValue: T
}

// Make Wrapped support Decodable
protocol _OptionalProtocol {
    static func makeNone() -> Self
}
extension Optional: _OptionalProtocol {
    static func makeNone() -> Self { .none }
}
extension Wrapped: Decodable where T: Decodable {
    init(from decoder: Decoder) throws {
        let container = try decoder.singleValueContainer()
        if let type = T.self as? _OptionalProtocol.Type, container.decodeNil() {
            wrappedValue = type.makeNone() as! T
        } else {
            wrappedValue = try container.decode(T.self)
        }
    }
}

// Setup
let decoder = JSONDecoder()
let jsonWithValue = #"{"property":"Hello"}"#.data(using: .utf8)!
let jsonWithNull =  #"{"property":null}"#.data(using: .utf8)!
let jsonEmpty =     #"{}"#.data(using: .utf8)!

// A regular decodable struct: full success
struct Struct: Decodable {
    var property: String?
}
try! decoder.decode(Struct.self, from: jsonWithValue).property        // "Hello"
try! decoder.decode(Struct.self, from: jsonWithNull).property         // nil
try! decoder.decode(Struct.self, from: jsonEmpty).property            // nil

// A decodable struct with wrapped property: CAN'T DECODE MISSING KEY
struct WrappedStruct: Decodable {
    @Wrapped var property: String?
}
try! decoder.decode(WrappedStruct.self, from: jsonWithValue).property // "Hello"
try! decoder.decode(WrappedStruct.self, from: jsonWithNull).property  // nil
try! decoder.decode(WrappedStruct.self, from: jsonEmpty).property     // ERROR

Edit: I was wrong

1 Like
30

:unamused: well that puts a major wrench in this whole thing. Hopefully I can find a workaround but it doesn't look promising... Any idea if this is viewed as a bug that will be fixed?

31

Echoing previous sentiment about invasiveness and tight coupling to the model, what happens if I want my type to be encodable and decodable in multiple formats? For example, JSON and XML?

32

The value of this approach is it works for any encoder/decoder. As you can see in the Unit Tests the same customization works for both JSONEncoder and PropertyListEncoder.

It wouldn't work for different behaviors between formats, but that's a different (less common) use case than what I'm trying to solve. Though when Property Wrapper composition is fixed it would be possible to have one decoding and a different decoding.

33

I'm not sure it can be qualified as a "bug", because backward compatibility, for both successes and failures, is paramount for Decodable. There may be ways to improve the situation, though, by allowing init(from decoder: Decoder) to run even if the key is missing, and thus allow property wrappers to handle this particular case. Before I would jump on this wagon, though, I would read all posts by @itaiferber in order to become very intimate with the distribution of responsibilities in the design of Codable.

Now I welcome your exploration, and find CodableWrappers inspiring, but I agree with you: this is a major hindrance.

This does not mean one can't enjoy CodableWrappers as a third-party library. As the author of the library, you are responsible for its limitations, though.

34

Data with a missing key always fails ("keyNotFound") to decode a property wrapper. My expectation is this should behave just like a normal Optional property, but it does not. At the very least, it shouldn't fail with apparently no possible workaround. That's why this seems to me like a bug.

Though seemingly a bug without an obvious fix so it's understandable that it either fell through the cracks or wasn't seen as critical. And now that it's shipped...changing behavior gets more risky.

I've worked a bit with a custom encoder and my main take away was how generalized things need to be, so it seems like changes at the encoder level should be avoided.

My naive solution is that it could be handled by the generated Decodable implementation of containing Type, but I'm not aware of the details of how that works so don't know whether that fits well with how that feature is designed. It also wouldn't effect shipped code directly. Whether that's a win or a loss...:man_shrugging:

35

Let's try to adjust your expectations, and explain the observed behavior.

Both property wrappers and Decodable are all about code generation. From the original code:

struct Container: Decodable {
    @Wrapped var property: String? = nil
}

The compiler handles property wrappers first. It generates (from memory, excuse approximations) the following code:

struct Container: Decodable {
    var _property: Wrapped<String?> = Wrapped(wrappedValue: nil)
    var property: String? {
        get { _property.wrappedValue }
        set { _property.wrappedValue = newValue }
    }
}

Now, onto the second step, Decodable synthesis. Note how _property is not optional. This gives:

struct Container: Decodable {
    var _property: Wrapped<String?> = Wrapped(wrappedValue: nil)
    var property: String? { ... }
    enum CodingKeys: CodingKey {
        case property = "property"
    }
    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        self._property = try container.decode(Wrapped<String?>.self, for: .property)
    }
}

Now everything falls into places:

  1. The wrapped optional property generates a non-optional wrapper property
  2. Hence the non-optional wrapper property is decoded with a mandatory key.

At the very least, it shouldn't fail with apparently no possible workaround.

You can provide your own init(from decoder: Decoder) implementation.

(I'm not saying it's a satisfying answer. It's just an answer: there is a workaround.)

1 Like
36

So the question is: can we extend Decoder so that it becomes possible to 1. enter Decodable.init(from:) even if the key is missing (so that the property wrapper can be instantiated), and 2. allow this init(from:) to notice that it is decoding from a missing value and react accordingly? All of this in a 100% backward compatible way?

The answer to this question lies there. If it's possible, then this would give a great pitch.

37

I can think of 2 ways to do that,

  1. Resurrect ExpressibleByNilLiteral and have Decoder detects that instead.
  2. Find a way to express Wrapper<Value>?. Maybe @Wrapper? var value: Value

1 is already discouraged, and I do agree.
2 seems to be filling the gap in property wrapper, though I do question its usefulness outside of code synthesis.

38

I'm not sure this is the way to go, because I don't know how the wrapper could be reinstanciated after it is nil.

39

I understand the observed behavior and why it happens. My point is from a User POV wrapping a property shouldn't break decoding.

My concern with that is it's not done by Decodable, it's done by the implementation. I believe a custom version could already do this if desired, (besides the fact that it may break things). I don't think trying to force every Decoder implementation to use a specific approach is good solution.

40

Any thoughts on how to do this? @propertyWrapper? and @propertyWrapper([.optional]) are what first comes to mind.

Could a check be generated as part of the set?

set { 
    if _property == nil {
        _property = Wrapper(wrappedValue: newValue)
    }
    else { _property.wrappedValue = newValue }
}

_property.wrappedValue = newValue }